Folk Computer

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== November 2023 newsletter ======

===== What we've been up to =====


==== Applications and demos ====

  * [[https://twitter.com/naveenma|Naveen Michaud-Agrawal]] has been [[https://discord.com/channels/956758212152025098/956758650700046366/1170713666178515024|playing with the new shader system]] that we introduced last month to draw various complex things that were impossible (or at least very slow) before. [[https://discord.com/channels/956758212152025098/956758650700046366/1171072184601542678|3D bunny]] that Naveen [[https://discord.com/channels/956758212152025098/956758650700046366/1175604935312814203|rotates by rotating his cardboard dial]]:
    * {{newsletters:ezgif-4-7f386de9a4.gif?250}} {{newsletters:pxl_20231119_0111331012.mp4?300}}
    * (Omar: "it would be cool if instead of a planar dial you had some kind of [[https://3dconnexion.com/uk/spacemouse/|spacemouse]] or [[http://media.ee.ntu.edu.tw/research/DodecaPen/|dodecapen]]-like 3d proxy object/cube that you could orient arbitrarily to control the 3d model, that would have fiducials on all sides [maybe multiple cameras pointed at it too]")

  * [[https://cristobal.space|Cristóbal Sciutto]] has been working on a [[https://github.com/FolkComputer/folk/pull/111|new region implementation in C]] (mostly for performance reasons) and started on a convex hull demo
    * {{newsletters:img_8657.jpg?250}} {{newsletters:img_8650.mp4?250}}

  * [[https://hellowelcomeback.online|Tinnei Pang]] and [[https://twitter.com/djmicrobeads|dj microbeads]] [[https://x.com/localmaxim/status/1721262222872031662?s=46&t=AUhNsAKs8G7beopf-_iD3Q | drew a reindeer and pumpkin]] on their Folk system in London:
    * {{newsletters:image.jpeg?250}}
    * (I think this type of [[https://twitter.com/cwervo/status/1716149755686920481|drawing]] might be a nice [[https://github.com/FolkComputer/folk/issues/54|exercise]] for people who just got a Folk system and aren't sure what to do with it. It has both physical drawing and programming, and it has a good balance of structure and customizability where you can draw whatever you want.)

  * [[https://arcades.agency|Arcade Wise]] created [[folkarc/touch|an interactive 'page' on an iPad]], so you can touch the iPad and your touches propagate to the Folk system:
    * {{newsletters:pxl_20231103_191517745.ts.mp4?150}} {{newsletters:pxl_20231107_213647133.ts.mp4?450}}
    * {{:newsletters:pasted:20231130-161747.png?350}}

==== Tabletop / in-system editor ====

  * Naveen working on [[https://gist.github.com/nmichaud/f584bc39aa6f84cf7a275b920239b9da|a local editor]] on his system
    * Implemented inline logging (where you can call ''log'' and see the output in context on that line of code) {{newsletters:pxl_20231122_1331547582.mp4}}

  * Andrés working on multi-keyboard, moving keyboard into virtual program
    * (we'll finally be able to dump tcl-thread and pi.tcl and have everything in virtual programs and subprocesses and statement communication!)
    * Here's a preview of it monitoring key presses from multiple keyboards plugged into %%folk0%%:
    * {{:newsletters:pasted:20231130-223928.gif?350}}

==== Calibration and CNC ====

Omar has been continuing to work on 3D/real-world/accurate/metric calibration. One early application we plan on is an interface to control our CNC machine through Folk, including accurate (better than 1mm) projected 'print preview' of what lines the machine will cut on the actual material.

(the material might be an inch or two thick, so you need to be able to projection-map 3D surfaces above the base plane of the Folk system, and you need high accuracy -- current Folk is only calibrated to a single plane homography and is often 1cm off or worse)

Basically, in order to do 3D calibration, you need to hold up some known pattern in a few different poses and project onto it, and that lets you characterize the behavior of the camera and projector in 3D space. Most people say that a pretty simple pinhole-camera (linear) model with some adjustment for radial distortion is sufficient for better-than-a-pixel accuracy.

Using AprilTags for the projected calibration pattern is nice, because we already link the AprilTag library (so no need for more custom computer vision), and because they're self-verifying and error-correcting and give you a lot of information -- if you see an AprilTag, you know it was the exact AprilTag you tried to project, and you know its identity and corners and angle, as opposed to projecting stripes or something where you need your own thresholds and classification to figure out what stripes you're seeing (if any).

  * Early November: the first 3D calibration test/demo, using a calibration pattern based on [[http://www.ok.sc.e.titech.ac.jp/res/PCS/research/procamcalib/|Audet (2009)]] but using AprilTag fiducials and printing on the outer perimeter instead of alternating even/odd
    * {{:newsletters:pasted:20231130-183930.png?0x300}}
    * It's useful to leave gaps in the printed and projected patterns so it's easy to move the board & get the tags 'aligned enough' that the system can see at least 1 projected + 1 printed tag on the same plane, which is then enough information to properly rewarp all the projected tags to fit into the calibration board.
    * Then once you can see most of the projected + printed tags on the board, you record that as a set of example points for calibration (you now have both board-plane coordinates in meters and camera and/or projector coordinates in pixels for all corners of all those tags)
    * Notice how tags in the center of the calibration board are projected dynamically, while tags on the outer perimeter are printed:
      * {{:newsletters:pasted:20231106-180904.jpeg?250}} {{newsletters:img_1095.mp4?250}}
    * {{:newsletters:pasted:20231122-164229.png?350}}

  * Mid-November: We were running into a problem: how do we know when the rewarped projected tags have actually rewarped? there's some latency (tens of milliseconds? hundreds of milliseconds?) from when Folk rewarps the tags to fit the board and when that rewarp actually appears in the webcam feed. If you read the tags too early and do another rewarp based on that, the tags become unreadable and you have to abort calibration. And you obviously don't want to just have a fixed-length sleep (which is what we did at first), you want to dynamically adapt to whatever latency happens to be present.
    * So we introduced a concept of //versions//, where we rotate half the projected tags by 90 degrees every rewarp, so we can tell if the rewarp has gone through by the rotations of the detected tags
    * {{newsletters:img_1517.mp4?350}}

  * Late November: We started testing the 3D calibration but found that tag outlines were weirdly offset -- on folk-convivial, they were a few inches off, and on my home system, they were actually flipped, where the outline would appear on the other side of the table and move in the opposite direction from the physical tag.
    * Why was it projecting upside-down or offset? The answer is surprisingly cool and physically intuitive: I was using //camera frame coordinates// (x, y, z from the center of the camera lens) where I should have been using //projector frame coordinates// (x, y, z from the center of the projector lens), so I was projecting the outline as though the projector was where the camera is.
    * And in folk-convivial, the projector was a few inches from the camera but oriented the same way, while on my home system, the projector was upside-down relative to the camera!
    * Had to figure out extrinsic information during camera calibration and projector calibration (what is the rotation and translation of each pose relative to the projector & relative to the camera), then use those extrinsics to derive the rotation and translation of the projector frame relative to the camera frame ("stereo calibration")

  * Late November: Finally got 3D calibration to the point of testing it and seeing the outlines align with the tags:
    * {{newsletters:img_1639.mp4}}
    * The outline is still ~1cm off, but notice how it's the right size no matter what distance from the camera or skew I hold the tag up at. That's a 3D calibration :-) (and internally, coordinates here are all in meters from the center of the camera lens, too)

  * Next: try to improve smoothness and accuracy by using nonlinear refinement & adding a radial distortion term to the calibration

I'm excited about real-world calibration because it feels like the first system change in a long time (6 months? a year?) that will actually bring in new modes of interaction, that will //look// different to the untrained eye from putting pieces of paper on a table and watching them light up. (This is also why the [[guides/button|button]] is great.)

I've spent a lot of the last 6-12 months on performance and on making cleaner and more powerful APIs, but I haven't pushed the interaction paradigm or the application set enough yet. Seeing preview lines on chunks of wood on a CNC machine bed, having a projection-mapped dollhouse or town, highlighting and sharing sections of a book, writing in blank fields on printed programs -- these are the directions where we should be pushing.
==== Other system improvements ====

  * Naveen [[https://github.com/FolkComputer/folk/pull/113|hugely improved text rendering]] using [[https://blog.mapbox.com/drawing-text-with-signed-distance-fields-in-mapbox-gl-b0933af6f817|smoothstep technique from this Mapbox article]]
    * {{newsletters:595061ff-e618-4c97-92ed-ccc817b62c43_1_105_c.jpg?250}} {{newsletters:61e30bb0-ae18-4601-b95c-18d460e7f29c_1_105_c.jpg?250}}
    * {{newsletters:01688b49-cb59-427b-85fe-4e72b43e5d40_1_105_c.jpg?250}} {{newsletters:71c17aac-f28d-48a7-af01-65cccac9ad68_1_105_c.jpg?250}}
    * {{newsletters:2796c561-e0f7-43ae-ad25-e37c49421581-1140-0000dcd1db1b2e72.png?400}}
    * The text rendering improvement boils down to a really simple diff, basically just:
      * {{:newsletters:pasted:20231130-173930.png}}

  * Naveen added text alignment to custom anchors: instead of always centering on the position, you can have the text go to the right, or to the left, etc

  * Naveen added [[https://github.com/FolkComputer/folk/pull/112|non-uniform (different in X and Y) image scaling]]

  * Naveen added [[https://github.com/FolkComputer/folk/pull/119|mailbox clearing when a subprocess gets killed by unmatch]]
    * but **not** (yet) when subprocess goes down due to an internal crash (like a segfault) or due to arbitrary external Unix kill (we have this whole supervisor thing already that's meant to be able to waitpid on each subprocess and detect those conditions, we're just not using it)
    * see [[https://discord.com/channels/956758212152025098/956758650700046366/1175908063019008021|Discord discussion of mailbox clearing]]

=== Proposal: Parallel Folk ===

Omar has been throwing around ideas for a parallel Folk evaluator. It's an interesting design problem -- was talking it over in Discord with Naveen and Arcade and others.

The main motivation is actually sort of a user interface motivation: I want people to feel comfortable writing programs that hang or segfault, and I want people to feel comfortable putting 30 different programs on the table, without being scared that they'll crash the whole Folk system or break their (in-system) editor. That requires Unix process isolation, I think, and it requires preemptive multitasking.

And having the system schedule and parallelize everything itself could reduce latency: right now, even though the system is multiprocess (camera intake, tag detection, and display run on separate Folk subprocesses), the main process is a latency bottleneck for the entire system. 

<details>

{{:newsletters:pasted:20231130-193154.png?500|you can kind of think of latency as being about how long it takes, with respect to some user action like moving a page, to do the specific critical execution path of input (camera intake) -> compute (apriltag recognition, program execution, shader wish construction) -> output (vulkan draw)
and right now that blocks on the entire evaluation of a Folk step in the main thread which has tons of non-critical stuff in it as well
a vague thought -- i was talking to Cris about this a while ago -- is that a parallel evaluator wouldn't even have a canonical framerate or converged frames, it would just sample whatever display commands happen to be in the database every few milliseconds and blast those out}}

(And I think it'd be more elegant -- it's a return to some of the original tuplespace pitch, as a concurrency model -- it could free the user from some implementation details of processes and memory allocation and what calls block and what calls are fast -- it could let debugging tools have more access to the running database and not interfere with the runtime -- it would enforce a very attractive purity that you can't use global variables or functions, everything really must be shared through the database, forcing us to improve the database and language ergonomics accordingly)

> [[https://discord.com/channels/956758212152025098/956765077883744266/1176921576483078345|Continued thinking about parallel Folk:]] I feel like there are two sort of platonic designs with different advantages & disadvantages
> 
>   * One Unix process per Folk program (page or virtual program)
>
>   * One Unix process ~per CPU / process pool
> 
> The process-per-program design is probably more intuitive based on the interface we present to users -- you can have global variables or open file descriptors (or talk to the GPU, or a webcam, or whatever) and they'll work within the same program, it's like 'lexical scope'.
>
> But it doesn't parallelize at all: for example, what if you have a program that does  When /p/ is a tag { 5ms of computation on $p }? you'd want to parallelize that across 7 tags, not run that block serially. (maybe the answer here is the system can spawn threads within the process for within-program parallelism)
> 
> The process-pool design is easier to schedule and probably more efficient, you would do something like https://en.wikipedia.org/wiki/Work_stealing. But it would be really hard to get stuff like Vulkan or webcams to work without adding some barnacles like pinned blocks (since those rely on process-local resources).
> And I don't know how well we could do fault isolation in a process pool since the process boundaries would be sort of arbitrary (it might be OK, like you could just regenerate the pool and rerun things that go down, but it's less obvious than if program = process)
>
> One way to think about it is that there are a few properties we want, and they kind of conflict with each other. among those are:
> 
>   * if C code segfaults, we want to contain the fault to a reasonable level and not take down the whole system
>
>   * if arbitrary Folk code does sleep 5 or exec glslc or inf loops, we want it to not block the rest of the system (including potentially other invocations/matches of the same code that are responding to diff statements)
>
>   * we want to be able to open a webcam or Vulkan context and then be able to read/write it later
>

----

One thought is that a parallel Folk evaluator would basically be a mix of [[https://jim.tcl.tk/index.html/doc/www/www/index.html|jimtcl]] + a C kernel/database/evaluator which knows all the statements and can pull a When body string out and jimtcl-evaluate it on any CPU
 
(since those strings should be completely self-contained, other than parameters which come in via bound variables on the When)

{{:newsletters:pasted:20231130-201215.png?500|What's the runtime penalty with jimtcl?
osnr — 11/22/2023 1:58 PM
Image
I think it's up to 2x but depends on workload
my thinking for parallel Folk is honestly that I would give up even 3x or 4x on throughput in exchange for better latency and not degrading under load
jimtcl doesn't do bytecode compilation so it can cope better with dynamic code
arcade — 11/22/2023 2:42 PM
As long as the display stays fast and relative responsive, low "update rates" are ok
nmichaudagrawal — 11/22/2023 2:47 PM
Yeah Folk has the feeling that things eventually converge
}}

</details>

=== Proposal: Real-world 3D regions/points ===

Now that 3D calibration is almost ready, Omar is starting to think about how the geometry representation and display API in Folk will have to change to accommodate points in real-world units and real-world 3D coordinate frames. (and how to future-proof so you can eventually throw more projectors/cameras/other sensors and actuators at a Folk system and integrate all the data cleanly)

It will probably require dumping/breaking a lot of existing hard-coded display and geometry code that thinks in terms of 2D planar pixel coordinates.

> [[https://discord.com/channels/956758212152025098/956765077883744266/1179174170559516752|Ideas on new point and region formats:]]
> 
> A point would be a list {frame x y z} where frame is a string that (ideally) universally-uniquely identifies a projector/camera/apriltag/other coordinate system, and x y z are in meters. transformation between frames should just be rigid (rotation and translation) since they're all real-world coordinates
> 
> An example frame would be folk0:/dev/video0, indicating a point in the coordinate system of the camera /dev/video0 on the node folk0. Then x y z would be in meters from the center of that camera lens, where a point 2 meters away would have z=2, etc. (similar for a projector or an apriltag id being the frame -- in those cases, you'd measure from center of projector lens or from center of apriltag, respectively)
> 
> A region might be similar to regions now -- list of 2D points and list of edges between those points, or maybe some other way of specifying 2D areas -- but with the addition of a frame field like the frame field on points, and with all 2D points now being in meters. (Many regions would probably have frame being an AprilTag)
> 
> (we might not actually need points and could do everything with regions) 
> 
> one goal is to make it possible to have ensembles of cameras and projectors and seamlessly integrate them
> as well as depth cameras, RFID, phone/tablet localization, etc 

==== Outreach and new systems ====

  * Built a new folk-beads system in the front of Hex House, for [[https://www.amant.org/programs/111-for-your-reference|event with Amant]]
    * (using 1080p Optoma EH320UST ultra-short-throw we found on eBay, which is fun because it's relatively easy to mount and very bright)
    * {{newsletters:img_1503.jpeg?0x250}} {{newsletters:img_1497.jpeg?0x250}} {{newsletters:img_1612.jpeg?0x250}}
    * Would be cool to put this on a cart and be able to wheel it around and bring it in and out
    * Fun interim on-a-wall stage during setup, before we pointed it down at the table:
      * {{:newsletters:pasted:20231130-161524.png?250}}
      * I think this image is a fun provocation because it's sort of //about the wall//, the drawing on the whiteboard, instead of being about the system. Imagine wheeling a system over in 10 seconds and pointing it at part of your environment because you want a thin layer of computation or sharing or dynamism on top of what's there (and not because you want a generic "Folk system" set up that could be set up anywhere)
      * (btw, whenever I see a projection mapping demo, I always feel like we should be able to subsume it and make it really easy to set up and reprogram, like it was trivial to make the meow text here wiggle around over time)

  * Simplified install instructions to just use Vulkan binaries from apt, instead of compiling mesa from scratch

  * Another successful open house on the 29th:
    * {{newsletters:img_1626.jpeg?0x200}} {{newsletters:image2.jpg?0x200}} {{newsletters:img_1625.jpeg?0x200}} {{newsletters:img_1618.jpeg?0x200}}
    * It's nice that we can put up and tear down folk-beads in like 15 minutes total -- the PC doesn't need to be reformatted and we know the exact physical setup to do (including getting the projector at the right height and clamping it to be stable) and recalibration is fast
      * Got a Logitech C922 camera for folk-beads: at least it can do 60fps at 720p, much smoother-feeling than the C920 or Azure Kinect which can only do 30fps at any resolution

===== Hmm =====

{{newsletters:pasted:20231130-213756.png?550|one observation re: how people pattern-match and overfit to the examples they see is we get a lot of questions and comments about the AprilTag black-and-white markers we use, even though I would say we view those as much less essential than the programming system and database

and there are a lot of applications like CNC preview or various projection-mapping demos where I think you should be able to get lots of value out of using Folk w/o having any tags visible in the end interface at all}}

(I think part of our job has to be to just broaden the examples so that fitting to them still gives you a pretty productive working set)
===== What we'll be up to in December =====

  * **[[https://partiful.com/e/MJ6V5SfEKRPPjOskxeti|The next Folk open house]] will be on the evening of Wednesday, December 13, at our studio in East Williamsburg, Brooklyn.** :-)

  * Still need to complete RFID refactor. I need to make the ringbuffer faster and I need to wait on less data to start parsing ..

  * Finishing work on new calibration -- trying to bring up accuracy and stability now, then introduce new 3D/frame-oriented coordinates

  * Might work on Vulkan user-facing shader language

  * Hacking on parallel Folk?

  * Finishing up tabletop editor


===== Links we've enjoyed =====


==== Andrés ====
  * [[https://github.com/tophtucker/printcode|Toph made a tool to print out a repo of code nicely]]
  * [[https://twitter.com/disconcision/status/1710716024323006878?s=12&t=AUhNsAKs8G7beopf-_iD3Q|Fence diagrams]]
  * [[https://www.youtube.com/watch?v=Y8Qm48_GhJ8&list=PLYCO5FdtsVwtOT66VfhUkkRjIyLpKkTLB&index=7&loop=0|Andrew Sutherland and the Paper Computer, interactivity, computability and no screens!]]
  * [[https://www.midst.press/|Midst - What if you could watch your favorite poet write?]]
  * [[https://dailypapert.com/hard-fun/|Hard Fun — Seymour Papert]]
  * [[https://pixelpoppers.com/2018/03/who-frustration-is-good-for/|Who Frustration is Good For
]]

==== Omar ====

  * [[https://hal.science/hal-01389076/document|Practical and precise projector-camera calibration]]
  * [[https://en.wikipedia.org/wiki/Work_stealing]]
  * [[https://sosheskaz.github.io/tutorial/2016/09/26/Avahi-HTTP-Service.html|How to use Avahi to advertise an HTTP service]]
  * [[https://sci-hub.ru/10.1109/2.44903|Linda meets Unix]]
  * [[https://types.pl/@disconcision/111467347191732238|nool sounds]]
  * [[https://blog.geocities.institute/archives/7047|Shawna, a Set Maker]]