August 2026 newsletter

Our next Folk open house will be in the evening on Wednesday, September 30, at our studio in Williamsburg, Brooklyn.

• Amanda Yeh made a musical instrument that plays scales based on how you move an object (using the new SAM2 tracking stuff Omar and Karl and Audrey have been working on). Here, tracking a cup and making sounds:
• Audrey Gu has been working on a book interface for Prev and Next-type interactions (moving discretely forward and backward through a space); she wrote a very interesting blog post with some examples & thoughts about why and how this interaction feels different from moving pieces of paper, even though it's 'technically' equivalent
  • 
• Rob Fielding has been working on a dotframes demo – more on this next month
  • “In folk the tangible computing could be about exploring the computation. It is eyeopening to see how the computation is running. Like a debugger. The hard part is scaling this concept without creating a mountain of paper or hidden complexity in code.”
  • 
    • “Cursed corners stemming from halo noise as observed through the interim stage of the pipeline itself. The issue a combination of the proximity of the dots and the threshold of the detection.”

Mason: Working with 2D points can get pretty annoying, especially when you have multiple programs involved (what would a point on program 2 be when projected from program 1?), so I decided to sit down and think through what would be an intuitive API to use. I settled on a "points" API , that lets someone transfer a point from one program to another, or (eventually) a point from a program onto the table. This means that point projection isn't limited to just the table, but can be from any program to any other program.

This is currently a bit clunky, so please let me know if you have suggestions, but this is what it looks like from the user's point of view:

Expect! the points library is /pointsLib/
When point [$pointsLib point 0.1 0.1] has point /newPoint/ on 123 {
  lassign $newPoint type surface px py
  Wish to draw a circle onto 123 with center [list $px $py] \
       radius 0.1 color green filled true
}

It needs to be in a When, since as the programs move in physical space, their coordinates update, so this also needs to reactively update. I currently use a When when for this, so the user can just write a When and have it automatically start running point conversions.

Here's a video showing the API running right after I got it working:

I got this API finished just in time for a hackathon I went to, which was a lot of fun! We ended up doing a calculus visualization for the definition of a limit (shout-out to Katy who I did the hackathon with! She came up with the idea for the hands among other things, and picked up Folk impressively fast):

Brian and Omar remounted the camera:

Omar figured out that we could turn the AAXA 4K1 projector's keystone off to cover more of the table (see how left is smaller, right is bigger) (he noticed a discrepancy between the boot screen and the projected screen):

Ended up with a pretty good calibration:

Omar and Brian started on reactivating the CNC machine that we set up Folk for a few years ago. (We haven't used the CNC machine since moving from Hex House to the new studio, and we haven't set up a Folk system over the machine again. We had to get a new shop vac just now, at minimum.)

As part of other CNC work Omar is doing, we're going to try to modernize the folk-cnc stuff (make compatible with folk2, make reliable and robust & usable by a random studiomate who walks up to it) and make it a primary way to drive the machine.

How do we mount the projector above the CNC machine? We have already wanted ceiling-mount tech for our folk-convivial system, so we got steel unistrut and threaded rod and some other things to actually attach to ceiling. Hopefully this is a general tactic we can use to mount a bunch of systems to the ceiling. (It should be able to bear hundreds of pounds of weight if you actually use a flange hanger and wood screw into a real wood joist in ceiling.)

Ultimately, ladder didn't go high enough, so we haven't been able to ceiling-mount yet:

Also got shop vac and (thanks to John Williams) cyclone filter:

In general, it's nice that we're developing decent craft knowledge of how to mount medium-sized projectors (above the Projector Sewing level, but not more than 10-15 lbs)

• Omar fixed saved-hold write concurrency bug, so now we hang on to the global holds mutex while writing the saved-hold file
  • The problem was if you did multiple Hold! -save at once, especially to the same file, they could all run in parallel and tear up the file with conflicting writes. I think this was causing a lot of the problems with calibration, since calibration writes a bunch of separate statements in sequence
  • This bug causing weird torn saves on calibrate/settings change:
    • 
    • (Omar would also look at these erroring hold files and find weird syntactically invalid data: mismatched curly braces, statements cut off mid-line, etc, which hopefully is all just manifestations of this problem)
• Expect! now inherits destructors into current match: need this to safely & automatically retain camera slice for pose registration in object tracking (so we can display it for debugging in our 'legend')
  • There is a design question here about whether Expect!, Query!, etc are default safe (and therefore come with leak risk, if you do something like Expect! on every iteration of a loop), or default unsafe (as is suggested in part by their exclamation points, although I think of exclamation point as more connoting 'not reactive')
    • leaning more toward default safe at this point, having been bitten by bugs like this
• Tried to fix atomically pin bug where the root Match of an atomically was never reaped
  • This could have affected anything in an Atomically convergence-tracking context, so downstream of a page quad, camera slice, etc – all those statements would stick around
    • Probably manifested most in page titles and quad masks that would stick around, since a lot of other drawing is on the page canvas which isn't affected by this
• Statement lifecycle fix which should also help fix statement leak bugs on kept statements (-keep), which could stick around forever due to getting revived before removal but the old timeout preempts future removal forever
  • This probably mostly affects titles as well in practice

For the last few months, we've been working on object tracking with Karl Rosenberg and Audrey Gu.

Tracking a weird squiggly thing and an army toy using Segment Anything 2 video predictor and some post-processing to get the green 'center point':

I want to step back and talk about object tracking strategies, because there are a few different problems to solve to get the system we want, and we've tried a lot of different things at this point.

How do you distinguish 'objects' from the background of the viewport? (i.e., given, basically, a RGB camera slice of the viewport) (Assumption: you preregister 1-4 poses of the object you want to track first, maybe have 1 or 2 measurements you type in, but don't have a full 3D model)

• Basic findContours on binarized camera slice
• Segment Anything 2: register the object you want to track by (x, y), then track the surrounding segment's movement forward from there (SAM2 can do frame-by-frame tracking of a segment in a video or camera feed)
• CoTracker initialized with center point of object: similarly, register the point on the object, then assume the object is whatever is around it over time

How do you identify those objects?

• Basic findContours: judge by color distribution and maybe shape of the contour?
  • The problem here is that contours can change shape radically as you rotate the object and move it relative to the camera, if it's a non-flat object (like army guy).
• SAM2: for free from segment identity
• CoTracker: for free from point identity
• but what if the object is moved out of frame and you need to re-identify it?
  • SAM2: can jiggle it / place back at registration site to get re-recognition, but annoying
  • CoTracker: even worse
  • this leads to some of the embedding stuff we're doing in September

How do you get an accurate footprint contour (for things on table to collide with)?

• Retain the rigid 3D shape of every object and match it to the 2D image? then derive footprint
• Same-side arc contour and flip
• Draw the footprint contour; derive from center and orientation.
  • But then how do you get an accurate center?
    • Raw centroid of contour
    • Compute centroid transform by registering centers at a few different points
    • Same-side arc contour or opposite-side arc contour: if you just take the side either near the camera or away from the camera, you know it's consistently the arc of the top or bottom of the object
      • 
      • This was most promising for a while and felt most principled

Interpolation (SAM2 is pretty slow, 60-100ms per frame)?

• Optical flow: I feel like this should work and let us run at 60fps, but in practice it was jittery? Maybe worth looking back into if we go back to something as slow as SAM2

One thing we've wanted is for object registrations (“track this cup”, “track this ball”) to persist across system crashes/reboots. Then we would feel more confident playing around with it even if the system crashes, and it would feel more like a basis for a permanent game or rulebook instead of an ephemeral tech demo. (it'd be more like a peer of AprilTag recognition, I guess, which similarly works immediately after you restart the system, so you can take it as a solid primitive)

Implemented by saving all this random Python state from SAM2, which does work to some extent if the scene continues similarly enough after restart, but has re-ID problems, and (inelegantly) has to save to a separate file instead of being in a persisted statement.

Registration code has you put the object at 3 poses, so it needs to save the camera slice at each pose (the camera slice's lifetime has to be extended beyond normal).

Problem: registration uses Expect! to get the pose, which wasn't inheriting (keeping-alive) the destructor for the camera slice, unlike a When would. So you get random use-after-free where your registration poses are garbage. Garbage in registration poses due to use-after-free:

Another use-after-free, seen from the web ui:

Solution for now: Expect! inherits destructors into current match. It's actually not that clear that this is the right long-term design choice, since it'll now leak if you use Expect! in an infinite loop or something, but seems ok for now. (You also have to be careful to use Expect! inside the Hold – so it's in same block as the statement claim – and not separately before the Hold, since the Hold also won't inherit the destructors. Still lots of pitfalls with destructors/Hold/Expect.)

Last month, we got the physical setup and basic tag identification to work again. That all only uses the in-band radio and software.

The next step is to resume building the software to run on the out-of-band radio, which is what hops around different frequencies and acquires the wide-band channel estimate that we actually use to localize.

Omar: A lot of the work this month was trying to align in-band with out-of-band signals so we can localize. (the IB radio and OOB radio are different antennas and different computers and will be on different frequencies most of the time) That's the “systems problem” we were discussing last month. We have successfully done that, for the most part, which is a nice advance over the first time we did this a few years ago.

We started by having IB run on 915MHz (as usual), and then also having OOB run on 915MHz, then trying to align them. This is the easy case – no need to only sync when you're sure OOB is on same frequency, no disruption due to OOB hopping around, you can always just parse exactly what you know you sent from IB. You can just lock onto the signals you know you're sending from IB with known absolute timing.

I started by just sending a pulse of known length (number of samples, basically equivalent to number of microseconds in theory) with some padding of known length around it. This didn't work that well. Here're some unsuccessful alignments:

Now a half-successful alignment:

I found a few issues this month:

• Bad sync pattern: the single pulse I was using (from IB to itself and to OOB) was not distinguishable enough from actual RFID protocol communications. I probably could have tuned the thresholds and detection algorithm, or just have OOB decode IB RFID messages directly, but instead using a longer / more unique pattern for now. We sync every round, not just at reset time, because there is drift
• Tearing: rounds would stomp previous rounds in RAM on the radio mid-copy, so even if they were fine on the OOB radio, we were getting torn data on the PC which was totally wrong, so you'd get apparent desync in the middle of the round. Need to lock the slots on the circular buffer on the radio.

The upshot of this is that if we have a full alignment and a validated EPC, we now have known-good data over the wide band that we can use to localize! That's the basic ingredient to actually get something out of this project.

I've started doing localization. It feels more valid than a couple of years ago – less garbage and random flickering, and when it works it works.

A big limitation right now is just pickup range. I made a decoding improvement, to use phase and not just amplitude, which helped a lot, but still more to do next month.

We can definitely pick up on the tag at 1ft – 3ft is when it gets sketchier. (way below the advertised 10m range of UHF RFID…) And if we can't pick up on the tag, we definitely can't localize:

Suppose our localization system (in-band radio + out-of-band radio + antennas + PC) wants to localize a tag. Here are the 'tiers' in my head of what kind of response we might get.

1. No response from the tag even to our initial QUERY (we failed to power it, probably it's too far away)
2. Successful RN16 response from tag, but we fail to decode it (maybe drowned out by noise?) (maybe failed to meet response deadline)
3. Successful RN16 response from tag, but we decode it incorrectly, send the wrong RN16 back, and then never get a EPC from tag (RN16 has no checksum!)
4. Successful RN16 response from tag, we decode it correctly, send the correct RN16 back, then get an EPC from tag, then fail to decode it (maybe drowned out by noise?)
5. Successful RN16 response from tag, we decode it correctly, send the correct RN16 back, then get an EPC from tag, then decode it incorrectly and fail checksum
6. Successful RN16, decode correctly, send correct RN16 back, get EPC from tag, decode correctly, checksum succeeds. We have a valid EPC from the tag!

(These are tiers for an individual hop. There's a further sequence of tiers that is about the whole round of hops, where we're hopping every 20MHz to cover 800MHz to 1080MHz to get the wideband channel estimate.)

We're starting to break these tiers out into the PC UI.

Note that we can only localize with data where we know we've decoded the tag response correctly, since only then do you have verified-good bit timing boundaries and reflective/non-reflective distinction! For now, we've been using only tier 6 data, but I guess you could use lower tiers if you have a known-good RN16 but not EPC? But in practice, I don't know if there's a way to establish that. The checksum is really reassuring compared to anything else where you might literally be decoding noise (our decode is often 'maximum likelihood' style where you assume meaning is there and find the most likely place for it to be, rather than yes/no).

Mason: Having time during the summer really makes a big difference in what I'm able to get done! I rewrote the whole core of Zicl to use a new object type that's much more C-like, a simple allocation with reinterpretable memory for shimmering. That touched pretty much every part of the interpreter, so a lot of time was spent porting my own code over to the new system. The rest of my time has been spent implementing the standard library (and fixing dictionary sugar bugs, my gosh that took so long). The biggest things I landed:

• Much more of the standard structure manipulation commands: [dict], [list], [lsort], etc
• Capabilities! Capabilities are a threadsafe way to open and close anything (files, sockets, processes, pointers, etc). There's a number of features that they have that make multithreading easier to safely expose to Tcl, such as use-after-free protection (both preventing using after freeing, and preventing closing too soon), internal reference counting, and being able to round trip to a string and back. Currently capabilities look like

  <zicl://folk-peridot.local/file-handle/sVye-a_2s3tvQm8xR4pLnW>

  , since they're a unique URL per system. The id part is a randomly generated 128-bit key to prevent forging.

• [letrec] - Mutual recursion is now possible, and has a string serialized form! This one was tricky to implement, but so far it's been really solid. It even works with methods.
• Implementing [exec] - More on this next.

I've started getting really bogged down by the details of correctly implementing IO. At first I vibe-coded some of the IO stuff like [exec], but it turns out multithreaded IO with POSIX semantics are kind of terrible to nail down correctly, and the code had some major issues. Did you know pids can be recycled? Or that the thread someone's running on might disappear before you can use it, triggering UAF? What about buffered IO? When do you flush? Positional vs streaming reads? What if you have multiple readers from different threads? Where does [gets] buffer the results when searching for the terminator? What does O_APPEND guarantee? Opening a directory as a fd to read files from, or read the file path directly? The semantics Tcl exposes aren't great for handling these either.

Zig's std.Io has helped a lot with getting the semantics right, but I ended up needing to vendor it since it doesn't work nicely with Folk's scheduler. Hopefully I can emerge at some point and do nice user-level things, but for now I'm in the guts of IO gotchas.

August 7-9, Daniel and Mason: We displayed our system at a booth at the Craft Lake City DIY Festival in Utah. We had a chance to show it off to many people with a wide variety of backgrounds. It resonated with everyone, but only after some demos and interacting with the system. This reinforces the importance of having a range of compelling demos. I borrowed heavily from Forrest O's my-folk-data repository for photo uploads and dial demo. This feels like a good way we can share virtual programs with each other, checking the ~/folk-data folder into git.

Sam Whitlock and his brother came by and made a hatchling animation.

Mason's thoughts: It was such a cool experience for two reasons: 1. It was one of the few times I got to work with another person on Folk in person! So much of working with Folk is asynchronous, at least from my side, so it was a ton of fun to get to bounce ideas off of Daniel. In particular Daniel helped walk me through getting Super Collider working, which I probably spent all of Saturday working on (at least during slow periods):

2. It was a really great chance to work on my pitch for Folk. At first Daniel and I really struggled to communicate what Folk is, because there's so many pieces to it. Over time though we really started to refine our pitch by starting with the obvious, and if someone stuck around, building up to the whole vision. Some examples: “We're trying to get the computer out from here <gesturing at laptop> to here <gesturing at table>.” “Here we have a perfectly normal lid, from a can of olives, and we can turn it into a dial by sticking this piece of paper on top.” (on the Elmo pointing program) “Here we have a program, and we can point it at other programs. See how the little Elmo animation pops up? The programs can sense each other.” And other such sayings. All in all it was a blast! I got to meet so many wonderful people, and got to share about how cool Folk is.

The event went really well and we'd love to show it off at more events. Contact [email protected] if you want us to come to one near you.

• Andrés and some of the SVA Interaction Design summer interns (Seong Ju Kim, Sophie Lee) revamped the folk-sva system:
  • 
  • 

• JP Posma, Louis Potok, Han Seoul-Oh (friends of JP's, who we know from SF and Paper Programs came by)
  • 
• Omar's friend Michelle Park visited in late August
  • 
• John Williams (currently at Recurse Center) visited, talked about CNC filtration

• We had a smaller open house this month, but still pretty active.
  • 
  • We showed off the latest object-tracking stuff, which works pretty well, and Amanda's music program that builds on the object tracking.
    • We also brought out the old raw contour tracker, which is still in tree and still totally works with the acrylic viewport, which is a nice feeling (this is the advantage of physical programs that just live in folders; you can dig them up later). Some very fun feedback effects, and the low latency of the raw contours is a nice feeling after dealing with SAM2 for the last few weeks:
      • 

• Our next Folk open house will be on Wednesday, September 30, at our studio in Williamsburg.
• Andrés: Mounting CNC
• Andrés: Simulating boat physics with model boats
• Omar: Setting up larger Folk system (with very long throw) for project in Vancouver
  • Bring back dual camera support?
• Omar: Continuing to iterate on RFID localization
• Omar: More object recognition work to try to get accurate center, orientation, re-ID when objects go offscreen

• Why your bank card is less secure than you think - video about magnetic striping

• https://en.wikipedia.org/wiki/Blokus
• https://research.swtch.com/pcdata
• https://without.boats/blog/coroutines-and-effects/
• https://www.publicbooks.org/every-novel-is-boring-until-it-isnt/
• https://old.reddit.com/r/computervision/comments/1rh05j4/how_much_of_a_pain_is_procam_projectorcamera/
• https://gitlab.com/kriwkrow/lighthouse-stylus
• https://github.com/NVlabs/Fast-FoundationStereo
• https://www.flong.com/archive/projects/free-universal-construction-kit/index.html
• https://30fps.net/pages/videogame-shadows/
• https://gwern.net/review/movie#rams
• https://en.wikipedia.org/wiki/Labanotation