Differences

This shows you the differences between two versions of the page.

--- newsletters:2026-09 [2026/10/02 20:13] – dpip
+++ newsletters:2026-09 [2026/10/07 21:10] (current) – [September 2026 newsletter (WIP)] osnr
@@ Line 1: / Line 1: @@
-====== September 2026 newsletter (WIP) ======
+====== September 2026 newsletter ======
 {{htmlmetatags>metatag-og:title=(Folk Computer September 2026 newsletter)
-metatag-media-og:image=(newsletters:FIXME_img_8432.jpeg)
+metatag-media-og:image=(newsletters:img_8752.jpeg)
-metatag-og:description=(Embedding-based object tracking; RFID localization works; Rob’s dotframes demo; SVA sound effect animals; Interface Studies podcast; color wheel contours)}}
+metatag-og:description=(Embedding-based object tracking; RFID localization works; SVA sound effect animals; Interface Studies podcast; color wheel contours; atomically and graphics speed fixes)}}
-Our next Folk open house will be in the evening on **[[https://luma.com/wa18i4hg|Wednesday, September 30]]**, at our studio in Williamsburg, Brooklyn.
+Our next Folk open house will be in the evening on **[[https://luma.com/lggnfaib|Wednesday, October 28]]**, at our studio in Williamsburg, Brooklyn.
@@ Line 14: / Line 14: @@
   * editor: [[https://github.com/FolkComputer/folk/commit/fe6d150de871fc7294a2b9500d14733814a026e8|Only target the smallest program the editor points at]]
+    * Fixes this issue where an editor pointing at two programs at once results in two unusable/wire-crossed editors:
+      * {{:newsletters:img_9552.mp4?250px}}
+      * (especially annoying when on a viewport, since the editor would always point at both the viewport and any program inside the viewport)
   * [[https://github.com/FolkComputer/folk/commit/4d423859b6dc12b95c93699a4202fffe24363fb8|Fixed how bare claims totally broke Folk]] by triggering everything
     * (for example, if you just put ''Claim'' on a line on its own in a program, it would break the whole system)
@@ Line 19: / Line 22: @@
     * Should finally fix the random space changer failures which would fill up the log! (was happening when space changer initialized to null changer and then filled later, but original changer function was not actually replaced)
   * /program/<whatever> Web endpoint: [[https://github.com/FolkComputer/folk/commit/5d2cca7d0908501b5938cdf9ed5b72d0b84106b2|Show the edited version of the program]] if it's been edited
+  * [[https://github.com/FolkComputer/folk/commit/723d284503373b09bdf5477f27a79dbf8c8f91a9|Made destructor inheritance dedupe on each set add]], which fixes destructor blowup (and therefore out-of-memory; you can have millions of destructors, only 4-5 of which are unique) in cycles of reused/reinherited statements (which seem more common now? not sure why? I think it was maybe from editor pointing at itself during rapid movement, causing an inheritance cycle?)
+  * Python FFI: [[https://github.com/FolkComputer/folk/commit/d53fd676079f98b34622a831d1b5df91d7230588|fixed spurious zombie detection that would kill off Python process when main thread died]], even if Folk was actually still running and we were just terminating main thread's worker
 === Atomically ===
-  - [[https://github.com/FolkComputer/folk/commit/61a193000af535e2c79d6fac83fb11fa63faecb1|dbInflightDecr on the exit path]] where provisional statement is immediately thrown away
+[[https://github.com/FolkComputer/folk/pull/278|Made a few major fixes to Atomically]] to fix blinking and stuckness issues with it.
+I saw a lot of these issues with RC visitors who were making a bouncing ball (clock time atomically) whose color was based on color wheel (quad atomically). Blinking:
+{{:newsletters:img_9136.mp4?200px}} {{:newsletters:img_9135.mp4?200px}} {{:newsletters:img_9133.mp4?200px}} {{:newsletters:img_9322.mp4?200px}}
+Fixes:
+  * [[https://github.com/FolkComputer/folk/commit/61a193000af535e2c79d6fac83fb11fa63faecb1|Add correct dbInflightDecr on the exit path]] where provisional statement is immediately thrown away (to make sure the inflight doesn't stick forever and keep the new version from converging)
     * Should fix "(edited)" titles sticking around randomly / after program revert.
-  - Multiple inheritance
+  * [[https://github.com/FolkComputer/folk/commit/f76f3564da9080b8dc1557bbc291355f806a23e3|Support multiple inheritance]] so a Match or Statement can be part of multiple AtomicallyVersions at once (from different keys)
-  - One-way hasConverged (latch convergence)
+  * Support auto-atomically in first pattern in & join
-  - Timeout by parent removal
+  * Collect retains previous version when atomic
+  * One-way hasConverged (latch convergence) to prevent 'livelock' where no version completes for a long time because versions keep getting unconverged by new work
+  * Timeout by parent removal instead of by time of last convergence: fixes title/outline blinking out on stabilized quad (where the last convergence is often very old but still valid)
+    * You can also use the -keep option on the When to set a custom timeout, for things where you expect re-convergence to take a long time or just never want expiry (it's a 'singleton')
+  * [[https://github.com/FolkComputer/folk/commit/3815a0d235f55de1185aea223dbe6303cceb5fb1|Do fresh atomically at match creation time instead of prologue]] to fix blinking because empty AtomicallyVersion (when root is invalidated) races and is accepted as valid
+Also comes with a [[https://folk.computer/notes/animation|new version of the animation program]] that can now blink much less and be more reliably timed, fixing bugs like this:
+{{:newsletters:img_9553.mp4?250px}}
 === Graphics speedup ===
 Omar: As long as I can remember, the editor has been the slowest thing in the system, where you point an editor at a program and the whole system slows down noticeably. It's especially bad on programs that have a whole page of code (that's like 2,000 glyphs).
 I benchmarked it this month and found that a 2,000-glyph editor slows the system down to 8fps (and this is on the global GPU thread, so it slows everything else down too).
-[[https://github.com/FolkComputer/folk/pull/279|Simple fixes: cache the C encoding of all Tcl draw calls and turn off the Vulkan validation layers.]] That immediately brings us to 40-50fps.
+[[https://github.com/FolkComputer/folk/pull/279|Simple fixes: cache the C encoding of all Tcl draw calls and turn off the Vulkan validation layers.]] That immediately brings us to 40-50fps. Much more usable.
+Next, I want to do more editing and terminal I/O in the system. I think we can also push draw call batching further and maybe then bring back the validation layer.
+==== RFID localization ====
+Current state of RFID localization is that it can talk to the tag at up to around 1.1 meters (but drops off afterward) and can also localize it reasonably well to within a couple cm. We went up from like 30% successful tag detection rate at 1.1m to around 90%:
+{{:newsletters:img_8722.mp4?400px}}
+At home -- it talks to the tag fine on table but not once I move it further outward to couch:
+{{:newsletters:img_8882.jpeg?0x200px}} {{:newsletters:img_8883.jpeg?0x200px}}
+In September:
+  * **Fixed channel estimation bug**: we were mistakenly treating all bits as flipping between reflective state and non-reflective halfway through, but only '0' bits do. This fix massively improved our distance estimates
+  * Subtract baseline carrier wave to improve detection range
+  * Faster RFID protocol implementation (integer arithmetic) so we are better at making the ACK deadline
+  * Retain the channel at each hop so we can still localize even if we didn't get a valid hop last round
+  * Persist stuff to disk, run headless for testing loop
+Hop alignment between RFID reader (in-band) and out-of-band localization helper radio looks pretty good:
+{{:newsletters:screenshot-2026-09-06-at-10.15.10-pm.png?450px}}
+Uncalibrated localization (we pick that first peak to probably be the line-of-sight path and tell us the estimated distance to the tag from that):
+{{:newsletters:screenshot-2026-09-06-at-11.42.20-pm.png?400px}}
+It's still a sort of coarse localization -- we need to use more fine phase data and also start triangulating from more than one OOB radio, I think. Would also be nice to increase speed. Will also do a Folk projector demo like what we used to have to make the detection accuracy more visceral.
 ==== Object tracking / DrawTalking ====
-=== DINOv2 ===
+=== DINOv2 (embedding model) ===
+Omar: This month, I've been trying object tracking using [[https://github.com/facebookresearch/dinov2|DINOv2]] (embedding model) to identify objects. Embedding-based tracking has a number of advantages compared to SAM2 tracking or raw contours ([[newsletters/2026-08#object-tracking|last month]]).
+It lets you do //re-identification//, where you take the object out of frame and bring it back seconds or minutes later. How do you maintain tracking without the visual continuity between frames? Embeddings use the inherent appearance of the object, which is also easy to serialize, instead of visual continuity like the SAM2 predictor.
+Embeddings are fast -- you can compute an embedding of a pose or viewport in like 10ms -- and are fast on almost any hardware, even my low-end home Intel N95 GPU (which can run DINOv2 in 100 or 200ms but takes 2 seconds to run SAM2).
+I started with pose registration that takes the embedding of the pose, and a 'evaluate' page that live-computes the embedding of whatever is on it and ranks which object it's most similar to:
+{{:newsletters:img_8614.jpeg?300px}}
+(Inelegant: pose registration step still requires SAM2, because SAM2 is how we cut the object out of the page/background so the embedding is only of the object itself. SAM2 can take a second or two on slower machines, as we saw last month, and you have to download the whole model. Did [[https://github.com/FolkComputer/folk/commit/209b19bbb7c0408d607488fe178f8d664d8ca9d0|fix some memory leak bugs in our use of it this month and simplified our API for this more limited use]]. Anyway, not as bad as having SAM2 in the inference phases, but would be nice -- for performance and simplicity -- to use some greenscreen technique instead of SAM2 in registration, IMO)
+Then I started doing localization, where we basically cut the viewport camera slice into cells and compute the embedding of each cell. You can get these really nice intepretable heatmaps of what the most 'cup-like' or 'army-toy-like' 'earbuds-case-like' (or whatever you want to find) regions of the viewport are.
+Here, the top heatmap shows bright army-toy-like regions, and the bottom heatmap shows bright cup-like regions (notice how they follow when I move the objects):
+{{:newsletters:img_8651.mp4}}
+More heatmaps:
+{{:newsletters:img_8740.jpeg?0x200px}} {{:newsletters:screenshot-2026-09-09-at-10.59.12-am.png?0x200px}}
+== DINOv2 with normalized cross-correlation ==
+You can find the hottest point in the heatmap to get a coarse location of the cup or army toy or whatever, but to get a fine-grained location, you need a refinement step that uses something else. We could use SAM2 again here, but that seems slow and may suffer from distortion again. Better idea (inspired by my old ScreenMatcher project): use [[https://en.wikipedia.org/wiki/Cross-correlation#Normalized_cross-correlation_(NCC)|normalized cross-correlation]] to directly template-match the local region of the coarse match against the pose. So you slide the pose over the coarse region and find the best alignment, so it could be pixel-accurate.
+Here's DINOv2 with NCC, which is remarkably fast and accurate for a radially symmetric object like this cup:
+{{:newsletters:img_8728.mp4?300px}}
+You can get pretty impressive fine centers:
+{{:newsletters:img_8752.jpeg?300px}}
+Only problem is that this doesn't account for rotation and different camera angles. Especially problematic to deal with asymmetrical objects.
+== Rotation tracking ==
+Solution: 1. record multiple physical poses with the object rotated differently, and 2. rotate the pose programmatically to get templates every 5 degrees so you can find the best fit (which also gives you an angle).
+[[https://github.com/FolkComputer/folk/tree/osnr/dinov2-rotation|Record several poses with known rotations, find the best cross-correlation, use that to give angle.]]
+It tracks the orientation of this army toy fairly well (except it sometimes flips by 180 degrees, which makes sense for this toy):
+{{:newsletters:img_9382.mp4?400px}}
+== Next steps ==
+The center (fine-grained localization) is still a bit unstable and sometimes makes mistakes or jumps a few centimeters, especially after lighting conditions have changed / if you have a concentrated lamp on one side. I improved it a little by weighting center pixels more compared to fringe pixels to try to get center aligned. Maybe finally introduce Kalman filter or something?
+I did a lot of speedup work: do FFT convolution, batch GPU operations, search nearby rotations first. Still a bit slow (100ms/frame?) but usable, but would be great to get to 60fps.
+=== DrawTalking integration ===
+We've started talking about the graphics pipeline that would be needed to draw many-point curves for sketches from DrawTalking. Some discussion about how to translate from DT procedural mindset to Folk reactive/data-oriented mindset.
+{{:newsletters:img_9140.jpeg?500px}}
+The graphics optimizations for the editor should help a lot here. Instead of reusing the existing line popeline, we probably want a custom pipeline with push constants for point A, point B, and transform, and then only the transform usually varies from frame to frame.
+I think this integration will be a good test of Folk performance for drawing complex objects and passing complex data through the DB (which is definitely something we want to support). Like a DT object with thousands or tens of thousands or hundreds of thousands of points.
+==== folk-cnc ====
+Omar: As part of the RTBW project, Brian, John Williams, and I have been mounting and bringing up the CNC system.
+{{:newsletters:img_8877.jpeg?0x200px}} {{:newsletters:img_8782.jpeg?0x200px}} {{:newsletters:img_8764.jpeg?0x200px}}
+Mounted steel strut to ceiling, hung 2 cameras and a projector (cantilevered on wood block).
+{{:newsletters:img_9595.jpeg?0x200px}} {{:newsletters:img_9596.jpeg?0x200px}} {{:newsletters:img_9594.jpeg?0x200px}}
+It took a while to figure out how to mount the projector, because it projects in a triangle upward from its bottom plate, and our strut wasn't really in the right spot to mount it widescreen. We tried complicated rigs like this to move the projector vertically away from the strut:
+{{:newsletters:img_9318.jpeg?0x200px}} {{:newsletters:img_9319.jpeg?0x200px}}
+but felt uncomfortable mounting them. Eventually just went with turning it 90 degrees and doing the cantilever for now. As you can see, it's still not perfectly aligned with the left edge of the cut area on the bed, so we'll probably tweak it soon:
+{{:newsletters:img_9381.mp4?300px}}
+Weirdly, both of our Intel N95-based mini PCs seemed to have fried SSDs when I tried them. I ran into this issue on my home system after about a year, too. Replaced the SSD and put a 3D-printed spacer in to make the new SSD the right length:
+{{:newsletters:img_9586.jpeg?0x100px}} {{:newsletters:img_9587.jpeg?0x100px}}
+Verified that the CNC machine itself works to cut some wood for the mount:
+{{:newsletters:img_8977.mp4?200px}}
+=== folk-convivial ===
+We also remounted folk-convivial from the ceiling, getting rid of the C-stand that was always at risk of tipping over (and took up floor space, and made the table impossible to move without toppling the stand):
+{{:newsletters:img_8876.jpeg?0x150px}} {{:newsletters:img_8975.jpeg?0x150px}}
+Good alignment of projector and camera now:
+{{:newsletters:img_9090.jpeg?300px}}
+=== Telephoto camera ===
+For the RTBW project, I've been mounting [[https://www.arducam.com/arducam-high-quality-complete-usb-camera-bundle-12mp-1-2-3-inch-imx477-camera-module-with-2-8-12-mm-varifocal-lens-c20280m12-metal-enclosure-tripod-and-usb-cable-b0288.html|a varifocal-lens camera]] 15-20ft away (alongside the 4K projector) to try to target a 6ft x 10ft CNC bed.
+I can configure the lens to the right field of view, but can't mechanically get it to the right place to focus it while keeping it all attached. Need to buy a ring adapter to push the C-mount lens further out, I think? Haven't fully figured out.
+{{:newsletters:img_9069.jpeg?200px}} {{:newsletters:img_9065.mp4?200px}}
 ==== [Expect] and rewindability ====
 Mason: Omar and I (among others) have been continuing to discuss what an [Expect] command would look like. The idea is you could use [Expect] like:
@@ Line 111: / Line 265: @@
 Oh, one other detail, specific to rewinding: there needs to be some way to undo every side effect caused by the scope(s) during rewind. I'm thinking if we have an interpreter log of every side effect that happens, you could run that log in reverse when rewinding, up to the point of the Expect. This would avoid any side effects from being missed. The cool thing about this log is when running in reverse it might even go back up stack frames, so your Expect could be five function calls deep, and the log would just run all the way through the stack back to the Expect.
 </details>
 ==== Open Sound Control API ====
-Paul: I am in the process of building out a Folk music toolkit, and one of the core components needed for this system is the ability to send OSC messages from my Folk machine to arbitrary OSC servers to control synthesizers. I managed to implement simple OSC send functionality which allows Folk users to send OSC messages using the native Folk `Notify:` syntax. A Folk environment first registers the address of the OSC server using `Claim the NetAddr is <ip>:<port>` then once an endpoint is registered Folk programs can send OSC messages using the syntax `Notify: OSC at /path/ with args /values/`.
+Paul: I am in the process of building out a Folk music toolkit, and one of the core components needed for this system is the ability to send OSC messages from my Folk machine to arbitrary OSC servers to control synthesizers. I managed to implement simple OSC send functionality which allows Folk users to send OSC messages using the native Folk ''Notify:'' syntax. A Folk environment first registers the address of the OSC server using ''Claim the NetAddr is <ip>:<port>'' then once an endpoint is registered Folk programs can send OSC messages using the syntax ''Notify: OSC at /path/ with args /values/''.
 Next I will add support for OSC bundles, which will allow us to write proper tidalcycles-esque pattern languages native in Folk!
@@ Line 125: / Line 280: @@
 === Interface Studies podcast ===
+Omar and Daniel showcased Folk on the first episode of "Conversations," Interface Studies' new podcast. We covered tangible interaction, natural language programming, and highlighted community and collaboration.
+{{youtube>uXHHnps4tU8}}
+Andrés: I've watched [[https://www.youtube.com/@interfacestudies|nearly every video Saleh's videos]], have started sending them to my interaction design students, and (if you watch a lot of educational/edutainment YouTube you'll know this is a high compliment) I think he's the [[https://www.3blue1brown.com/|Three Blue One Brown]] of interface design. I think [[https://www.youtube.com/watch?v=uXHHnps4tU8&lc=Ugwy-XkrsTxLt0vF6Lt4AaABAg|one particularly nice comment]] on the podcast Omar and Daniel were on sums up why I'm so impressed with the channel:
+{{newsletters:screenshot-2026-10-07-at-16.48.38.png?400px}}
+=== SVA system ===
+FIXME: Andrés
+=== Recurse Center and SVA visitors ===
+Tracey Le has been bringing in smaller groups from the current Recurse Center batches who want to do some programming in our systems at 87R (and maybe bring knowledge and practices back to the RC system).
+{{:newsletters:img_8647.jpeg?200px}}
+== Friday, September 4 ==
+{{:newsletters:img_8656.jpeg?0x150px}}
+== Friday, September 18 ==
+{{:newsletters:img_9125.jpeg?0x150px}} {{:newsletters:img_9126.jpeg?0x150px}} {{:newsletters:img_9131.jpeg?0x150px}} {{:newsletters:img_9134.jpeg?0x150px}}
+Tracey has been showing the raw contour + color-wheel stuff to a lot of visitors:
+{{:newsletters:screenshot-2026-09-18-at-12.31.57-pm.png}}
+== Friday, September 25 ==
+{{:newsletters:img_9457.jpeg?0x150px}} {{:newsletters:img_9461.jpeg?0x150px}} {{:newsletters:img_9460.jpeg?0x150px}} {{:newsletters:img_9459.jpeg?0x150px}} {{:newsletters:img_9471.jpeg?0x150px}}
+We messed with the object-tracking label program to label with real-world centimeter positions:
+{{:newsletters:img_9467.mp4?350px}}
+and to make big labels:
+{{:newsletters:img_9465.mp4?250px}}
 === Open house ===
+Had a pretty big open house (we've been having big open houses for the last few months). Tracey brought a bunch of people from Recurse Center; Alan Laidlaw stopped in for the first time in a while; Paul came up from Philadelphia.
+{{:newsletters:img_9599.jpeg?0x150px}} {{:newsletters:img_9619.jpeg?0x150px}} {{:newsletters:img_9618.jpeg?0x150px}} {{:newsletters:img_9616.jpeg?0x150px}} {{:newsletters:img_9614.jpeg?0x150px}} {{:newsletters:img_9610.jpeg?0x150px}} {{:newsletters:img_9609.jpeg?0x150px}} {{:newsletters:img_9607.mp4?84px}} {{:newsletters:img_9600.jpeg?0x150px}}
+Nice to have: New fast editor + lack of multi-editor bug when pointed at multiple things.
+Ran into bug with editor still crashing/multi-appearing a couple of times:
+{{:newsletters:img_9615.jpeg?200px}}
+=== Utah Meetup ===
+We had our first Utah meetup in Lehi, UT. We chatted about the reactive database and introduced the system to our host Michael.
+We plan to do this monthly. If you are interested, reach out on discord or email <[email protected]>
+{{:newsletters:september-utah-meetup.webp?300px}}
 ===== What we'll be up to in October =====
   * Our next [[https://luma.com/wa18i4hg|Folk open house will be on Wednesday, September 30]], at our studio in Williamsburg.
-  * Omar:
+  * Omar: Ship RFID tracker; do projection demo of distance to confirm visually that it works
-  * Andrés:
+  * Omar: Build automatic folk-cnc driver and preview application
+  * Omar: Calibrate RTBW CNC system with 2 cameras
+  * Omar: More work to make editor usable on table
+  * Omar: Filter DINOv2+NCC+rotation tracker to be more stable; integrate with DrawTalking strokes
+  * Andrés: Lamp! I'm working on a gadget-inside-a-lamp project using a few [[https://catalog.lightingspecialties.com/item/personal/luxo-ls-series/k110660001|Luxo task light lamps]] that were dinged up and unused at the SVA Interaction Design program. Will have more to show when I get more time in the studio next month!
 ===== Links we've enjoyed =====
 ==== Andrés ====
-  *
+  * FIXME
 ==== Omar ====
   * http://batteries.parametrek.com/index.html?mah=1514,_&voltage=7.4V,8.4V