Differences

This shows you the differences between two versions of the page.

--- newsletters:2026-10 [2026/11/11 04:47] – osnr
+++ newsletters:2026-10 [2026/11/11 17:30] (current) – osnr
@@ Line 2: / Line 2: @@
 {{htmlmetatags>metatag-og:title=(Folk Computer October 2026 newsletter)
-metatag-media-og:image=()
+metatag-media-og:image=(newsletters:img_9820.jpeg)
-metatag-og:description=()}}
+metatag-og:description=(DrawTalking integration; stereo camera calibration and CNC preview; 4K camera upgrade; MIDI controllers and laser reflectors; editor autocomplete; system reliability fixes; Zicl runs core libraries)}}
 Our next Folk open house will be in the afternoon on **[[https://luma.com/l9smgdzl|Sunday, November 29]]**, at our studio in Williamsburg, Brooklyn.
@@ Line 70: / Line 70: @@
 Omar: I was chatting with [[https://ikeadrift.com/|Ben Zweig]] about object detection models. He was excited about [[https://github.com/roboflow/rf-detr|RF-DETR]], so I decided to spend a few minutes trying it.
-(If you know [[https://en.wikipedia.org/wiki/You_Only_Look_Once|YOLO]] or models like that, RF-DETR seems like sort of their modern descendant, plus segmentation. So it has a few dozen known object classes out of the box, it's pretty fast, etc. Not that similar to the open-ended embedding and SAM2 and contour tracing stuff we've tried for object detection under the DrawTalking project, which I realized after this experiment.)
+(If you know [[https://en.wikipedia.org/wiki/You_Only_Look_Once|YOLO]] or models like that, RF-DETR seems like sort of their modern descendant, plus segmentation. So it has a few dozen known object classes out of the box, it's pretty fast, etc.)
 Tracking a keyboard and my hands and the cup (notice how hand ['person'] tracking works by far the best, and how the model is generally very fast, probably 30fps-ish):
@@ Line 77: / Line 77: @@
 Here're the [[https://gist.github.com/osnr/bafa9a014b5af0d78216839cfa02dcee|RF-DETR engine]] and [[https://gist.github.com/osnr/ec701b29e165d3848da8ce7183cbab09|RF-DETR on-viewport displayer]] programs. (The locking bit in there is ugly, and there are still issues with slow boot time / lack of persistence, so if you cover it up it's annoying to wait for the model to re-load, but it generally works well.)
+Realized after this test why I hadn't looked into RF-DETR before. It's not that similar to the open-ended embedding and SAM2 and contour tracing stuff we've tried for object detection under the DrawTalking project. It's not super useful (at least out of the box), IMO, because 1. it can't detect most of the things we actually want to track (it's not well-trained on the from-above view, and the cup is really small in the sliced image, and it only knows its fixed object classes anyway), and 2. if you just want to track hands, which it is good at, you can probably do better/faster with a dedicated model for that anyway.
 (also, this detector + switching folk-hex to 4K led me to add [[https://github.com/FolkComputer/folk/commit/00fb46336a4b98db9b6abafe4accea87b408fd47|configurable camera slice width]] so the 4K image doesn't kill performance)
@@ Line 110: / Line 112: @@
     * [[https://github.com/FolkComputer/folk/commit/c8035cdc7b4edd2fb28a8125d54231548afbfeae|Destructor race/UAF fix]], where we now establish destructors before publishing/reacting statements, so that there's no risk of a subgraph spawning too early before destructor inheritance in the parent, parent going away, & then subgraph (with empty destructor set) accessing mistakenly-freed data that should've been destructor-guarded
     * [[https://github.com/FolkComputer/folk/commit/f48ce703b99337c3c6f0ba2efaaee7d9aad8771e|Handle JPEG decode failures better]], reject those frames if possible, don't crash
-    * [[https://github.com/FolkComputer/folk/commit/01325087f6eca47066eb1ec08414fbc14bcc353d|Fixed a race in Collect]] where an older query could complete after newer results come in and overwrite them with stale old results: fixes were to increment sequence number before the query isntead of after & advance stored sequence number even if set is unchanged
+    * [[https://github.com/FolkComputer/folk/commit/01325087f6eca47066eb1ec08414fbc14bcc353d|Fixed a race in Collect]] where an older query could complete after newer results come in and overwrite them with stale old results: fixes were to increment sequence number before the query instead of after & advance stored sequence number even if set is unchanged
     * [[https://github.com/FolkComputer/folk/commit/f9b8bd6a7bedf85f0759638d40234f61275cfa2f|Fixed a Firefox (and other browsers?) memory leak with image objects in /camera]]
     * (Motivated by MIDI controller pushing it harder) [[https://github.com/FolkComputer/folk/commit/11121f5ee7cfd6ad762e0cb78824dd21134047d1|Keep Tidal FIFO open even on worker cancellation]], which should keep music from breaking until restart which used to happen a lot
@@ Line 123: / Line 125: @@
 Omar's been setting up the RTBW CNC Folk system in Vancouver.
-This month, 2 major fixes to get that system running:
+This month, a few major fixes to get that system running:
-  * Figure out how to focus long-throw 4K cameras so we can actually calibrate it (projector and cameras are mounted 15-20ft away from bed)
+  * Figure out how to focus long-throw 4K cameras so we can actually calibrate it (projector and cameras are mounted 15-20ft away from the 6ft x 10ft CNC bed)
+    * {{:newsletters:img_9952.jpeg?300px}} {{:newsletters:screenshot-2026-10-13-at-3.51.43-pm.png?300px}}
     * Needed [[https://www.amazon.com/dp/B09B6Y29QC?ref=ppx_yo2ov_dt_b_fed_asin_title|spacer]] [[https://www.amazon.com/dp/B0H8295RZW?ref=ppx_yo2ov_dt_b_fed_asin_title|rings]] to get the right focal distance:
       * {{:newsletters:img_9949.jpeg?150px}}
@@ Line 133: / Line 136: @@
   * Implement CNC preview demo
-Omar made a big board and successfully calibrated a giant 6ft x 10ft Folk instance (normal-sized board for size comparison):
+=== Calibrating big system ===
+Made a small board at first, which did work, but wasn't very useful:
+{{:newsletters:screenshot-2026-10-14-at-1.30.45-pm.png?300px}}
+Finally made a big board:
 {{:newsletters:img_9993.mp4?220px}}
+And successfully calibrated the giant Folk instance (normal-sized board for size comparison):
 {{newsletters:978526_B673C685-4EF3-4926-A991-4FE42382166A.jpg?0x250px}}
 {{newsletters:049697_image.png?0x250px}}
-{{newsletters:907412_D792DBED-4567-4F0B-865D-CAFA81D45016-666-0000258A3FFCA172.png?250px}}
+{{newsletters:907412_D792DBED-4567-4F0B-865D-CAFA81D45016-666-0000258A3FFCA172.png?0x250px}}
-mm tags:
-{{:newsletters:img_0003.jpeg?200px}}
 Surprising and impressive that normal-size tag detection works here (4K camera for 6ft x 10ft bed is not a lot of dpi!), although we probably won't use it much on this system:
 {{:newsletters:img_0009.jpeg?0x200px}} {{:newsletters:img_0006.jpeg?0x200px}}
+=== SAM2 workpiece detection ===
 Next, using SAM2 and the stereo cameras' calibrations (I calibrated the 2 cameras separately, then manually tweaked the hold file so all the statements could be present together) to automatically find and projection-map workpieces on the CNC bed:
@@ Line 153: / Line 162: @@
 {{:newsletters:img_0012.jpeg?0x300px}} {{:newsletters:img_0014.jpeg?0x300px}} {{:newsletters:img_0017.jpeg?0x300px}} {{:newsletters:img_0026.mp4}}
-CNC preview demo (runs autonomously right now):
+{{:newsletters:screenshot-2026-10-15-at-4.14.55-pm.png?550px}}
+We use 2 cameras so that we can find the physical (in meters) locations of each workpiece (if you have the 4 corners in both camera spaces, you can triangulate). Once we have the physical locations, we can (at least roughly) projection-map onto the workpiece surfaces by just using extrinsics to map the location into projector space. (That lets us roughly project where the CNC machine will cut, in theory.)
+I think that's probably not accurate enough if you want sub-centimeter accuracy, so one plan I have is to project AprilTags onto that rough area and use those to develop a homography for each workpiece. We did something similar (but manual) in the old folk-cnc. You can see it run here:
+{{:newsletters:img_0031-cropped.mp4?450px}}
+=== CNC preview ===
+New CNC preview demo (runs autonomously right now) (different code from the [[https://github.com/FolkComputer/folk-cnc|old demo]], but [[https://folk.computer/newsletters/2024-05#cnc|related ideas]]):
 {{:newsletters:img_0048.jpeg?300px}}
+It doesn't account for bed/workpiece heights/locations yet, just assumes you're cutting straight on the bed plane, but wanted to test the pipeline of loading from the CNC controller, parsing gcode, etc.
 Powered by a AutoHotKey script that reports (using curl to POST to Folk / endpoint) from controls on the CNC controller PC to Folk statements.
@@ Line 252: / Line 273: @@
 </code>
 </details>
+It claims out the CNC head location and gcode, which is (in theory) enough information to render the preview of where exactly it'll cut:
+{{:newsletters:screenshot-2026-10-15-at-5.52.26-pm.png}}
 ==== Multi-camera calibration ====
@@ Line 260: / Line 285: @@
     * {{:newsletters:1e1c056e-d8d2-4bc3-b0d8-305c8465a5c0-54556-000001675ece95ea.png?300px}}
     *  If "Detect AprilTags" is not checked for a given camera, you won't get tag detection on that camera. (We want it to be opt-in since many systems will have secondary cameras for laser detection, workpiece-finding, depth estimation, documents, etc.)
-  * 4K cameras get high decimation (we should probably make this configurable) to make performance better
+  * 4K cameras get [[https://github.com/FolkComputer/folk/commit/e013608548c7fa08e4791371f810ed3d98378052|high decimation on AprilTag detector]] (we should probably make this configurable) to make performance better
+    * Entire-frame detections go from 15fps to 60fps (60ms per frame to 10ms per frame)
   * Moderate calibration bug fixes: we weight pixel errors properly instead of squaring, we increase maxiters from 200 to 400, and we normalize homographies so one pose can't throw rest off. This should make calibration behave better and more reliably (helped on rtbw cnc folk)
@@ Line 277: / Line 303: @@
 {{:newsletters:img_0219.jpeg?0x300px}} {{:newsletters:img_0220.jpeg?0x300px}} {{:newsletters:img_0249.jpeg?0x300px}}
-The new cameras have 4K60, not just 1080p60; have tried running at 4K and works most of the time. Excited to do more [[newsletters/2026-06#fields-local-feedback-calibrationnudging-experiment|fields OCR]] and fine-grained detection stuff (snapping to text in books, reading drawings, etc).
+The new cameras have 4K60, not just 1080p60; have tried running at 4K and works most of the time, not even noticeably slower (thanks to incremental tag detection). (When running a detector like RF-DETR, you start getting blinking-out, because the 4K does put a lot more load on the PC, and the scheduler starts missing stuff. One of the biggest things is JPEG decompression, which takes 10+ ms per 4K frame as opposed to 3ish ms per 1080p frame.)
+Excited to do more [[newsletters/2026-06#fields-local-feedback-calibrationnudging-experiment|fields OCR]] and fine-grained detection stuff (snapping to text in books, reading drawings, etc).
@@ Line 290: / Line 318: @@
        * {{newsletters:img_4929.mp4?250px}}
     * This video shows Audrey getting transforms and scale factors translating between DT and Folk in realtime:
-       * {{newsletters:462166_clip.mp4?250px}}
+       * {{:newsletters:462166_clip-sdr.mp4?250px}}
-Example interface point where a JS module in DrawTalking claims a DT object into Folk:
+How the interface point looks where a JS module in DrawTalking claims each DT object into Folk:
 {{:newsletters:img_9670.jpeg?300px}}
-Early prototype that just did quads:
+Early prototype that just drew quad in Folk for each DrawTalking entity:
-{{:newsletters:img_9677.jpeg?400px}} {{:newsletters:img_9679.mp4?400px}}
+{{:newsletters:img_9677.jpeg?250px}} {{:newsletters:img_9679.mp4?250px}}
 On folk-hex:
@@ Line 319: / Line 347: @@
 {{:newsletters:img_0462.jpeg?300px}}
-{{:newsletters:orbital-best.mp4?400px}}
+Dog following cup:
+{{:newsletters:img_5162.mp4?400px}}
+Orbital demo:
+{{:newsletters:orbital-best.mp4?400px}}
 === Object tracking ===
-Not much TODO
+Not much new on the actual object tracking front -- I did make a version of object registration that no longer depends on SAM2 to segment the poses out from background; instead, it prints this known purple backdrop and tries to chromakey it out. Doesn't work perfectly yet, though.
 {{:newsletters:img_0443.jpeg?300px}}
 ==== Zicl update ====
-Mason: this month has been pretty slow for Zicl, since I haven't had a ton of time to dedicate to it. But! Something exciting happened. Enough of the the system is functional that I was able to get all the core libraries to compile and evaluate, and so I was able to benchmark it relative to Folk's current interpreter. And...
+Mason: this month has been pretty slow for Zicl, since I haven't had a ton of time to dedicate to it. But! Something exciting happened. Enough of the system is functional that I was able to get all the core libraries to compile and evaluate, and so I was able to benchmark it relative to Folk's current interpreter. And...
 {{newsletters:screenshot_20261103_232737.png?600|}}