Differences

This shows you the differences between two versions of the page.

--- newsletters:2026-02 [2026/03/10 02:01] – [Outreach] osnr
+++ newsletters:2026-02 [2026/03/11 19:14] (current) – osnr
@@ Line 2: / Line 2: @@
 {{htmlmetatags>metatag-og:title=(Folk Computer February 2026 newsletter)
-metatag-media-og:image=(newsletters:IMAGE_FILENAME_HERE.jpg)
+metatag-media-og:image=(newsletters:pasted:20260311-191224.jpeg)
-metatag-og:description=(Brief description of key updates this month)
+metatag-og:description=(Python FFI, image segmentation, text recognition; sound wheel; stdout and stderr redirection; Folk February; Hamish talk; post-calibration refinement)
 }}
@@ Line 69: / Line 69: @@
 At first, I was trying dup2 to replace stdout and stderr with the program-local files right before we eval any When body on a thread. But those file descriptors 1 and 2 are process-wide, so multiple threads can't have different stdout and stderr at the same time! so we'd get weird cross-printing into the wrong streams when programs were executing in parallel.
-We need thread-local stdout and stderr. We could hook ''puts'' or the aio system in Tcl to use the right local file descriptors. But... we also want C FFI and C library output to come out on the program-local stdout and stderr, and those don't go through Tcl at all.
+We needed thread-local stdout and stderr. We could hook ''puts'' or the aio system in Tcl to use the thread-local file descriptors. But... we also want C FFI and C library output to come out on the program-local stdout and stderr, and those don't go through Tcl at all, so the hooks wouldn't apply there.
 So, a really weird hack: [[https://github.com/FolkComputer/folk/blob/d08cbc2095a7fb2f86785b42efed664c932fc73f/output-redirection.c|use library interposition]] (LD_PRELOAD on Linux, dyld interposing on macOS) to replace all the file write operations so that if the write is to stdout or stderr, we look up the latest thread-local file descriptor in a thread-local variable and use that. Works fine on both macOS and Linux so far.
@@ Line 78: / Line 78: @@
   * we occasionally want user code to be able to print to real stdout and stderr (for creating the terminal UI, for example) -- issue for a while was how to reopen real stdout and stderr on different fds so you can write to them separately without being interposed. Can't reopen /dev/fd/1 and /dev/fd/2 as you might think, because it's a socket on systemd, so it broke Folk in systemd. And it's not correct after initial boot if you do it on each interpreter (since new interpreters may come online later). [[https://github.com/FolkComputer/folk/commit/41dbc531bc9c4f4443dbdb1c136e1ac2a6a81a20|Have to dup real stdout and stderr at C level once, globally.]]
   * have to [[https://github.com/FolkComputer/folk/commit/7cc101bdae603472d12a2947b4a1883d1eb2731e|specifically apply output redirection]] to Tcl ''exec ... &'' so that the subprocess gets plugged into the program-local stdout and stderr instead of the global ones (this is used for stuff like running Python and running curl in background)
+  * Can't open the .stdout and .stderr files per-thread on demand and store them globally in each Tcl interpreter, because then you have 1000+ open file descriptors (each program often gets its stdout/stderr reopened across multiple threads), which strains the default OS settings.
+    * {{.:pasted:20260310-022915.png?300px}}
+    * solution: [[https://github.com/FolkComputer/folk/commit/b52b50f19fd62fa91dc3ccbb965a2715c06883da|do the fd tracking at C level]] so you have one fd per file, so you go from ~1000 open fds to ~300 open fds
 === Motivation ===
@@ Line 152: / Line 155: @@
 {{.:pasted:20260305-220931.png?400px}}
+{{newsletters:screenshot-2026-02-11-at-5.01.50-pm.png?400px}}
 but ended up replacing it with plain Unix domain socket to save on a dependency (we weren't using most of it, and it has transitive dependencies).
-(Part of why I was excited about both msgpack and zeromq is there seemed to be reasonable [[https://pyzmq.readthedocs.io/en/latest/howto/serialization.html|Python]] and [[https://github.com/camgunz/cmp|C]] bindings for both that were used to transmit images and stuff.)
+(Part of why I was excited about both msgpack and zeromq is there seemed to be reasonable [[https://pyzmq.readthedocs.io/en/latest/howto/serialization.html|Python]] and [[https://github.com/camgunz/cmp|C]] bindings for both that were used to transmit [[https://github.com/jeffbass/imagezmq|images]] and stuff.)
 A weird thing is that zeromq did handle threading on its own, and now we have to use Python threading on the Python side.
@@ Line 205: / Line 210: @@
 and with this in place, a program can just When to get the detector function and call it on a camera frame / camera slice Image and get back a list of text boxes. (The FFI will automatically convert any Python list/dict/etc into a Tcl object by using JSON serialization, and vice versa, in arg or return types.)
+{{newsletters:img_2224.jpeg?250px}}
 (I do wonder if this binding even needs to be built into Folk or if it should just be part of the user program on the table, it's so short... and it would've been nice, at the last open house, if people could just see this caller implementation on the table, rather than having to take our word for it.)
@@ Line 224: / Line 231: @@
 {{newsletters:img_1853.mp4?450px}}
+=== Text detection + handwriting recognition ===
+[[https://github.com/FolkComputer/folk/blob/c4d80ff2474fad6de32c327b1a3c6b8ed7b3fde6/builtin-programs/recognition/trocr.folk|Here's the binding to the TrOCR text recognizer.]]
+TrOCR binding side-by-side with the CRAFT text detector binding:
+{{.:pasted:20260310-024607.png?600px}}
+This doesn't work great yet -- I wonder if we need higher resolution or what -- but here it's finding the slice where "text" is handwritten and then OCRing it:
+{{newsletters:img_1874.mp4?300px}}
+=== Image segmentation using SAM2 ===
+I don't have a good photo or video of this yet (needs to be set up in the new space and have more interesting demo built around it, maybe whole table instead of camera slice), but [[https://github.com/FolkComputer/folk/blob/3efc3b2687050eb8b3560134e009a28654be6b84/builtin-programs/recognition/sam2.folk|here's the binding]] for [[https://github.com/facebookresearch/sam2|SAM2]].
+Would be useful for pointing to an arbitrary object and tracking or projection mapping it.
 ==== ''Expect'' experiment ====
@@ Line 254: / Line 278: @@
 {{.:pasted:20260307-012646.png?500px}}
-Have been picking back up what used to be ''table-refine.folk'' -- a post-calibration refinement step, basically a new kind of calibration that is interactive. It still requires that you do the traditional calibration step first, because it needs a guess that's good enough (lets it see enough tags) that it can refine it.
+Have been [[https://github.com/FolkComputer/folk/tree/osnr/interactively-refine-calibration|picking back up]] what used to be ''table-refine.folk'' and is now [[https://github.com/FolkComputer/folk/blob/200f1c8e9197a09f7d063fe13420f95f1b50b760/builtin-programs/calibrate/interactively-refine.folk|''interactively-refine.folk'']] -- a post-calibration refinement step, basically a new kind of calibration that is interactive or 'online'. You set the calibration board in a pose and the system projects and adjusts its full end-to-end calibration for a while until it can nail that pose:
-TODO: originally, problem was I wasn't giving enough residuals, so the system was underdetermined?
+{{newsletters:img_2220.mp4?400px}}
+This uses the Levenberg-Marquardt optimization that we also use in the traditional calibration, but in the middle of the evaluation function, we project with the current candidate calibration and fully loop that through the camera and tag detector, so we get a 'real' end-to-end error.
+It still requires that you do the traditional calibration step first, because it needs a guess that's good enough (lets it already see enough tags) that it can refine it.
+=== Various issues ===
+Originally, problem was I typoed and wasn't giving enough residuals (need more residuals than parameters), so the system was underdetermined and would fail:
 {{.:pasted:20260307-012756.png?400px}}
-TODO: for a while, problem was LM step size wasn't large enough, so the changes to estimate gradient were below the noise floor, so it wouldn't improve. you can see that it improves a lot now:
+For a while, problem was LM step size wasn't large enough (it wasn't tuning the parameters by a big enough offset, like 0.00001 instead of 0.001), so the changes to estimate gradient were below the noise floor of tag detection, so it couldn't estimate the gradient and know which direction to improve things.
+You can see that the refinement improves a lot now (orignorm -> bestnorm):
 {{.:pasted:20260307-012844.png?500px}}
@@ Line 266: / Line 300: @@
 ==== Hamish Todd's talk ====
+[[https://hamishtodd1.github.io|Hamish Todd]] gave a [[https://www.youtube.com/watch?v=8BKqlrPm3mU|talk]] in our Discord about 3D interaction and some perspective problems with tabletop projection that he's been exploring. We had a good discussion about what's currently possible or interesting to do with the system.
+{{newsletters:screenshot-2026-02-04-at-11.06.48-am.png?300px}}
+{{youtube>8BKqlrPm3mU?}}
 ==== Outreach ====
@@ Line 273: / Line 315: @@
 === Open house ===
-  * We hosted our last monthly open house at our Hex House studio on Thursday, February 26th, a few of Andrés' students from SVA stopped by and we had fun demoing the animation program as well as segmentation that Omar's been working on and a sound-effect wheel that Andrés has been making:
+  * We hosted our last monthly open house at our Hex House studio on Thursday, February 26th -- a few of Andrés' students from SVA stopped by and we had fun demoing the animation program, as well as segmentation that Omar's been working on, and a sound-effect wheel that Andrés has been making:
-    * {{.:pasted:20260305-190452.jpeg?400px}} {{.:pasted:20260305-190510.jpeg?400px}}
+    * {{.:pasted:20260305-190510.jpeg?0x200px}} {{.:pasted:20260305-190541.jpeg?0x200px}}
-    * {{.:pasted:20260305-190541.jpeg?400px}}
+    * {{.:pasted:20260305-190531.jpeg?0x300px}} {{newsletters:img_2222.jpeg?0x300px}} {{newsletters:img_2226.mp4?200px}}
-    * {{.:pasted:20260305-190531.jpeg?300px}}
+    * {{newsletters:img_2228.jpeg?0x300px}} {{newsletters:img_2232.jpeg?0x300px}}{{newsletters:img_2233.jpeg?0x300px}} {{newsletters:img_2237.jpeg?0x300px}}
+== Atomically bug ==
+Our friend [[https://x.com/perlinwarp|Peter Walkington]] found a bug with Atomically that shows up after a little while on this program:
+{{newsletters:img_2242.jpeg?400px}}
+It should alternate between :O and :), but it ends up in this superposition where both :O and :) are visible at the same time:
+{{newsletters:img_2241.jpeg?0x400px}}
+It looks like we're leaking versions, like we're 20,000 versions behind on item 5 here (1557's clock time):
+{{newsletters:screenshot-2026-02-27-at-4.48.51-pm.png?500px}}
+We saw similar issues with Atomically at the party end of last year -- it's nice to have a concrete example and a sense of what the issue is. Need to debug.
 ===== What we'll be up to in March =====