Differences

This shows you the differences between two versions of the page.

--- newsletters:2024-11 [2024/12/01 01:46] – [New Folk evaluator] osnr
+++ newsletters:2024-11 [2024/12/01 04:09] (current) – osnr
@@ Line 1: / Line 1: @@
-====== November 2024 newsletter (WIP) ======
+====== November 2024 newsletter ======
+(Our next Folk open house is [[https://partiful.com/e/eZXJm5gn0rxglOGi5Jsa|at noon on Sunday, December 15]], at our studio in East Williamsburg, Brooklyn.)
-(Our next Folk open house is at noon on Sunday, December 15, at our studio in East Williamsburg, Brooklyn.)
 ===== What we've been up to =====
 ==== Small system improvements ====
-  * TODO: Print notice when you hit Print, both on web and tabletop editor
+  * Andrés added:
-  * TODO: New /programs design with categories; much easier to read
+    * Print notice when you hit Print, both on web and tabletop editor
+      * {{newsletters:img_8911-medium.jpeg?300px}}
+    * New ''/programs'' design with categories; much easier to read:
+      * {{:newsletters:pasted:20241201-025825.png?300px}}
 ==== New Folk evaluator ====
-Omar: I've spent most of this month working on the new evaluator (folk2). I've made a huge amount of progress; it honestly feels like I spent months stuck on blinking/flickering/ghosting bugs and then solved them all last week.
+Omar: I've spent most of this month working on the new evaluator (folk2).
+I've made a huge amount of progress; it honestly feels like I spent months stuck on blinking/flickering/ghosting bugs and then solved them all last week.
 (Remember, the goal of folk2 is to have 60fps under bigger workloads, so we can use multiple tabletop editors at once & do other demanding stuff natively in the system. I think it's just important that the system feel really good and smooth, anyway. Need to be running programs in parallel, avoiding queueing, also just general evaluator optimization. Another goal is to make the system hard to crash or freeze just because one program misbehaves, as in any other operating system.)
@@ Line 17: / Line 23: @@
 There's still a lot of blinking under the new evaluator, but no ghosting, and I think the blinking all now emerges from genuine failures to meet timing (the label blinks out because the program region wasn't regenerated in time for the current frame).
-=== What were the bugs? ===
+Notice how the camera hits 60fps but the full-frame detection is only 20-30fps (this is using a 1080p image):
-TODO
+{{:newsletters:pasted:20241201-040919.png?550px}}
-=== Other improvements ===
+=== The bugfixes and improvements ===
-  * ''When (serially)''
+  * ''When (serially)'' construct to replace infinite loops (statements retract properly)
-  * tail trace
+  * Tail trace so you can trace latest stuff going on
-  * atomic gen+rc+alive
+  * Replaced the broken refcount scheme with gen+rc+alive in one atomic variable, which makes a lot more sense and seems to work
-  * timestamp work items, figure out elapsed, schedule based on that
+  * Now timestamping work items, figure out elapsed time, scheduling based on that
-  * fully separate match/statement 'removal' (which removes descendants) from memory freeing (which happens when rc = 0 and not alive)
+  * Fully separated match/statement 'removal' (which removes descendants) from memory freeing (which happens when rc = 0 and not alive)
-  * fix locking of destructor list
+  * Fixed locking of destructor list
     * {{:newsletters:pasted:20241201-014313.jpeg?250px}}
-  * fix locking of child match/statement lists
+  * Fixed locking of child match/statement lists
-  * steal-half
+  * Steal-half now actually works (fixed overflow, fixed stealing 100% instead of half)
-  * fix lsorts in infinite loops (not using now that we have serial When though)
+  * Fixed lsorts in infinite loops (not using now that we have serial When though)
-  * show each statement's match children & statement grandchildren in web UI (makes debugging much nicer!)
+  * Now showing each statement's match children & statement grandchildren in web UI (makes debugging much nicer!)
     * {{newsletters:image101912312.png?400px}}
-  * ports from folk1:
+  * Ports from folk1:
     * dashed-line (want it for gadget marching ants)
     * camera-rpi throughput improvement using YUV420 format
     * implemented clock time (also want it for marching ants, and any kind of animation really)
       * now runs on an independent clock from the frames! every 2ms
-    *
+    * hack for 'descriptor indexing' on Pi GPU so images & text work
+    * incremental tag detection to be able to hit 60fps
@@ Line 47: / Line 54: @@
 == Performance profiling ==
-TODO: Performance profiling remaining.
+Performance (CPU, stack) profiling is most of what I'm up to now, trying to eliminate the blinking.
-I'm surprised at how much of the flamegraphs are just threading stuff, not CPU. Maybe I'm misreading and need different graphs.
+I've already fixed a lot of weird issues using flamegraph profiling and made it noticeably smoother. Like replacing rand with rand_r to fix this:
-== Heap profiling ==
+{{:newsletters:pasted:20241201-034841.png?400px}}
-There are still persistent and pretty bad memory leaks on folk2.
+I'm surprised at how much of the flamegraphs are just threading stuff, not Tcl-bound. Maybe I'm misreading and need different graphs (the mutex time is just time when it's scheduled off the CPU, not wasted time). Here is an average worker thread:
-TODO: Memory profiling remaining.
+{{:newsletters:pasted:20241201-034557.png?600px}}
+Now working on actually using [[https://github.com/wolfpld/tracy|Tracy]] to get a full view of what's going on, which I've been curious about for years.
+== Heap profiling ==
+There are still persistent and pretty bad memory leaks on folk2, from 10MB to 200MB per second depending on how much stuff is out. I want to get to these soon after I fix the performance stuff.
 === Various unsuccessful quests ===
   * Heap profiling with gperftools
+    * Too slow, doesn't seem to detect most of the heap?
   * mimalloc arenas
+    * Might still be useful for images and stuff, but we'd need to migrate the arena between matches and statements as statements get reused, it's more complicated than expected
   * Sealed Tcl objects
+    * I had been hoping to do this forever -- it seems elegant and cool and efficient -- but it turned out to be much harder than expected when I looked at it -- would need to change a lot of callsites in Jim Tcl (anywhere it shimmers an object) to make it work, or do horrible hacks with COW pages.
+    * {{:newsletters:pasted:20241201-040815.png?550px}}
   * CPU-oriented scheduling
   * Spinlocks
+    * [[https://www.realworldtech.com/forum/?threadid=189711&curpostid=189723|Linus is probably right]]
+    * I got too excited since we have fairly tight locking discipline in folk2
+    * Look at all this spinning; it's not an improvement over mutexes:
+      * {{:newsletters:pasted:20241201-034649.png?500px}}
 ==== RFID localization ====
 Omar: Didn't have time to do much this month, but I did figure out how to make the IB program not crash in its new connection-dependent style. We now just drop samples from the radio until a PC connects and we set up a FIFO and RFID protocol parser then. (The current goal is to get IB and OOB both running and synced up to a reset signal over TCP from a PC, then stream info from IB and OOB to the PC.)
+This project isn't particularly blocked on anything, thankfully -- just needs more time to put IB and OOB together and implement localization and then release it.
 ==== Portable gadget ====
@@ Line 93: / Line 116: @@
 These are all 'minor' additions to the gadget that I want to throw into future revisions (and have ordered parts for).
-I've been talking about a trigger button for a while. Realized that some commercially available camera grips just use an audio jack. This seems good. Threw it into part orders and will probably throw it into next design rev.
+I've been talking about a trigger button for a while, but been blocked by figuring out what connectors/cables to use so the handle grip with trigger button can be removable (in case you want to mount onto an arm). Realized that [[https://www.amazon.com/CB-800-Universal-Pistol-Shutter-Release/dp/B01KMKZ3AY/|some commercially available camera grips just use a 2.5mm jack for a button output.]] This seems good. Let's use a 3.5mm audio jack (TRRS) instead. Threw that into part orders and will probably throw it into next design rev.
+as for what to do with the trigger button, I'm thinking about the Super Mario Galaxy Wii remote interactions and sounds...
 (I also threw a speaker into part orders.)
-I'm thinking about the Super Mario Galaxy Wii remote interactions and sounds for some reason.
+(And the battery system -- there are a bunch of Waveshare products that can supply the needed 5V5A to power everything -- I'm hoping something more hat-like will work and can be built into a future gadget case. It'll be bigger, but I'm tired of carrying around power supply, battery charger, battery module all separately, and I don't think we can easily just do USB-C anyway so the connection'll be awkward if it's an external module.)
 === New off-the-shelf revision ===
@@ Line 111: / Line 137: @@
 So I spent a while looking at Pi alternatives.
-  * The [[http://www.orangepi.org/html/hardWare/computerAndMicrocontrollers/details/Orange-Pi-5.html|Orange Pi 5]] would be good if it worked (I already have at least one or two lying around, and it's easy to get), but there's [[https://old.reddit.com/r/OrangePI/comments/109cgz8/camera_for_orange_pi_5/|almost]] [[https://old.reddit.com/r/OrangePI/comments/1b4z0ar/camera_options/|no way]] to connect a decent camera to it without an additional board/adapter. And it's a different form factor than the Raspberry Pi, so we couldn't use a battery hat.
+  * The [[http://www.orangepi.org/html/hardWare/computerAndMicrocontrollers/details/Orange-Pi-5.html|Orange Pi 5]] would be good if it worked (I already have at least one or two lying around, and it's easy to get), but there's [[https://old.reddit.com/r/OrangePI/comments/109cgz8/camera_for_orange_pi_5/|almost]] [[https://old.reddit.com/r/OrangePI/comments/1b4z0ar/camera_options/|no way]] to connect a decent camera to it without an additional custom board/adapter. (and if you're fabbing a custom board/adapter anyway, maybe just use the CM5 as below?) And it's a different form factor than the Raspberry Pi, so we couldn't use a battery hat.
     * It does work (but, again, the lack of a decent camera solution is a dealbreaker):
       * {{newsletters:img_5377.jpeg?300px}}
@@ Line 123: / Line 149: @@
   * Had our monthly open house:
-    * {{:newsletters:pasted:20241201-002712.jpeg?0x250px}} {{:newsletters:pasted:20241201-002730.jpeg?0x250px}} {{:newsletters:pasted:20241201-002742.jpeg?0x250px}} {{:newsletters:pasted:20241201-002806.jpeg?0x250px}} {{:newsletters:pasted:20241201-002813.jpeg?0x250px}} {{:newsletters:pasted:20241201-002824.jpeg?0x250px}} {{:newsletters:pasted:20241201-002833.jpeg?0x250px}} {{:newsletters:pasted:20241201-002856.jpeg?0x250px}}
+    * {{:newsletters:pasted:20241201-002730.jpeg?0x250px}} {{:newsletters:pasted:20241201-002742.jpeg?0x250px}} {{:newsletters:pasted:20241201-002806.jpeg?0x250px}} {{:newsletters:pasted:20241201-002824.jpeg?0x250px}} {{:newsletters:pasted:20241201-002833.jpeg?0x250px}} {{:newsletters:pasted:20241201-002856.jpeg?0x250px}}
     * Nice to see people playing around with the cube, the portable gadget (now that we have two, we can have one arm-mounted and one handheld), and editing programs on the table
     * Special guests: Paul from Greenville, SC where they've set up a Folk system, and Jessie who set up the Folk system at Recurse Center in downtown Brooklyn. Was nice to be able to talk through specific details/questions about people's Folk setups
@@ Line 129: / Line 155: @@
 ===== What we'll be up to in December =====
-  * **Our next Folk open house is at noon on Sunday, December 15, at our studio in East Williamsburg, Brooklyn.**
+  * **Our next Folk open house is [[https://partiful.com/e/eZXJm5gn0rxglOGi5Jsa|at noon on Sunday, December 15]], at our studio in East Williamsburg, Brooklyn.**
   * Fix blinking in new evaluator (that is, performance optimization so the programs resolve under 16ms)
   * Create a pull request for the new evaluator
@@ Line 138: / Line 164: @@
   * Gadget autoadjust exposure maybe?
   * RFID
+  * Actually revisit video support
+  * More convenient shader language
+  * Experiment with more tangible papercraft interface elements for Folk programs (using cardboard, rubberbands, etc.)
+  * Revisit a bunch of smaller system improvements that we've had in the backlog for a while
 ===== Links we've enjoyed =====
@@ Line 152: / Line 182: @@
 ==== Andrés ====
+  * [[https://www.instagram.com/reel/DBgo4oOCPQ6/|This exhibit]] showcases a variety of objects that reveal the use of touch and haptic feedback in every day objects.
+  * [[https://www.instagram.com/reel/DBrR7YPqour/|This clever cardboard pattern]] creates a neat holder for a variety of complex objects out of a single sheet of cardboard. In the video it's used for packaging but I've been thinking about how to use it to hold phones or small electronics on the table so they have a stable position for use in Folk installations.
+  * [[https://www.instagram.com/reel/DBmLZs1vbKG/|In this video]] a skirt transforms into a dress using multiple receipt printers
+  * [[https://folduptoys.com/|Fold Up Toys]] is a great resource for papercraft template
+  * Leaflet is hosting an online [[https://leaflet.pub/358545fa-c80d-4b7a-8b71-9d9d40593161|Zine Week event]]
+  * [[https://x.com/mayfer/status/1861610349876633880|tweet]] — Three.js interactive cube demo runs really well on Daylight's screen
+  * [[https://gen-omnimatte.github.io/|Generative Omnimate]] — "Learning to decompose video into layers"