Differences

This shows you the differences between two versions of the page.

--- newsletters:2024-12 [2024/12/31 11:28] – osnr
+++ newsletters:2024-12 [2024/12/31 22:24] (current) – [New parallel evaluator] osnr
@@ Line 1: / Line 1: @@
-====== December 2024 newsletter (WIP) ======
+====== December 2024 newsletter ======
 Thanks for following along so far! We've been working on a lot of big projects in 2024 -- many still under way (parallel evaluator, gadget design, RFID tag localization, video playback) -- but we've gotten a decent amount done this year:
@@ Line 7: / Line 7: @@
   * first version of [[newsletters/2024-09#handheld-folk-gadget|handheld Folk gadget]] ([[https://www.youtube.com/shorts/SHHPDiTw8jo|basic video]])
   * more installations around the world: [[newsletters/2024-08#upstate-carolina-linux-users-group|South Carolina]] ([[https://youtu.be/jP1FiJDB-tg?t=2100|video]]), Recurse Center, [[newsletters/2024-10#friends-and-outreach|SPLASH]], [[https://merveilles.town/@s_ol/112461205136786159|Milan]], etc
-  * Jessie Grosen's SuperCollider music systems
+  * Jessie Grosen's SuperCollider [[newsletters/2024-07#demos|music]] [[newsletters/2024-09#applications-and-demos|systems]]
   * s-ol bekic's [[2024-06#folkjs|folk.js]] library
   * [[https://github.com/FolkComputer/folk-cnc|CNC preview]] [[newsletters/2024-05#cnc|project]]
@@ Line 13: / Line 13: @@
   * presented Folk at [[newsletters/2024-10#talks-and-presentations|Causal Islands conference in Berlin]]
-We're looking forward to seeing more big projects come to fruition and support new capabilities and applications in 2025.
+In 2025, we're looking forward to seeing more of these big projects come to fruition, then playing with the resulting new capabilities and applications.
 (If you want to stop by next month, our next Folk open house will be [[https://partiful.com/e/hK7MZmjeattV5QBH9Mln|in the evening on Wednesday, January 29]], at our studio in East Williamsburg, Brooklyn.)
@@ Line 83: / Line 83: @@
 The problems with the Orange Pi were that it doesn't have a usable CSI camera & its pinout is weird (you can't fit it on top of a Waveshare UPS hat, for instance). But these are surmountable problems compared to the projector just not working on the Rock 5B or Raspberry Pi 5. I can use the standalone [[https://www.waveshare.com/wiki/UPS_Module_3S|Waveshare 3S UPS module]] (which might be more appropriate anyway -- it has broken-out power switch and charging port) and (importantly; I didn't consider this possibility before) I can just use a small USB camera.
-= USB cameras =
+<details>
+<summary>USB cameras</summary>
 So I did a small dive into USB cameras. I had been sour on them because I assumed they were all unwieldy, big, webcam-shaped, but there are a lot of decent ones with USB output from [[https://www.amazon.com/dp/B0D9793XZN?ref=ppx_yo2ov_dt_b_fed_asin_title|ELP]] and [[https://www.amazon.com/dp/B0CLXZ29F9?ref=ppx_yo2ov_dt_b_fed_asin_title|Arducam]] that are more Pi-camera-shaped (small) and that you can just buy on Amazon or AliExpress.
@@ Line 92: / Line 93: @@
 (I also spent a while digging through USB-A and USB-C cables and adapters to find a short cable with minimal plug footprint, 90-degree angle, etc, since the normal USB cables and plugs are obviously a lot bigger than I'd like to fit inside the chassis, in contrast to the Pi CSI camera's ribbon cable and connector.)
+</details>
-I seem to have all the parts now, so the next steps are system/software testing (does Folk run OK on the Orange Pi 5 these days? does the camera work? does the camera work with the USB 2.0 cables I have lying around?) and printing the new (somewhat bigger) chassis design I just did that fits everything (battery + Orange Pi + projector).
+I seem to have all the parts now, so the next steps are system/software testing (does Folk run OK on the Orange Pi 5 these days? does the camera work? does the camera work with the USB 2.0 cables I have lying around?), and putting it in the new (somewhat bigger) chassis design I just printed that fits everything (battery + Orange Pi + projector):
+{{newsletters:img_6236.jpeg?200px}} {{newsletters:img_6241.jpeg?200px}}
 ==== New parallel evaluator ====
@@ Line 110: / Line 114: @@
 === Performance and memory leaks ===
-For most of this month, I've been staring at increasingly elaborate [[https://github.com/wolfpld/tracy|Tracy]] views of folk2 (and adding more hooks to folk2 to report more info to Tracy, and fixing performance bottlenecks and behavioral bugs and leaks flagged by the Tracy view). Tracy is really, really amazing software.
+For most of this month, I've been staring at increasingly elaborate [[https://github.com/wolfpld/tracy|Tracy]] views of folk2 (and adding more [[https://github.com/osnr/db/blob/e5a7256863927f4a1a9fd2202a10b6fc0aedc3a4/prelude.tcl#L240|hooks]] to folk2 to report more info to Tracy, and fixing performance bottlenecks and behavioral bugs and leaks flagged by the Tracy view). Tracy is really, really amazing software.
 {{newsletters:82202b3d-c55c-4057-bd1b-267b69c8eb64-45141-0000018e64246b6f.png?500px}}
-Fixed a ton of memory leaks; I think folk2 might be less leaky than folk1 now.
+Now that I've added enough hooks, I get these extremely rich views of all Folk programs/Whens executing (including tag recognition time), frames (when does the GPU frame run? how long does it take? what about the camera frame?), draw count (so we can see when stuff is blinking out and debug that), lock contention, thread behavior, stack profiles for performance analysis, etc.
+{{newsletters:img_5888.jpeg?400px}}
+Some of the resulting improvements this month:
+  * Fixed a ton of memory leaks (mostly freeing clauses and terms in clauses, and some transient objects in C); I think folk2 might be less leaky than folk1 now.
+    * The Tracy memory visualization is very cool:
+      * {{newsletters:img_5767.mp4?250px}}
+  * Saw that we were spending a lot of time waiting on the one big trie lock, so replaced the trie with an immutable path-copying trie (and an [[https://aturon.github.io/blog/2015/08/27/epoch/|epoch-based reclamation scheme]]) so readers never have to block and writers only have to retry if someone else changed the trie while they were writing (like a transaction)
+  * Fixed a [[https://github.com/osnr/db/commit/6397d3e9c30949a236d382b771cf95bbd8c0e603|huge 2-character bug with collect that was causing us to do way too much work]]
+  * Created (by patching up memory leaks) and then fixed a problem where we would leak all the statement slots which eventually caused the whole system to blow up
+    * {{newsletters:img_5829.jpeg?350px}}
+    * "We were not freeing aborted ref which caused us to run out of statement slots which caused statementNew to hang which caused the global workqueue to overflow."
+  * Fixed horrible bug where my steal-half deque operation was wrong (so were dropping lots of operations, the system was not behaving correctly), replaced with more standard steal-one
+I'm kind of addicted to using Tracy -- it feels like it obsoletes a lot of the monitoring stuff I'd hacked into folk2, because you can just use Tracy to //see// everything (as long as you add the right hooks). No need for perf probes, or gperftools heap monitoring, or the trace log endpoint. The Tracy hooks are low-overhead (nanoseconds), networking is taken care of for you, and they automatically give you great visualizations that you can zoom and pan and see in context with all other info (which also reduces the need for boiling things down to summary statistics).
+And using Tracy feels //powerful//, UI made for experts with high information density and a high ceiling.
+{{newsletters:img_5900.jpeg?300px}}
 ==== RFID localization ====
@@ Line 133: / Line 158: @@
 (Even before this, we've been kind of poking at Linux to get stuff to work, locking threads at high priority and putting them on specific CPUs, etc. If I turn that stuff off and rely on default scheduling behavior, we do get worse at talking to the tags, so it's doing //something//...)
+{{newsletters:img_5973.jpeg?350px}}
 === Rounds ===
@@ Line 149: / Line 176: @@
 ==== Other Folk system improvements ====
-  * Sébastien Lerique has been setting up a system in France; fixed an error in calibration
+  * Sébastien Lerique has been setting up his system in France; [[https://github.com/FolkComputer/folk/pull/182|fixed an error in calibration]]
     * We might need a better strategy for calibrating ultra-short-throw / highly distorted projectors (maybe conservatively only support a single plane?)
-  * Daniel Pipkin fixed SSID detection on macOS Sequoia
+  * Daniel Pipkin [[https://github.com/FolkComputer/folk/commit/206c0185568792d554293c757975a4162fa7afba|fixed SSID detection on macOS Sequoia]]
-  * Andrés added support for single-l ''labeled'' (it gives a warning)
+  * Andrés [[https://github.com/FolkComputer/folk/pull/181|added support for single-l ''labeled'']] (it gives a warning)
 ==== Friends and outreach ====