Differences

This shows you the differences between two versions of the page.

--- newsletters:2024-12 [2024/12/31 11:54] – [New parallel evaluator] osnr
+++ newsletters:2024-12 [2024/12/31 22:24] (current) – [New parallel evaluator] osnr
@@ Line 1: / Line 1: @@
-====== December 2024 newsletter (WIP) ======
+====== December 2024 newsletter ======
 Thanks for following along so far! We've been working on a lot of big projects in 2024 -- many still under way (parallel evaluator, gadget design, RFID tag localization, video playback) -- but we've gotten a decent amount done this year:
@@ Line 13: / Line 13: @@
   * presented Folk at [[newsletters/2024-10#talks-and-presentations|Causal Islands conference in Berlin]]
-We're looking forward to seeing more big projects come to fruition and support new capabilities and applications in 2025.
+In 2025, we're looking forward to seeing more of these big projects come to fruition, then playing with the resulting new capabilities and applications.
 (If you want to stop by next month, our next Folk open house will be [[https://partiful.com/e/hK7MZmjeattV5QBH9Mln|in the evening on Wednesday, January 29]], at our studio in East Williamsburg, Brooklyn.)
@@ Line 95: / Line 95: @@
 </details>
-I seem to have all the parts now, so the next steps are system/software testing (does Folk run OK on the Orange Pi 5 these days? does the camera work? does the camera work with the USB 2.0 cables I have lying around?) and printing the new (somewhat bigger) chassis design I just did that fits everything (battery + Orange Pi + projector).
+I seem to have all the parts now, so the next steps are system/software testing (does Folk run OK on the Orange Pi 5 these days? does the camera work? does the camera work with the USB 2.0 cables I have lying around?), and putting it in the new (somewhat bigger) chassis design I just printed that fits everything (battery + Orange Pi + projector):
+{{newsletters:img_6236.jpeg?200px}} {{newsletters:img_6241.jpeg?200px}}
 ==== New parallel evaluator ====
@@ Line 112: / Line 114: @@
 === Performance and memory leaks ===
-For most of this month, I've been staring at increasingly elaborate [[https://github.com/wolfpld/tracy|Tracy]] views of folk2 (and adding more hooks to folk2 to report more info to Tracy, and fixing performance bottlenecks and behavioral bugs and leaks flagged by the Tracy view). Tracy is really, really amazing software.
+For most of this month, I've been staring at increasingly elaborate [[https://github.com/wolfpld/tracy|Tracy]] views of folk2 (and adding more [[https://github.com/osnr/db/blob/e5a7256863927f4a1a9fd2202a10b6fc0aedc3a4/prelude.tcl#L240|hooks]] to folk2 to report more info to Tracy, and fixing performance bottlenecks and behavioral bugs and leaks flagged by the Tracy view). Tracy is really, really amazing software.
 {{newsletters:82202b3d-c55c-4057-bd1b-267b69c8eb64-45141-0000018e64246b6f.png?500px}}
-I'm kind of addicted to it -- it feels like it makes a lot of the monitoring stuff I've hacked into folk2 obsolete, because you can just use Tracy to //see// everything (as long as you add the right hooks). No need for perf probes, or gperftools heap monitoring, or the trace log endpoint.
+Now that I've added enough hooks, I get these extremely rich views of all Folk programs/Whens executing (including tag recognition time), frames (when does the GPU frame run? how long does it take? what about the camera frame?), draw count (so we can see when stuff is blinking out and debug that), lock contention, thread behavior, stack profiles for performance analysis, etc.
+{{newsletters:img_5888.jpeg?400px}}
+Some of the resulting improvements this month:
+  * Fixed a ton of memory leaks (mostly freeing clauses and terms in clauses, and some transient objects in C); I think folk2 might be less leaky than folk1 now.
+    * The Tracy memory visualization is very cool:
+      * {{newsletters:img_5767.mp4?250px}}
+  * Saw that we were spending a lot of time waiting on the one big trie lock, so replaced the trie with an immutable path-copying trie (and an [[https://aturon.github.io/blog/2015/08/27/epoch/|epoch-based reclamation scheme]]) so readers never have to block and writers only have to retry if someone else changed the trie while they were writing (like a transaction)
+  * Fixed a [[https://github.com/osnr/db/commit/6397d3e9c30949a236d382b771cf95bbd8c0e603|huge 2-character bug with collect that was causing us to do way too much work]]
+  * Created (by patching up memory leaks) and then fixed a problem where we would leak all the statement slots which eventually caused the whole system to blow up
+    * {{newsletters:img_5829.jpeg?350px}}
+    * "We were not freeing aborted ref which caused us to run out of statement slots which caused statementNew to hang which caused the global workqueue to overflow."
+  * Fixed horrible bug where my steal-half deque operation was wrong (so were dropping lots of operations, the system was not behaving correctly), replaced with more standard steal-one
+I'm kind of addicted to using Tracy -- it feels like it obsoletes a lot of the monitoring stuff I'd hacked into folk2, because you can just use Tracy to //see// everything (as long as you add the right hooks). No need for perf probes, or gperftools heap monitoring, or the trace log endpoint. The Tracy hooks are low-overhead (nanoseconds), networking is taken care of for you, and they automatically give you great visualizations that you can zoom and pan and see in context with all other info (which also reduces the need for boiling things down to summary statistics).
 And using Tracy feels //powerful//, UI made for experts with high information density and a high ceiling.
@@ Line 122: / Line 140: @@
 {{newsletters:img_5900.jpeg?300px}}
-Fixed a ton of memory leaks; I think folk2 might be less leaky than folk1 now.
 ==== RFID localization ====
@@ Line 141: / Line 158: @@
 (Even before this, we've been kind of poking at Linux to get stuff to work, locking threads at high priority and putting them on specific CPUs, etc. If I turn that stuff off and rely on default scheduling behavior, we do get worse at talking to the tags, so it's doing //something//...)
+{{newsletters:img_5973.jpeg?350px}}
 === Rounds ===
@@ Line 157: / Line 176: @@
 ==== Other Folk system improvements ====
-  * Sébastien Lerique has been setting up a system in France; fixed an error in calibration
+  * Sébastien Lerique has been setting up his system in France; [[https://github.com/FolkComputer/folk/pull/182|fixed an error in calibration]]
     * We might need a better strategy for calibrating ultra-short-throw / highly distorted projectors (maybe conservatively only support a single plane?)
-  * Daniel Pipkin fixed SSID detection on macOS Sequoia
+  * Daniel Pipkin [[https://github.com/FolkComputer/folk/commit/206c0185568792d554293c757975a4162fa7afba|fixed SSID detection on macOS Sequoia]]
-  * Andrés added support for single-l ''labeled'' (it gives a warning)
+  * Andrés [[https://github.com/FolkComputer/folk/pull/181|added support for single-l ''labeled'']] (it gives a warning)
 ==== Friends and outreach ====