Differences

This shows you the differences between two versions of the page.

--- newsletters:2025-11 [2025/12/08 18:08] – [ForEach! control structure support] osnr
+++ newsletters:2025-11 [2025/12/08 20:37] (current) – [JPEG frames and Term struct] osnr
@@ Line 27: / Line 27: @@
 From the user point of view, the biggest change is that you now need a blank to edit as your starting point to write a new program. But you can also quickly edit existing programs, which makes the system feel a lot more flexible and powerful. (you're not stuck with whatever programs you printed out, and you can 'ask questions' about anything on the table by opening it in the editor)
+=== Fixed title blinking ===
+Using titles to show the "Edited" notice meant we noticed that titles blinking was a real problem.
+{{newsletters:img_6613.mp4?300px}}
+Debugged it for a while. Found that the display drawlist was empty when blinking out:
+{{newsletters:screenshot-2025-11-20-at-1.03.52-pm.png?550px}}
+Then found that the display drawlist was empty because we weren't able to acquire the collection statement:
+{{newsletters:screenshot-2025-11-20-at-1.09.18-pm.png?550px}}
+[[https://github.com/FolkComputer/folk/commit/13128c99838ad4d0364d71c004ed51df010f26a3|Fixed by retrying acquisition of that collection statement until it works]], since it should always eventually be present.
+==== Other improvements on folk2 ====
+Many small improvements as we use the editor and label more intensively.
+  * Andrés [[https://github.com/FolkComputer/folk/commit/6066a1917f8b82fbbe4975ad4b0540410fa4b981|fixed rotation of virtual pages in the web editor]]
+  * [[https://github.com/FolkComputer/folk/compare/594a4f1d...8feaceda|Created /camera live stream page]] that we can use in both /calibrate and /setup (as opposed to /camera-frame that only shows a static snapshot)
+    * Would be nice to put camera controls on here
+  * [[https://github.com/FolkComputer/folk/commit/07966c9e9b4791c1164fafac481c6017881f724b|Replaced ''(keep 10ms)'' with ''-keep 10ms'']], which finally makes all Folk command parameters consistent (also consistent with Tcl parameters)
+  * [[https://github.com/FolkComputer/folk/commit/aff8c3d087f72638582797ac75fa476d26c55fb7|Auto-shrink labels based on length of longest line]] -- makes label much more usable when feeding arbitrary text / command output into it
+    * Also: [[https://github.com/FolkComputer/folk/commit/a10edd1deef5e3fbfd452cb846c7b985cc60d28a|auto-position label slightly lower]] so it doesn't constantly overlap with tags
+    * (might be good to break really long lines and mask out tags as well)
+  * [[https://askubuntu.com/questions/3375/how-to-change-time-zone-settings-from-the-command-line|Set timezone]] on folk-hex, so printed programs have a locally meaningful timestamp finally:
+    * [[https://askubuntu.com/questions/3375/how-to-change-time-zone-settings-from-the-command-line|{{newsletters:screenshot-2025-11-19-at-3.48.29-pm.png?300px}}]]
+    * (motivated by wanting to format and project edit times for live-edited programs after Mason's new editor)
+  * Here is a nice trick that gives you live-reload on Folk web pages:
+    * {{newsletters:15b7133c-a75d-4bb3-9379-94d067a4d7b4-8805-00001151c29d0045.png?500px}}
@@ Line 208: / Line 240: @@
 === ForEach! control structure support ===
-We [[https://github.com/FolkComputer/folk/commit/2da1bba6019a14269e244d07bac1ee126291e4f5|refactored the printing query/control flow to use ForEach!.]]
+To work toward Daniel's interest in adding new printing features, we [[https://github.com/FolkComputer/folk/commit/2da1bba6019a14269e244d07bac1ee126291e4f5|refactored the printing query/control flow to use ForEach!.]]
-{{.:pasted:20251208-180715.png?400px}}
+{{.:pasted:20251208-180715.png?450px}}
 [[https://github.com/FolkComputer/folk/commit/2da1bba6019a14269e244d07bac1ee126291e4f5|But ForEach! didn't support early return, continue, etc like you would hope that a loop would, so we fixed that.]] It also had [[https://github.com/FolkComputer/folk/commit/3df83c12df142a24d5cd21226bbf3374830f03ae|terrible stack traces, which we fixed.]] (also hopefully will help web handlers have better stack traces)
@@ Line 216: / Line 248: @@
 === -atomically auto-expiration ===
-Omar wanted to fix an issue where any -atomically statement would never get retracted if you deleted its ancestor (since it never got obsoleted by a fresher version and there's no other retraction mechanism for -atomically statements).
+Omar and Daniel [[https://github.com/FolkComputer/folk/commit/dbae48b7f45dd3c275f4f2e01bdc68a339c60218|fixed an interpreter design issue]] where any -atomically statement would never get retracted if you deleted its ancestor (since it never got obsoleted by a fresher version and there's no other retraction mechanism for -atomically statements).
-This introduced a bug where stuff would blink out.
+The fix was to add a timeout (50-100ms) to AtomicallyVersion that garbage-collects it if it hasn't had a new convergence in a while. sysmon walks every -atomically key and checks to do that garbage collection every 20 ticks. Ideally, that timeout would be domain/statement-specific...
-Now it's fixed. See how Mega Man doesn't blink:
+== Fixing reintroduced blinking ==
+This introduced a bug where stuff would blink out if we missed a frame once in a while.
+Fixed in a hacky way by just [[https://github.com/FolkComputer/folk/commit/bfba3ab953ad624573a27c0fd08a8c189af7110d|raising the timeout to 100ms.]] See how Mega Man doesn't blink:
 {{.:pasted:20251207-220711.mp4}}
-==== Major memory leak fix ====
+We probably need to think about this a bit more and see if we should expose a richer API so you can choose these timeouts at the caller.
+==== Other folk2 debugging ====
+=== Major memory leak fix ===
+Omar: For a couple months, folk2 has had another major memory leak (a few MB per second). It turned out to be a leak of clauses/terms, specifically, not Jim Tcl data structures or other Folk structures.
+I spent a while tracking it down and [[https://github.com/FolkComputer/folk/commit/ac285cb80e03f467363120f79c6487449cc99b25|managed to fix it]]: it's because we weren't freeing speculatively allocated clauses when their parents no longer existed on statement creation.
+=== "Added tag" pileup (not fixed yet) ===
+I've been hunting down this pileup bug for months but not made much progress on it.
+{{newsletters:screenshot-2025-11-19-at-4.41.04-pm.png?0x200px}} {{newsletters:screenshot-2025-11-19-at-6.40.38-pm.png?0x200px}} {{newsletters:screenshot-2025-11-20-at-2.31.12-pm.png?0x200px}}
+It comes from this claim that tag has a program getting recreated:
+{{newsletters:screenshot-2025-11-20-at-3.06.51-pm.png?500px}}
+I thought I'd tracked it down, but they were just speculative recreations that didn't actually commit to the db.
+=== AtomicallyVersion leak (not fixed yet) ===
-Omar: For a couple months, folk2 has had another major memory leak (a few MB per second). I spent a while tracking it down. It was a leak of clauses/terms, specifically.
+I haven't been able to reliably replicate this. You end up spending huge amounts of time per sysmon tick on garbage collection, because so many AtomicallyVersions are around.
+{{newsletters:screenshot-2025-11-26-at-1.18.13-am.png?500px}}
+{{newsletters:screenshot-2025-11-26-at-1.37.17-am.png?600px}}
 ==== Setup branch ====
@@ Line 272: / Line 331: @@
 === JPEG frames and Term struct ===
-I was thinking about how to speed up the /camera-frame view when we're streaming it in setup and calibrate, and I realized that we're decompressing every webcam frame (for AprilTag detect) (since webcams send you frames as a JPEG stream), then recompressing every frame as a JPEG to send to the web client. Why not put the original JPEG from the webcam in a statement, before we decompress, and just send that to the web client?
+I was thinking about how to speed up the /camera-frame view when we're streaming it in setup and calibrate, and I realized that we're decompressing every webcam frame (for AprilTag detect) (since webcams send you frames as a JPEG stream), then recompressing every frame as a JPEG, writing to disk, then reading back from disk to send to the web client. Why not put the original JPEG from the webcam in a statement, before we decompress, and just send that buffer to the web client?
 That sent me on a journey to try to embed the raw JPEG data in terms/statement clauses/the trie, which led me to [[https://github.com/FolkComputer/folk/compare/90d0a582...3cb882e0|replace the null-terminated C strings with a new length-prefixed Term struct]]. Now you can (in theory) have null characters in terms, so you can put binary data in terms directly. That's a change that Mason wanted to try a few months ago, since it seems both safer and potentially faster.
@@ Line 282: / Line 341: @@
 I didn't end up using this for JPEG transmission -- instead making an Image-like struct that points at the JPEG data on the heap -- because I was paranoid about term copying in and out of interpreters, and the indirect struct lets us only copy when we actually want to read the JPEG data (on output to web client and on decompress).
-Anyway, more on using the JPEG data next month -- it gives us color preview and faster livestream, which is great, and I'm hoping to use the color in more places soon.
+Here's what /camera-frame looks like after this change (no more recompress, no more write to disk):
+{{.:pasted:20251208-203638.png?600px}}
+Anyway, more on using the JPEG data next month -- it gives us color preview and faster livestream, which is great, and I'm hoping to use the color in more places soon. I [[https://github.com/FolkComputer/folk/commit/4f901e20cc56f198622f8825deaab8e6b9ebc5b0|switched to the TurboJPEG API]], which I think might let us [[https://graphicdesign.stackexchange.com/a/56130|crop the JPEG without decompressing it first]], which would be great for cropped cameras and stereo camera.
 === Web handler improvements ===
@@ Line 296: / Line 359: @@
   * [[https://github.com/FolkComputer/folk/commit/594a4f1d4df3efa978dd1a26e816c0bd00d106aa|Auto-query-param]]
     * web handlers now automatically get ''?foo=bar&x=y'' query params parsed into ''$QUERY(foo)'', ''$QUERY(bar)'' (you don't need to specify them in the route regex) (think of ''$_GET'' in PHP)
-==== Other improvements on folk2 ====
-Many small improvements as we use the editor and label more intensively.
-  * Andrés [[https://github.com/FolkComputer/folk/commit/6066a1917f8b82fbbe4975ad4b0540410fa4b981|fixed rotation of virtual pages in the web editor]]
-  * [[https://github.com/FolkComputer/folk/commit/07966c9e9b4791c1164fafac481c6017881f724b|Replaced ''(keep 10ms)'' with ''-keep 10ms'']], which finally makes all Folk command parameters consistent (also consistent with Tcl parameters)
-  * [[https://github.com/FolkComputer/folk/commit/aff8c3d087f72638582797ac75fa476d26c55fb7|Auto-shrink labels based on length of longest line]] -- makes label much more usable when feeding arbitrary text / command output into it
-    * Also: [[https://github.com/FolkComputer/folk/commit/a10edd1deef5e3fbfd452cb846c7b985cc60d28a|auto-position label slightly lower]] so it doesn't constantly overlap with tags
-    * (might be good to break really long lines and mask out tags as well)
-  * [[https://askubuntu.com/questions/3375/how-to-change-time-zone-settings-from-the-command-line|Set timezone]] on folk-hex, so printed programs have a locally meaningful timestamp finally:
-    * [[https://askubuntu.com/questions/3375/how-to-change-time-zone-settings-from-the-command-line|{{newsletters:screenshot-2025-11-19-at-3.48.29-pm.png?300px}}]]
-    * (motivated by wanting to format and project edit times for live-edited programs after Mason's new editor)
-  * Here is a nice trick that gives you live-reload on Folk web pages:
-    * {{newsletters:15b7133c-a75d-4bb3-9379-94d067a4d7b4-8805-00001151c29d0045.png?500px}}
 ==== Outreach ====