Differences

This shows you the differences between two versions of the page.

--- newsletters:2025-01 [2025/02/03 02:37] – [General Folk system improvements] osnr
+++ newsletters:2025-01 [2025/02/03 03:46] (current) – [Portable Folk gadget] osnr
@@ Line 34: / Line 34: @@
 === Smoothing ===
+Mason has been testing [[https://github.com/smj-edison/folk/commit/c4ee3e45e6992d19e76fcacea8473bb3974392f0|some pose stabilization logic that he has on his fork.]]
+> The idea is to use a continuous average, but if the movement is too large to jump to the new location… Right now I'm just averaging the matrix coefficients, so I'm sure there's better ways to interpolate…
+>
+> I'm not completely satisfied with how it's designed right now as it's a linear interpolation of the matrix coefficients. I'm wondering if I could do something like a quadratic or hermite spline of position + rotation as quaternion
+It reduces jitter a lot; probably most useful for the code editor where jitter gets very annoying. Videos are on Discord if you're on Discord: https://discord.com/channels/956758212152025098/956765077883744266/1334712969853931550
+(Omar: Some interesting questions about what the right policy here is. It does look like it adds latency, and maybe the right behavior is application-dependent, which is why we haven't tried it ourselves. I need to look at this again once some of the perf improvements land.)
+=== Edit running programs ===
+[[https://x.com/_rpf|Rob Fielding]] in New Zealand made a little editor hack to let you "edit a running program by 'touching' it on to the whisker":
+<code>
+When /page/ is a keyboard with path /kbPath/ /...anything/ &\
+   /page/ is an editor &\
+   the /kbPath/ cursor is /cursor/ {
+  set id "$page$kbPath"
+  When $page points up at /target/ & /target/ has program code /runningCode/ {
+    Wish $target is labelled "targetting..."
+    Hold "cursor$kbPath" {
+      # Claim the $kbPath cursor is [list 0 0 ]
+      Hold "code$kbPath" {
+          # Claim $id has editor code [deleteToBeginning $editorCode $cursor]
+          Claim $id has program code $runningCode
+          Claim $id has editor code $runningCode
+      }
+      lassign $cursor x y
+      Claim the $kbPath cursor is [list 0 $y]
+    }
+  }
+}
+</code>
+> This editor hack allows you to edit a running program by "touching" it on to the whisker - you need to move it away again otherwise I think it fights with resetting the editor code.  Also having two running codes can sometimes lead to weirdness.  I added this hack in virtual-programs/editor.folk I wanted to add it as a separate program, and make it a "mixin" but I couldn't seem to get the scope of the code editor correct and the cursor misbehaved.  Next, I'd to make the editor more useful to me I want to make the viewport scrollable and add vim-keys.
 ==== New parallel evaluator ====
-Omar: Progress on this has been limited this month.
+Omar: Progress on this has been limited this month. A lot of looking at these specific spans in Tracy where there was a blink-out of a tag outline:
-A lot of looking at these specific spans in Tracy where there was a blink-out of a tag outline:
+{{:newsletters:pasted:20250131-195852.png?400px}} {{:newsletters:pasted:20250131-195919.png?400px}}
-{{:newsletters:pasted:20250131-195852.png?400px}} {{:newsletters:pasted:20250131-195919.png}}
+You can see the drawcount plot mentioned at the bottom -- when that dips from 4 (the edges of the page outline) to 0, you know you have an unwanted blink (assuming the page was visible on camera the whole time). Tried adding more messages so we can see the actual statements that are evaluating (and getting stuck on).
-You can see the drawcount plot at the bottom -- when that dips from 4 to 0, you know you have an unwanted blink. Tried adding more messages so we can see the actual statement.
+You can see some of the red spans in the above images where threads are getting preempted while running Folk When code, which delays all downstream statements, and they might be delayed for 10ms+ while the thread isn't on the CPU, which means you'll miss the frame deadline and the page blinks out! Here's another one (the SAY I'm hovering over and the HOLD above it have both been preempted and delayed in that way):
-Red span
+{{:newsletters:pasted:20250203-031534.png?500px}}
-I cleaned up the detection of past versions (it makes an explicit folk.pid file) and have been rethinking our thread spawning/killing policies. I think a lot of the blinking is just because we have too many threads for the number of CPUs and they're preempting each other mid-task, which means that the evaluation doesn't converge in time for the frame.
+After getting back in late January, I ran into issues running the real-time scheduler branch from last month (it's hard to find & kill those threads by process name? do they dominate their CPUs too much?). So I cleaned up the detection of past versions (Folk now makes an explicit folk.pid file) and have been rethinking our thread spawning/killing policies.
+Again, I think a lot of the blinking is just because we have too many threads for the number of CPUs and they're preempting each other mid-task, which means that the evaluation doesn't converge in time for the frame. So need to manage the number of threads in a more reasonable way. Maybe based on I/O-bound-ness (is the thread in Linux Running or Blocked state?) instead of just pure execution time which was always a hack...
 ==== Portable Folk gadget ====
-All the components are in. This revision uses the Orange Pi 5.
+Omar: I've been working on a new portable Folk gadget that uses off-the-shelf parts, so anyone will be able to build/buy one, along with some other improvements: built-in battery, trigger button you can press with your finger, built-in speaker.
+All the components are in place and I'm starting to fix up the software to make Folk work on the platform.
 {{:newsletters:pasted:20250131-194410.jpeg?250px}} {{:newsletters:pasted:20250131-194400.jpeg?250px}}
+This revision uses the Orange Pi 5 instead of the Raspberry Pi 5, and a stereo USB camera instead of Pi wide-angle camera, which has caused various random issues.
 === Network setup ===
@@ Line 60: / Line 108: @@
   * Still struggling to make it do ad-hoc or access point mode by default
     * (ideally it would work in concurrent mode, so could both get on known existing networks and make its own for wild environments:)
-      * {{:newsletters:pasted:20250131-200028.png?300px}}
+      * {{:newsletters:pasted:20250131-200028.png?400px}}
 === The stereo camera ===
-The stereo camera works! Its field of view is huge, though:
+The stereo camera works! Its field of view is huge, though (the subregion we actually care about in each stereo image is mostly the red-circled region where we can project):
-{{newsletters:imagestereo.png?400px}}
+{{newsletters:foooo.jpg?400px}}
-Added crop support to just take the center part of the left image for now so we don't waste time processing the rest and can start trying to calibrate right away.
+[[https://github.com/FolkComputer/folk/pull/189|Added crop support]] to just take the center part of the left image for now so we don't waste time processing the rest and can start trying to calibrate right away.
+Haven't been able to calibrate yet, though; I think I'm having issues with exposure so I can't read enough tags to calibrate.
 === Current state & next steps ===
-really heavy
+Overall, the new gadget revision is in OK shape, although not done yet:
-button works
-battery works
-sound not working yet
+  * It has a working trigger button! That'll open up a lot of fun point-and-drag and lasso interactions. Here I'm having the 'marching ants' rectangle turn from white to green when I press the button down:
+    * {{newsletters:img_7370.mp4}}
+    * This is the program that implements that (I just put it in setup.folk, since we don't have physical programs here yet):
+      * <code>
+exec gpio mode 7 up
-need to figure out how to calibrate.
+When display /disp/ has width /w/ height /h/ {
+    When the clock time is /t/ {
+        set pressed [expr {![exec gpio read 7]}]
+        set color [expr {$pressed ? "green" : "white"}]
+        Wish to draw a dashed stroke with points [list [list 0 0] [list $w 0] [list $w $h] [list 0 $h] [list 0 0]] \
+          color $color width 10 dashlength 40 dashoffset [expr {fmod($t, 10)*-120}]
+    }
+}
+</code>
+      * it basically just sets a pullup resistor on GPIO 7 (where I plugged the button in) & shells out to [[https://github.com/orangepi-xunlong/wiringOP|wiringOP]] every frame to read that pin; to avoid the multi-ms overhead of shelling out, would be good to make a Tcl-C library to call into wiringOP and do this (I think the C header and library is already on the device anyway!)
+  * It has a working battery, so you don't need to plug it in & carry a long extension cord! I haven't tested battery life but probably at least an hour? And I can even get live reports over serial about battery percentage, just like a phone or laptop shows battery percentage:
+    * {{:newsletters:pasted:20250203-034338.png?600px}}
+    * (will be fun to get this info into Folk and display it ambiently in the corner or on the marching ants border or whatever)
+  * It's really heavy and kind of big, notably heavier than the older gadget (because it has 3 18650 batteries and a heavier mini projector with a full chassis). Not sure what to do about this. Maybe go back to the concept of the detachable battery pack (a standard USB-C battery pack would be ideal, but I don't think I could quite get that to work without more custom power stuff).
+  * It has a speaker! I haven't gotten this to work yet (I haven't gotten the Orange Pi 5 3.5mm audio jack to work at all. Linux audio struggles). Also curious how loud it'll be without an amplifier.
+  * The biggest issue is that I haven't been able to calibrate it yet, but I just need to spend more time on that...
-need to do some physical revision
+Finally, need to finish physical design (I want to put on the back panel so people feel more comfortable handling it, less wires poking out, fix the speaker in place, maybe use a different camera, make the grip feel less precarious since it only has 1 bolt) and publish on GitHub.
 ==== RFID localization ====
@@ Line 101: / Line 166: @@
 ===== What we'll be up to in February =====
-  * **Our next Folk open house is [[https://partiful.com/e/RdYuL5NiNl8XI6AmebDS|in the evening on Wednesday, February ??]], at our studio in East Williamsburg, Brooklyn.**
+  * **Our next Folk open house is [[https://partiful.com/e/RdYuL5NiNl8XI6AmebDS|in the evening on Monday, February 24]], at our studio in East Williamsburg, Brooklyn.**
+  * Omar: Finish the new gadget revision (back panel, audio support, calibrate camera) and document it on GitHub for people to try.
+  * Omar: Set up the out-of-band RFID radio and maybe start on localization.
+  * Omar: Figure out thread management on the new evaluator so work items don't get delayed for 10ms at a time (hopefully will fix blinking)
+  * Andrés: Work on video
+  * Andrés: Dot detector port to work under 3D calibration
   * We'll be speaking at the [[https://interactiondesign.sva.edu/|School of Visual Arts' Interaction Design]] department’s 2025 winter/spring lecture series. The talk will be recorded so we'll link it here as soon as it's online.
-  * We'll also be showing Folk as an interactive prototype in a workshop in the Design in Action course at Columbia GSAPP. The students' [[https://gsapp-cdp.github.io/archive/|past work]] is very Folk-adjacent so we expect to get a lot of interesting feedback and are excited to show an enthusiastic technical audience Folk up-close.
+  * We'll also be showing Folk as an interactive prototype in a workshop in the Design in Action course at Columbia GSAPP. The students' [[https://gsapp-cdp.github.io/archive/|past work]] is very Folk-adjacent so we expect to get a lot of interesting feedback and are excited to show an enthusiastic technical audience Folk up close.
+  * Meeting with the people from [[https://spatialpixel.com|Spatial Pixel]] this week; excited to compare approaches and see where we can collaborate.
 ===== Links we've enjoyed =====