Folk Computer

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== January 2025 newsletter ======

===== What we've been up to =====


==== General Folk system improvements ====

  * Andrés and [[https://github.com/smj-edison|Mason Jones]] improved calibration instructions: [[https://github.com/FolkComputer/folk/pull/190|clarified wording to be more explicit about lifting the board above the table]] and [[https://github.com/FolkComputer/folk/pull/188|added an example video of calibration]]; a lot of people have been really confused by calibration and have reported that it doesn't work (when actually they aren't calibrating it right; usually they're only training on the table plane so it's degenerate), so we've been meaning to clear it up
  * Omar added crop support to camera driver so you can focus on a subregion of the camera image (see gadget section below)


=== Updating the dot detector ===

Andrés: I made [[https://folk.computer/notes/circle-detector?s[]=dot&s[]=detector|a wiki page]] for Naveen Michaud-Agrawal's circle/dot detector that integrates OpenCV's circle detection with Folk. Unfortunately when I tried this on the most recent version of Folk I noticed it needed to be updated to translate coordinates to our 3D coordinate system. Omar helped me figure out the linear algebra to do this translation and I have a branch that's starting to work. I should have an updated example the first week of February and am looking forward to developing demos that use dots as interface elements (connect-the-dots & count dots to generate numbers come to mind).

=== Handwriting recognition proposal ===

Omar: I've finally starting to look at putting in blanks, recognition regions, etc inline w/ code on printed programs. I think it would be cool to provide an API like this:

{{:newsletters:pasted:20250131-202950.jpeg?400px}}

{{:newsletters:pasted:20250131-202955.jpeg?400px}}

where you get ''"X"'' as a string as the return value of ''Blank2''. This would be great instead of (or in addition to) a more explicit API where you have to like ''Wish $this prints rectangle with width 4cm height 2cm x 0cm y 10cm'' and ''When'' query for the camera slice of the field and run OCR on that etc etc. You'd be able to use handwriting input out of the box, as easily as you read from the terminal in basic programming on a laptop.

There are technical subtleties to making this work that we'll need to figure out, since the call to ''Blank2'' (maybe call it ''Input'' and have it automatically auto-find its height?) will need to yield to the Folk database and do a bunch of querying and slow computer vision. We'll need to think about bringing ''Observe'' in, finally, maybe.

You could imagine more easy primitives like this built into Folk, like multiple-choice bubbles that the user fills in, or a slider, or whatever.

I also like how this starts to really use the fact that the code is printed on the program. Now the printed code is not just for documentation; you can start to modify the program by physically modifying the code (which is also nice since it's a different interaction -- using pen or pencil -- as opposed to the usual coarse moving pieces of paper around).

This also depends on a lot of the new 3D calibration stuff we've brought in, because it requires really good (1mm-accurate?) camera calibration to get the exact camera slice of the field to OCR. (and it would be nice but not as necessary to also have good projector and stereo calibration so you can call out that field with a projected outline if you want, or have even richer back-and-forth interactions)

=== Smoothing ===

Mason has been testing [[https://github.com/smj-edison/folk/commit/c4ee3e45e6992d19e76fcacea8473bb3974392f0|some pose stabilization logic that he has on his fork.]] 

> The idea is to use a continuous average, but if the movement is too large to jump to the new location… Right now I'm just averaging the matrix coefficients, so I'm sure there's better ways to interpolate…
>
> I'm not completely satisfied with how it's designed right now as it's a linear interpolation of the matrix coefficients. I'm wondering if I could do something like a quadratic or hermite spline of position + rotation as quaternion

It reduces jitter a lot; probably most useful for the code editor where jitter gets very annoying. Videos are on Discord if you're on Discord: https://discord.com/channels/956758212152025098/956765077883744266/1334712969853931550

(Omar: Some interesting questions about what the right policy here is. It does look like it adds latency, and maybe the right behavior is application-dependent, which is why we haven't tried it ourselves. I need to look at this again once some of the perf improvements land.)

=== Edit running programs ===

[[https://x.com/_rpf|Rob Fielding]] in New Zealand made a little editor hack to let you "edit a running program by 'touching' it on to the whisker":

<code>
When /page/ is a keyboard with path /kbPath/ /...anything/ &\
   /page/ is an editor &\
   the /kbPath/ cursor is /cursor/ {

  set id "$page$kbPath"

  When $page points up at /target/ & /target/ has program code /runningCode/ {

    Wish $target is labelled "targetting..."

    Hold "cursor$kbPath" {
      # Claim the $kbPath cursor is [list 0 0 ]
      Hold "code$kbPath" {
          # Claim $id has editor code [deleteToBeginning $editorCode $cursor]
          Claim $id has program code $runningCode
          Claim $id has editor code $runningCode
      }
      lassign $cursor x y
      Claim the $kbPath cursor is [list 0 $y]
    }
  }
}
</code>

> This editor hack allows you to edit a running program by "touching" it on to the whisker - you need to move it away again otherwise I think it fights with resetting the editor code.  Also having two running codes can sometimes lead to weirdness.  I added this hack in virtual-programs/editor.folk I wanted to add it as a separate program, and make it a "mixin" but I couldn't seem to get the scope of the code editor correct and the cursor misbehaved.  Next, I'd to make the editor more useful to me I want to make the viewport scrollable and add vim-keys. 
==== New parallel evaluator ====

Omar: Progress on this has been limited this month. A lot of looking at these specific spans in Tracy where there was a blink-out of a tag outline:

{{:newsletters:pasted:20250131-195852.png?400px}} {{:newsletters:pasted:20250131-195919.png?400px}}

You can see the drawcount plot mentioned at the bottom -- when that dips from 4 (the edges of the page outline) to 0, you know you have an unwanted blink (assuming the page was visible on camera the whole time). Tried adding more messages so we can see the actual statements that are evaluating (and getting stuck on).

You can see some of the red spans in the above images where threads are getting preempted while running Folk When code, which delays all downstream statements, and they might be delayed for 10ms+ while the thread isn't on the CPU, which means you'll miss the frame deadline and the page blinks out! Here's another one (the SAY I'm hovering over and the HOLD above it have both been preempted and delayed in that way):

{{:newsletters:pasted:20250203-031534.png?500px}}



After getting back in late January, I ran into issues running the real-time scheduler branch from last month (it's hard to find & kill those threads by process name? do they dominate their CPUs too much?). So I cleaned up the detection of past versions (Folk now makes an explicit folk.pid file) and have been rethinking our thread spawning/killing policies.

Again, I think a lot of the blinking is just because we have too many threads for the number of CPUs and they're preempting each other mid-task, which means that the evaluation doesn't converge in time for the frame. So need to manage the number of threads in a more reasonable way. Maybe based on I/O-bound-ness (is the thread in Linux Running or Blocked state?) instead of just pure execution time which was always a hack...
==== Portable Folk gadget ====

Omar: I've been working on a new portable Folk gadget that uses off-the-shelf parts, so anyone will be able to build/buy one, along with some other improvements: built-in battery, trigger button you can press with your finger, built-in speaker.

All the components are in place and I'm starting to fix up the software to make Folk work on the platform. 

{{:newsletters:pasted:20250131-194410.jpeg?250px}} {{:newsletters:pasted:20250131-194400.jpeg?250px}}

This revision uses the Orange Pi 5 instead of the Raspberry Pi 5, and a stereo USB camera instead of Pi wide-angle camera, which has caused various random issues.

=== Network setup ===

Annoyingly, the Orange Pi 5 doesn't have built-in Wi-Fi, so I'm using an Edimax EW-7611ULB USB dongle (RTL8723BU chipset) I had lying around. (Another problem is that we have really limited USB ports on the OPi 5 -- it only has two, we're already using one for the camera, so with the Wi-Fi dongle, there are no free ones left for keyboard or anything.)

  * Set up Netplan, struggled to get it on my network, realized that it only supports 2.4GHz Wi-Fi.
  * Still struggling to make it do ad-hoc or access point mode by default
    * (ideally it would work in concurrent mode, so could both get on known existing networks and make its own for wild environments:)
      * {{:newsletters:pasted:20250131-200028.png?400px}}

=== The stereo camera ===

The stereo camera works! Its field of view is huge, though (the subregion we actually care about in each stereo image is mostly the red-circled region where we can project):

{{newsletters:foooo.jpg?400px}}

[[https://github.com/FolkComputer/folk/pull/189|Added crop support]] to just take the center part of the left image for now so we don't waste time processing the rest and can start trying to calibrate right away.

Haven't been able to calibrate yet, though; I think I'm having issues with exposure so I can't read enough tags to calibrate.

=== Current state & next steps ===

Overall, the new gadget revision is in OK shape, although not done yet:

  * It has a working trigger button! That'll open up a lot of fun point-and-drag and lasso interactions. Here I'm having the 'marching ants' rectangle turn from white to green when I press the button down:
    * {{newsletters:img_7370.mp4}}
    * This is the program that implements that (I just put it in setup.folk, since we don't have physical programs here yet):
      * <code>
exec gpio mode 7 up

When display /disp/ has width /w/ height /h/ {
    When the clock time is /t/ {
        set pressed [expr {![exec gpio read 7]}]
        set color [expr {$pressed ? "green" : "white"}]
        Wish to draw a dashed stroke with points [list [list 0 0] [list $w 0] [list $w $h] [list 0 $h] [list 0 0]] \
          color $color width 10 dashlength 40 dashoffset [expr {fmod($t, 10)*-120}]
    }
}
</code>
      * it basically just sets a pullup resistor on GPIO 7 (where I plugged the button in) & shells out to [[https://github.com/orangepi-xunlong/wiringOP|wiringOP]] every frame to read that pin; to avoid the multi-ms overhead of shelling out, would be good to make a Tcl-C library to call into wiringOP and do this (I think the C header and library is already on the device anyway!)
  * It has a working battery, so you don't need to plug it in & carry a long extension cord! I haven't tested battery life but probably at least an hour? And I can even get live reports over serial about battery percentage, just like a phone or laptop shows battery percentage:
    * {{:newsletters:pasted:20250203-034338.png?600px}}
    * (will be fun to get this info into Folk and display it ambiently in the corner or on the marching ants border or whatever)
  * It's really heavy and kind of big, notably heavier than the older gadget (because it has 3 18650 batteries and a heavier mini projector with a full chassis). Not sure what to do about this. Maybe go back to the concept of the detachable battery pack (a standard USB-C battery pack would be ideal, but I don't think I could quite get that to work without more custom power stuff).
  * It has a speaker! I haven't gotten this to work yet (I haven't gotten the Orange Pi 5 3.5mm audio jack to work at all. Linux audio struggles). Also curious how loud it'll be without an amplifier.
  * The biggest issue is that I haven't been able to calibrate it yet, but I just need to spend more time on that...

Finally, need to finish physical design (I want to put on the back panel so people feel more comfortable handling it, less wires poking out, fix the speaker in place, maybe use a different camera, make the grip feel less precarious since it only has 1 bolt) and publish on GitHub.
==== RFID localization ====

Omar: Good progress here. My networking code (writefull and readfull) was broken, which was causing failures after a few seconds, and it seems reliable now. 

Hit rate looks pretty good when live-streaming data from the IB radio -- see how almost all hops are green:

{{newsletters:img_6313.mp4}} {{:newsletters:pasted:20250131-200127.png?400px}}

(Rounds are made of hops; each green hop is a valid roundtrip to the RFID tag where the result passed checksum, so they are guaranteed good data for localization + they tell you the tag ID. Right now, a round takes about 250ms and a hop takes about 15ms, so we're not quite at real-time interaction rate yet, but hopefully we can get there.)

Next step is to set up the OOB radio -- can just have it read at 915MHz to start (and that'll help make sure the right stuff is happening on IB & that we can sync).

==== Friends and outreach ====

  * Small open house at the end of January:
    * {{newsletters:img_7137.jpeg?200px}} {{newsletters:img_7140.jpeg?200px}} 
===== What we'll be up to in February =====

  * **Our next Folk open house is [[https://partiful.com/e/RdYuL5NiNl8XI6AmebDS|in the evening on Monday, February 24]], at our studio in East Williamsburg, Brooklyn.**
  * Omar: Finish the new gadget revision (back panel, audio support, calibrate camera) and document it on GitHub for people to try.
  * Omar: Set up the out-of-band RFID radio and maybe start on localization.
  * Omar: Figure out thread management on the new evaluator so work items don't get delayed for 10ms at a time (hopefully will fix blinking)
  * Andrés: Work on video
  * Andrés: Dot detector port to work under 3D calibration
  * We'll be speaking at the [[https://interactiondesign.sva.edu/|School of Visual Arts' Interaction Design]] department’s 2025 winter/spring lecture series. The talk will be recorded so we'll link it here as soon as it's online.
  * We'll also be showing Folk as an interactive prototype in a workshop in the Design in Action course at Columbia GSAPP. The students' [[https://gsapp-cdp.github.io/archive/|past work]] is very Folk-adjacent so we expect to get a lot of interesting feedback and are excited to show an enthusiastic technical audience Folk up close.
  * Meeting with the people from [[https://spatialpixel.com|Spatial Pixel]] this week; excited to compare approaches and see where we can collaborate.
===== Links we've enjoyed =====

==== Omar ====

  * https://ninenines.eu/articles/erlang-scalability/
  * https://queer.party/@bri_seven/113889078107377452
  * https://www.realworldtech.com/forum/?threadid=189711&curpostid=189752
  * https://viewsproject.wordpress.com/2025/01/30/reviewing-latin-the-roman-writing-system/
  * https://www.particleink.com

==== Andrés ====
  * [[https://www.youtube.com/watch?v=j2elKdupD6c|Localhost: Nolen Royalty's One Million Checkboxes]]