Folk Computer

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== April 2024 newsletter ======

===== What we've been up to =====

==== Demos ====

  * [[https://haiperspace.com|Jacob Haip]] has been doing microcontroller integrations with Folk, so you can press a physical button or slide a real slider or light up real LEDs: he wrote a great [[notes:mqtt|note about these integrations and his use of the MQTT protocol]]
    * Omar: I really liked this example of 4 buttons that can play different songs on Spotify:
      * {{notes:button-kid-press.jpeg?300px}}
      * More specific and approachable and fun physical interface than pages, little kid can play with it like a normal toy, but Folk makes it easy to 'rewire' and get into reprogramming (and to cross with other sensors and actuators in novel ways) because it's all Folk programs in the backend:
        * {{:newsletters:pasted:20240430-200353.png?500px}}
  * [[https://myfirstandlastname.com|Ashwin Agarwal]] made this program that lets you declare something ''an enemy of mega man''. When the mega man program points at the enemy he'll start shooting at it:
    * {{newsletters:img_5205.gif}}
  * Andrés:
    * I added a word search game to my collection of OpenCV enabled games. It was a fun challenge aligning the printed out characters with their virtual positions.
      * {{newsletters:wordsearch.gif}}
    * I took an afternoon to explore tools for playing with text in Folk:
      * The GIF below demonstrates 3 programs working together: one renders the text "hello" and the other two are rotating dials that allow you to change the font family and font size of the text.
        * {{newsletters:adjustablefontdemo.gif}}
      * This GIF is a program that shows all the fonts installed on the system at various sizes. Useful for testing their readability.
        * {{newsletters:fontsizepreview.gif}}
==== Live USB ====

Omar: as part of getting the CNC project (next section) polished and distributed for a wider audience, I've been making a [[https://github.com/FolkComputer/folk-live-build|self-contained Folk live USB for amd64 PCs. It's mostly ready! You can download it now!]] 

It just uses whatever is display 0 and whatever is webcam 0, so the live USB stick (plugged in on the right) runs straight on a cheap laptop:

{{:newsletters:pasted:20240501-080545.png?300px}}

(You can think of this as a replacement for most of the traditional [[https://git.folk.computer/folk/about/|normal manual Folk setup process]] where you install Linux, install a bunch of dependencies, clone Folk, etc.)

It's in pretty good shape now: you download the .img file, flash it to a USB stick, and configure/update the Folk system on the USB stick from your normal computer. There's a writable FAT32 partition that contains all the Folk stuff that isn't the core Linux OS, so it's easy to update both the Folk repo and any custom programs you want to preinstall:

{{:newsletters:pasted:20240502-011852.png?600px}}

Then you plug the USB stick into the PC that you want to run Folk, and the PC should boot right to a working network-connected Folk as long as you configured Wi-Fi and have /dev/video0 and display 0.

Some problems I encountered while building the live USB:

  * Adding a writable FAT32 partition (so we don't have to completely rebuild the image every time we want to change anything, & so the end user can set their Wi-Fi on there and add custom programs) was a terrible process
    * had to reverse-engineer a lot of [[https://twitter.com/rsnous/status/1781742786909184171|what Debian live-build does]] / how live USBs work, because they don't really support this out of the box, live USBs still use [[https://twitter.com/rsnous/status/1780343861472059467|live CD infrastructure]] that is all immutable. weird filesystem layouts and boot layouts that work in practice but make no sense in theory and can't really be changed. have to generate a non-bootable disk with live-build, mutate the partition table ourselves, and generate our own EFI boot stuff
    * was excited about the live-build/live-boot [[https://manpages.debian.org/testing/live-boot-doc/persistence.conf.5.en.html|automatic detection of ''persistence'' partitions]], realized it basically doesn't work with FAT32 because the permissions don't work to make it user-writable, gave up and implemented my own hacky script at boot to mount the partition (which also required understanding more of live-build's stages and different kinds of hooks)
    * although the writable partition has ended up being pretty cool and useful and inspiring
      * way easier to update the live USB / live image and iterate on it since the OS doesn't need to be reflashed
      * makes Folk feel much more stable and appliance-like; I can just boot an extra Chromebook into Folk to test something and know exactly what I'm depending on, not worry about messing up the PC, know that if I reboot it'll come up without needing to be touched
      * feels kind of like microPython or something in how you can program it like a USB drive
  * Memory leak freezes/crashes Folk on Chromebook for some reason
  * Wi-Fi setup is finicky and slows down boot, had to do a iwctl scan before connecting to make it mostly reliable, still slow though

Remaining to do on live USB (although it's already pretty useful):

  * Support configuring which display & camera to use at minimum, ideally support multiple (this is important for getting laptops to be useful Folk systems, since they already have a built-in webcam and display that you wouldn't want to lock onto)
  * Build out the setup process (calibration, printer setup, maybe interactive Wi-Fi setup) (although there will probably always be an [[https://social.omar.website/@omar/statuses/01HW6GAPSTB294WG193GG5WDCM|escape hatch to Linux terminal]], we can't cover all problems with custom UI); get this setup persisted on the writable partition across reboots
  * Build/test the process of permanent installation to hard drive -- we've just been using it in live mode so far
  * Debug memory leak
  * More testing (does MBR even work? have only tested EFI)
  * Maybe a Pi image?

==== folk-cnc ====

Omar: I've been porting [[https://owentrueblood.com|Owen's]] [[https://github.com/jmpinit/projected-toolpath-preview|projected CNC toolpath preview tool]] from last fall to Folk, so you should be able to flash Folk to a live USB, clone folk-cnc onto the live USB, and boot a PC (that's connected to a webcam and projector) right into Folk with the CNC preview application pre-loaded.

Then, at least for now, you use a Web frontend much like the one in Owen's system to set up the preview: you calibrate the preview to your projector & camera & CNC machine and the material that you want to cut, and you upload a gcode file to preview. Here's the Web UI:

{{:newsletters:pasted:20240501-190329.png?500px}}

And here's a video of the process of adjusting the tag's position and size so that it sits on the material we want to preview:

{{newsletters:mymovie.mp4?600px}}

You can see the coordinates of the calibration tag in camera-space change live (the tag is continuously being re-detected by Folk's built-in AprilTag detector). It's nice that they don't fluctuate more than a pixel in any direction, mostly. They're pretty stable.

Still need to port tool calibration (move your CNC machine to the 4 calibration points), homography calculation, gcode loading and preview, and then we should be good to write this up and announce it. (It all seems straightforward, although there are questions about how much to do on the JS side vs. the Folk side.)

(Later, I think we want to move the entire UI into Folk, so you can use pages to do all this, with the ultimate goal that you can do more of the design process inside Folk as well and have all kinds of physical-digital tools available on the CNC machine bed.)

==== Handheld Folk gadget ====

Omar: I've been working a little on a handheld Folk gadget, based on some [[https://www.ultimems.com/download/hd305d1-c1_rev1_0_20231005_english/|Ultimems HD305D1 focus-free laser pico projectors]] that [[https://github.com/antimatter15|Kevin Kwok]] and some of us specially group ordered from Taiwan. (Very similar to the [[https://www.nebra.com/collections/nebra-anybeam|Nebra AnyBeam]]; these are all hard or impossible to get retail now, I think, which is why we had to do a group order. I think there are slightly bigger LED pico projectors that are easier to find that could also work, although they're not focus-free.)

{{newsletters:a3533a42-4920-4845-b611-2c708001a9e3-1536-00000356ddf48212.png?0x300px}}
{{:newsletters:pasted:20240430-190113.png?0x300px}}

Each of these white+tripod units is a completely self-contained Folk system, including camera, computer unit, and projector, only needing external wall power to work. The one in the right image is pointed at some Folk programs on the cardboard and is executing them (you can see 3 of the 4 programs are running and displaying outlines), although they're not calibrated right yet.

The rough parts list is pico projector ($300-400), Raspberry Pi 4/5 ($100), Pi camera v2 ($30), Pi power supply, long 25ft AC extension cord (I was in a rush and didn't want to mess with batteries), & various USB-C and micro-HDMI cables that are short and right-angled to minimize space use. And a 3d-printed case and tripod...


Weird issues I encountered while building the Pi gadgets:

  * Had to hard-code HDMI1 (display number 1) instead of HDMI0 for some of the gadgets (change to Vulkan setup) because that was the only port that was physically close enough
  * Needed a new dtoverlay boot param, something with VideoCore4 Linux setup has changed with newer kernels?
  * Pi cameras are not like USB webcams: they require libcamera instead of raw v4l2 (I had to compile and use the libcamerify wrapper that uses LD_PRELOAD to provide a v4l2 interface) //and// they don't output MJPEG (I had to add support to Folk for whatever YUV format they emit)
  * Known memory leak in Folk (which crashes PC but is usually ok since the service restarts itself there) makes the Pi completely freeze, not even accessible to ssh, and the service doesn't restart, so the Pi goes down after 10-20 minutes usually and needs to be power-cycled
  * Calibration does not work, the pico projectors are seemingly too dim for our traditional graycode stripes calibration to resolve
  * Physical setup is janky, tripod is kinda ziptied in, cases break open (it's very hacked-together by poking holes and screwing things into existing case and using existing parts), people are scared to pick the things up or don't know how to interact with them safely

Remaining to do:

  * Finish hardware (case/handle) design
  * Fix calibration or come up with a new calibration scheme (or finish up 3D calibration),
  * Fix memory leak hard freeze: it should at least reset itself


(I'm working on a more unified case with a built-in [[https://www.thingiverse.com/thing:1966894|hand grip]] that should make the gadget much nicer and more approachable to pick up. Maybe a [[https://www.thingiverse.com/thing:3604809|trigger button]], too -- that just feels appropriate. It would be cool if the case was designed so you could either hold it as a handheld 'barcode scanner' gadget or dock it as a lamp head, and so that you could either plug in a wall power cable or attach a USB-C battery.)

I have some theories about interesting ways to interact with it -- see also [[https://www.lumen.world|Lumen]] -- and I think they require having a comfortable handheld, being able to dynamically move it in space to get the brightness/resolution you want, and having it be spatially aware. I'm hoping that will mitigate how dim it is at a distance (you'll continuously move it closer/further depending on your goals).

This all motivates the 3D calibration work a lot more, since you really want to calibrate the gadget intrinsics, not a fixed plane in front of the gadget (since the gadget should constantly be moving).

==== Parallel Folk (folk2) ====

Omar: continuing work on folk2, the rewrite of the Folk evaluator to run programs in parallel (also cleaning up a lot of the code organization and internal & external interfaces, [[newsletters:2024-02#parallel-evaluator|as mentioned in previous newsletters]]). The hope is to cut latencies by a lot and to scale better as more programs get added.

This stuff is all on the folk2 repo and not available in mainline Folk yet:

=== Performance monitoring ===

== Thread monitoring ==

I built a thread monitor, so you can go to ''/threads'' and see a snapshot of all threads in the Folk process and what they're doing (C stack trace & current Folk When block if applicable) and whether they're blocked or not:

{{:newsletters:pasted:20240501-161653.png?400px}} 

([[https://poormansprofiler.org|"poor man's profiler"]] energy)

You can see from the ''Run when'' descriptions above that thread 1 (3394) is running the camera loop, thread 2 (3401) is running the Web loop, thread 4 (3403) is running the GPU loop. (note that these are not //static// assignments, though! I never assigned those processes to threads by hand, I just wrote the straight-line Folk programs that each blocked on different stuff, and the folk2 scheduler assigned them to threads in the thread pool at runtime.)

You can see by squinting at the user stack traces below that thread 1 (3394) is running the camera loop, thread 10 (3476) is running an AprilTag detector worker thread, and thread 11 (3497) is running some Vulkan-related thread:

{{newsletters:2d4afa4c-4b79-4f73-84fd-5e0442b6b4bd-60804-000004a3f10e86e7.png?0x300px}} {{newsletters:1a400d61-c064-4a93-9d05-2921cc87efb6-60804-000004a43ae08f2c.png?0x300px}}

(Note that threads 10 and 11 are part of the folk Unix process, but they aren't Folk worker threads; they were independently spawned by the C libraries libapriltag and libvulkan)

(next step: use monitoring like this to dynamically adjust the thread pool up and down so that the system is always responsive -- it should be possible to block all the threads on arbitrarily long I/O and just spin up new ones to handle incoming work, then throw away threads once the I/O has all completed)

== perf events ==

After the thread monitor looked reasonable and had no clear things to fix, I added support for providing tracepoints for [[https://www.brendangregg.com/perf.html#DynamicTracing|Linux perf]] from folk2 programs.

I wanted to count inter-frame delays on various subprocesses: how long does it take between one camera frame and the next? between one AprilTag detection and the next? one GPU draw and the next? How many of each of these are we doing a second? is it what I'd expect (30Hz or 60Hz)?

perf should provide a whole suite to count and plot and correlate things once we give it simple tracepoint hooks from our program, and it saves me needing to make custom thread-safe data structures and functions in Folk to maintain event counts and so on.

Here's how you provide an event to perf in your Folk code:

{{:newsletters:pasted:20240501-204451.png?700px}}

For now, when you run Folk, it spits out commands you can run to add the probes to perf (because the C library name is random and different each run):

{{:newsletters:pasted:20240501-205140.png?500px}}

And here's how you [[https://developers.redhat.com/blog/2019/04/23/how-to-use-the-linux-perf-tool-to-count-software-events|count occurrences of that event]] per second:

{{newsletters:perf-stat.png?700px}}

(although it took a bit [[https://social.omar.website/@omar/statuses/01HWBB79447C991Q130YC7FERP|to even get perf to work]] on what should be a bog-standard amd64 Debian system... it [[https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2019247|wouldn't detect libtraceevent]] to record user events, and I had to recompile it myself)

Counting perf events immediately paid off, because I figured out from the above that (part of) the reason that folk2 is slow is probably that it does AprilTag detection in an infinite ''while 1'' loop, 100+ times a second, rather than just when the camera frame changes, 30/60x per second.

and the reason I made the detection process an infinite loop was because the AprilTag detector, like all C objects, was only callable from the thread it was created on. So I decided it was time to do a C refactor I'd been meaning to do for a while, to make the detector callable from any thread, so we can call it from a normal When body that responds to the camera frame (which can get scheduled onto any thread).


=== C refactor ===

Changed the C FFI to produce 'C library objects' that get returned from ''$cc compile'' and have all C procs from that compilation bound as methods. Refactored all the C usage (gpu, camera, apriltag detector, web monitoring hooks, etc) accordingly. The way it's now used in practice is

<code>
set cc [C]
$cc proc add {int a int b} int {
  return a + b;
}
set addLib [$cc compile]

puts "2 + 2 = [$addLib add 2 2]"
</code>

(I'd already previously reworked the C FFI in folk2 so that C procs are bound as methods on the ''$cc'' object after running ''$cc compile'', instead of just popping unprefixed into the current or global namespace. This is a further refactor that basically splits the 'compiler object' and 'library object' roles.)

The advantage of the C library objects is that they can be cleanly and transparently shared across threads in Tcl using [[https://github.com/osnr/db/blob/4ce5a5acdba23f38d0ba47969536f08b726a9b54/lib/c.tcl#L512|some trickery with ''unknown'']], because they're guaranteed to have no internal state, they're practically just a bundle of function pointers (plus the C glue code to put the commands into the Tcl interpreter, which you can run in each thread's local interpreter on demand).

Now you can use the C library object inside any When where it's in lexical scope (or matched out of a statement), without worrying about what thread that When body is running in; it all works as you would expect.

(I think both C refactors are part of a general movement away from stateful objects and global state in the Tcl interpreter [e.g., registering global Tcl command for each C function] and toward immutable data in normal variables. That data can then be passed through statements and through lexical scope, and the program doesn't have behavior dependent on running the right things in the right order. I've wanted to do this for a while, but it's really forced on us by multithreading :-))

== The object pattern ==

There are still some ergonomic issues with the C FFI that would be good to solve. The biggest one is how to codify the 'object pattern', where we build a [[https://jim.tcl.tk/index.html/doc/www/www/documentation/oo/|Jim OO]] object around a C library, and that Jim OO object has Tcl methods and Tcl state.

The object pattern usually looks like this:

<code>
class CameraManager {
    camLib {}
    width 0
    height 0
    camera {}
}
# ugly
CameraManager method unknown {procName args} {
    "$camLib $procName" {*}$args
}
CameraManager method init {w h} {
    # set camera member variable to C camera struct ...
}
CameraManager method frame {} { ... }

# ugly
set cam [CameraManager new [list camLib [$camc compile]]]

$cam init 1280 720
while 1 { set frame [$cam frame] }
</code>

It feels uglier now that 1. the C library object needs to get stored in the Tcl object so that Tcl methods can call into the C library and 2. we need to hook the ''unknown'' method if we want the Tcl object to also let its users call C methods and 3. the Tcl object doesn't transparently get shared across threads, since it's a Jim lambda reference. 

At the very least, it would be good to make a construct on top of ''class'' that factors out some of the common logic like the C library member, the ''compile'' call, and the ''unknown'' handler.

==== Friends and outreach ====

  * April open house:
    * Small open house this month: always fun showing Folk to friends old and new. (We may hold a bigger one as some of these larger projects merge in. We're about to hit 1000 programs, too.)
    * {{newsletters:april_open_house.jpeg?0x300px}} {{newsletters:img_7781.jpeg?0x300px}}

===== What we'll be up to in May =====

  * **Our next [[https://partiful.com/e/yPGAy7VccfYivP557hCn|Folk open house is in the afternoon of Sunday, May 26]] at our studio in East Williamsburg, Brooklyn.**
  * More side hacking on Folk gadget: calibration, hardware design
    * Maybe revisit 3D calibration for this
  * Finish up the CNC port, test it, and do a writeup/post (including end-to-end install instructions using live USB)
  * Continue to improve new parallel evaluator, especially performance, and port more functionality (text, images, Collect)
  * Maybe revisit RFID
  * Andrés will be showing Folk, specifically OpenCV game experiments, in an NYU ITP Project Fellows art show at [[https://tisch.nyu.edu/clivedavisgallery|The Clive Davis Gallery]] from May 23rd to May 30th.

===== Links we've enjoyed =====

==== Omar ====

  * [[https://rachelbythebay.com/w/2017/01/30/env/|Don't setenv in multi-threaded code on glibc]]
  * [[https://inpatientpress.bigcartel.com/product/small-press-tycoon]]
  * [[https://lwn.net/Articles/803421/|Please remember that the kernel is for running userspace!]]
  * [[https://journals.sagepub.com/doi/10.1177/01622439241245746|Coding Beauty and Decoding Ugliness: The Role of Aesthetic Concerns in Programming Practices]]
  * [[https://blurbusters.com/blur-busters-law-amazing-journey-to-future-1000hz-displays-with-blurfree-sample-and-hold/|Blur Busters Law: The Amazing Journey To Future 1000Hz Displays]]
  * [[https://news.ycombinator.com/item?id=33246186#33254142|"There is simply no single device which is 'the' camera"]]
  * [[https://twitter.com/xhfloz/status/1775181439526490329|"Anecdotally, when people describe what matters to them, second only to human relationships is usually the mastery of some demanding tool."]]
==== Andrés ====

  * https://shoppu.mochi.at/ — I love the aesthetic & very opinionated UI for this website.
  * [[https://www.happyfolding.com/instructions-arai-spinning_top|Origami spinning top]] — I've been looking at various designs for spinning tops as a potential way to make an all-paper dial interface in Folk.
  * [[https://www.instagram.com/reel/C5OScr6L9un/|Incredibly simple but mesmerizing projection feedback demo.]]
  * [[https://zine-machine.glitch.me/|Zine Machine]] — Use some clever CSS to turn a website into a printable 8-page zine.