Table of Contents

April 2024 newsletter

What we've been up to

Demos

Live USB

Omar: as part of getting the CNC project (next section) polished and distributed for a wider audience, I've been making a self-contained Folk live USB for amd64 PCs. It's mostly ready! You can download it now!

It just uses whatever is display 0 and whatever is webcam 0, so the live USB stick (plugged in on the right) runs straight on a cheap laptop:

(You can think of this as a replacement for most of the traditional normal manual Folk setup process where you install Linux, install a bunch of dependencies, clone Folk, etc.)

It's in pretty good shape now: you download the .img file, flash it to a USB stick, and configure/update the Folk system on the USB stick from your normal computer. There's a writable FAT32 partition that contains all the Folk stuff that isn't the core Linux OS, so it's easy to update both the Folk repo and any custom programs you want to preinstall:

Then you plug the USB stick into the PC that you want to run Folk, and the PC should boot right to a working network-connected Folk as long as you configured Wi-Fi and have /dev/video0 and display 0.

Some problems I encountered while building the live USB:

Remaining to do on live USB (although it's already pretty useful):

folk-cnc

Omar: I've been porting Owen's projected CNC toolpath preview tool from last fall to Folk, so you should be able to flash Folk to a live USB, clone folk-cnc onto the live USB, and boot a PC (that's connected to a webcam and projector) right into Folk with the CNC preview application pre-loaded.

Then, at least for now, you use a Web frontend much like the one in Owen's system to set up the preview: you calibrate the preview to your projector & camera & CNC machine and the material that you want to cut, and you upload a gcode file to preview. Here's the Web UI:

And here's a video of the process of adjusting the tag's position and size so that it sits on the material we want to preview:

You can see the coordinates of the calibration tag in camera-space change live (the tag is continuously being re-detected by Folk's built-in AprilTag detector). It's nice that they don't fluctuate more than a pixel in any direction, mostly. They're pretty stable.

Still need to port tool calibration (move your CNC machine to the 4 calibration points), homography calculation, gcode loading and preview, and then we should be good to write this up and announce it. (It all seems straightforward, although there are questions about how much to do on the JS side vs. the Folk side.)

(Later, I think we want to move the entire UI into Folk, so you can use pages to do all this, with the ultimate goal that you can do more of the design process inside Folk as well and have all kinds of physical-digital tools available on the CNC machine bed.)

Handheld Folk gadget

Omar: I've been working a little on a handheld Folk gadget, based on some Ultimems HD305D1 focus-free laser pico projectors that Kevin Kwok and some of us specially group ordered from Taiwan. (Very similar to the Nebra AnyBeam; these are all hard or impossible to get retail now, I think, which is why we had to do a group order. I think there are slightly bigger LED pico projectors that are easier to find that could also work, although they're not focus-free.)

Each of these white+tripod units is a completely self-contained Folk system, including camera, computer unit, and projector, only needing external wall power to work. The one in the right image is pointed at some Folk programs on the cardboard and is executing them (you can see 3 of the 4 programs are running and displaying outlines), although they're not calibrated right yet.

The rough parts list is pico projector ($300-400), Raspberry Pi 4/5 ($100), Pi camera v2 ($30), Pi power supply, long 25ft AC extension cord (I was in a rush and didn't want to mess with batteries), & various USB-C and micro-HDMI cables that are short and right-angled to minimize space use. And a 3d-printed case and tripod…

Weird issues I encountered while building the Pi gadgets:

Remaining to do:

(I'm working on a more unified case with a built-in hand grip that should make the gadget much nicer and more approachable to pick up. Maybe a trigger button, too – that just feels appropriate. It would be cool if the case was designed so you could either hold it as a handheld 'barcode scanner' gadget or dock it as a lamp head, and so that you could either plug in a wall power cable or attach a USB-C battery.)

I have some theories about interesting ways to interact with it – see also Lumen – and I think they require having a comfortable handheld, being able to dynamically move it in space to get the brightness/resolution you want, and having it be spatially aware. I'm hoping that will mitigate how dim it is at a distance (you'll continuously move it closer/further depending on your goals).

This all motivates the 3D calibration work a lot more, since you really want to calibrate the gadget intrinsics, not a fixed plane in front of the gadget (since the gadget should constantly be moving).

Parallel Folk (folk2)

Omar: continuing work on folk2, the rewrite of the Folk evaluator to run programs in parallel (also cleaning up a lot of the code organization and internal & external interfaces, as mentioned in previous newsletters). The hope is to cut latencies by a lot and to scale better as more programs get added.

This stuff is all on the folk2 repo and not available in mainline Folk yet:

Performance monitoring

Thread monitoring

I built a thread monitor, so you can go to /threads and see a snapshot of all threads in the Folk process and what they're doing (C stack trace & current Folk When block if applicable) and whether they're blocked or not:

("poor man's profiler" energy)

You can see from the Run when descriptions above that thread 1 (3394) is running the camera loop, thread 2 (3401) is running the Web loop, thread 4 (3403) is running the GPU loop. (note that these are not static assignments, though! I never assigned those processes to threads by hand, I just wrote the straight-line Folk programs that each blocked on different stuff, and the folk2 scheduler assigned them to threads in the thread pool at runtime.)

You can see by squinting at the user stack traces below that thread 1 (3394) is running the camera loop, thread 10 (3476) is running an AprilTag detector worker thread, and thread 11 (3497) is running some Vulkan-related thread:

(Note that threads 10 and 11 are part of the folk Unix process, but they aren't Folk worker threads; they were independently spawned by the C libraries libapriltag and libvulkan)

(next step: use monitoring like this to dynamically adjust the thread pool up and down so that the system is always responsive – it should be possible to block all the threads on arbitrarily long I/O and just spin up new ones to handle incoming work, then throw away threads once the I/O has all completed)

perf events

After the thread monitor looked reasonable and had no clear things to fix, I added support for providing tracepoints for Linux perf from folk2 programs.

I wanted to count inter-frame delays on various subprocesses: how long does it take between one camera frame and the next? between one AprilTag detection and the next? one GPU draw and the next? How many of each of these are we doing a second? is it what I'd expect (30Hz or 60Hz)?

perf should provide a whole suite to count and plot and correlate things once we give it simple tracepoint hooks from our program, and it saves me needing to make custom thread-safe data structures and functions in Folk to maintain event counts and so on.

Here's how you provide an event to perf in your Folk code:

For now, when you run Folk, it spits out commands you can run to add the probes to perf (because the C library name is random and different each run):

And here's how you count occurrences of that event per second:

(although it took a bit to even get perf to work on what should be a bog-standard amd64 Debian system… it wouldn't detect libtraceevent to record user events, and I had to recompile it myself)

Counting perf events immediately paid off, because I figured out from the above that (part of) the reason that folk2 is slow is probably that it does AprilTag detection in an infinite while 1 loop, 100+ times a second, rather than just when the camera frame changes, 30/60x per second.

and the reason I made the detection process an infinite loop was because the AprilTag detector, like all C objects, was only callable from the thread it was created on. So I decided it was time to do a C refactor I'd been meaning to do for a while, to make the detector callable from any thread, so we can call it from a normal When body that responds to the camera frame (which can get scheduled onto any thread).

C refactor

Changed the C FFI to produce 'C library objects' that get returned from $cc compile and have all C procs from that compilation bound as methods. Refactored all the C usage (gpu, camera, apriltag detector, web monitoring hooks, etc) accordingly. The way it's now used in practice is 

set cc [C]
$cc proc add {int a int b} int {
  return a + b;
}
set addLib [$cc compile]

puts "2 + 2 = [$addLib add 2 2]"

(I'd already previously reworked the C FFI in folk2 so that C procs are bound as methods on the $cc object after running $cc compile, instead of just popping unprefixed into the current or global namespace. This is a further refactor that basically splits the 'compiler object' and 'library object' roles.)

The advantage of the C library objects is that they can be cleanly and transparently shared across threads in Tcl using some trickery with ''unknown'', because they're guaranteed to have no internal state, they're practically just a bundle of function pointers (plus the C glue code to put the commands into the Tcl interpreter, which you can run in each thread's local interpreter on demand).

Now you can use the C library object inside any When where it's in lexical scope (or matched out of a statement), without worrying about what thread that When body is running in; it all works as you would expect.

(I think both C refactors are part of a general movement away from stateful objects and global state in the Tcl interpreter [e.g., registering global Tcl command for each C function] and toward immutable data in normal variables. That data can then be passed through statements and through lexical scope, and the program doesn't have behavior dependent on running the right things in the right order. I've wanted to do this for a while, but it's really forced on us by multithreading :-))

The object pattern

There are still some ergonomic issues with the C FFI that would be good to solve. The biggest one is how to codify the 'object pattern', where we build a Jim OO object around a C library, and that Jim OO object has Tcl methods and Tcl state.

The object pattern usually looks like this: 

class CameraManager {
    camLib {}
    width 0
    height 0
    camera {}
}
# ugly
CameraManager method unknown {procName args} {
    "$camLib $procName" {*}$args
}
CameraManager method init {w h} {
    # set camera member variable to C camera struct ...
}
CameraManager method frame {} { ... }

# ugly
set cam [CameraManager new [list camLib [$camc compile]]]

$cam init 1280 720
while 1 { set frame [$cam frame] }

It feels uglier now that 1. the C library object needs to get stored in the Tcl object so that Tcl methods can call into the C library and 2. we need to hook the unknown method if we want the Tcl object to also let its users call C methods and 3. the Tcl object doesn't transparently get shared across threads, since it's a Jim lambda reference.

At the very least, it would be good to make a construct on top of class that factors out some of the common logic like the C library member, the compile call, and the unknown handler.

Friends and outreach

What we'll be up to in May

Omar

Andrés