Table of Contents

March 2025 newsletter

(Our next Folk open house is in the evening on Sunday, April 27, at our studio in East Williamsburg, Brooklyn. Come by!)

What we've been up to

Demos and applications

General system improvements

Folk gadget & QR scanner

Omar: as a side project for QR Show (see retrospective), I wrote a QR scanner program in Folk for the handheld gadget.

QR Show was my excuse to (*extremely* last-minute) hack together a prototype that I've been thinking about for years…

point at the QR code, press trigger button to scan, then get output next to it; here, the QR code just says “Hello world!”



It kind of half-seriously answers the critique of, like, restaurant menu QR codes, where scanning a QR code locks you into your phone & isn't a process that people around you can participate in. Here, you scan the QR code in a 'social' way, and 'in situ', and other people can see/participate too

This is an interaction I'd had in mind for a few years, since I first got the Nebra AnyBeam in like 2021, so it's great to get to play it out now.

And it was not hard to write – came up with the idea and implemented it in like 100 lines of code in a couple of hours before showing up. (This is also a consequence of all the work we've put into making stable gadget hardware over the last few months; the gadget is a pretty good hardware platform for this in its current state as is!)

Here, I'm scanning a URL that Owen put up for his project (https://owentrueblood.com/experiments/autonomous_response/), and that web page gets rendered (using exec wkhtmltoimage) and projected by Folk on the right:

Since the show, I've been making small improvements. It now loads & renders the web page in the background instead of blocking on wkhtmltoimage, it lets you clear by holding the trigger button, it decodes continuously while you hold the button so you get more chances to capture the QR code…

Some future aspirations here:

The QR scanner is very different from most of the other stuff we've done, and it's a great test for Folk as a system:

Other gadget stuff

Wi-Fi issues

I mentioned this last month – I've had such trouble getting the gadget (with its Orange Pi 5 and USB Wi-Fi dongle) connected in wild environments (where it doesn't know the Wi-Fi upfront). The self-access point mode on the dongle just doesn't seem to work.

Successful calibration

Following up on last month, I did successfully calibrate this (new, ELP dual-camera) gadget.

(it hasn't been a priority, but I figured I might as well since I've been bringing it out so much for QR, and everything was in place already to try it)

wasn't too bad! the calibration is a bit off – a couple of centimeters – when trying to use it on top of folk0, but I hope some of the physical improvements below will help.

Upcoming

There's still a lot to do on the gadget. Bringing it out so often for QR demo has helped clarify what the outstanding issues are, at least:

folk2

Omar: Still a lot of work left on the new interpreter – I've been biting off more and more as I continue to encounter blinking problems and dropped frames – but I'm also increasingly excited about the potential frontier of performance (making Folk as responsive as possible), and I think the new system will be semantically better and cleaner, too (and more flexible for people to do new things). A lot of the changes are things we've wanted to do for years.

A few different branches are in play right now:

C collect

I noticed in the Tracy profiles that we were spending a lot of time doing recollections, which is implemented in this both-elegant-and-hacky pure-Folk way in folk2: 

When when the collected matches for /clause/ are /matchesVar/ /body/ with environment /e/ {
    Wish to collect matches for $clause
}
When /someone/ wishes to collect matches for /clause/ {
    When /someone/ wishes to recollect matches for $clause with timestamp /ts/ {
        Say the collected matches for $clause are [Query! {*}$clause]
    }

    # the Hold! should hopefully dedupe
    set Recollect! [list apply {{clause} {
        HoldStatement! [list recollect $clause] \
            [list virtual-programs/collect.folk wishes to recollect matches for $clause with timestamp [clock milliseconds]]
    }} $clause]

    When {*}$clause {
        {*}${Recollect!}
        On unmatch ${Recollect!}
    }
    {*}${Recollect!}
    On unmatch {
        HoldStatement! [list recollect $clause] {}
    }
}

Notice how each recollection has to do multiple passes through the workqueue and Tcl and the database (new statement for the collection triggers the When, which triggers the HoldStatement!, which triggers the Wish, which triggers the Query!, which finally gets dispatched to collectors by doing the Say).

(Even in folk1, Collect is implemented natively in C, so it's not surprising that this would be a step backward in performance.)

So I made a new folk2 branch where Collect is implemented in C, which should cut almost all of that database traffic.

It feels a lot more responsive compared to main folk2. The major bug (and why we haven't merged it into main folk2) is that it seems to introduce long periods where pages just blink out and vanish, like this:

Still need to debug this.

Page quads

This is the most important branch and probably the biggest potential performance improvement / hold-up to merging folk2. I've mostly been working on this for the last couple of weeks.

The goals here are to 1. render page graphics to textures in memory and composite those instead of drawing directly to screen buffer and 2. always represent draw coordinates in 'meters with respect to the page', all the way until the draw op hits the GPU. That means that almost all draw statements will be stable across frames, since the outline of a page doesn't change in that coordinate frame. That means we can hopefully get rid of like 80% of database traffic every frame, because we don't have to recompute every draw statement when a page wiggles slightly.

So this is both a performance improvement and a breaking user-facing API change (one we've badly wanted to make since we introduced 3D calibration late last year).

(and it's a correctness improvement – it means we should be able to draw text non-distorted in the real world, not relying on the projector pixel grid as we do now.)

Broke up the Vulkan gpu.folk more cleanly into gpu.folk, gpu/images.folk, and a new gpu/writable-images.folk. (Had to improve C import/extend technology so we could access the gpu C code from the other modules – you can even use global variables from another module now, at least by a _ptr() getter I added. Had to figure out the right Vulkan render pass and image format stuff so we can still have single pipelines that we reuse across both image-target and screen-target draw operations.)

Made more of the operations run concurrently (and in arbitrary threads) in normal Whens instead of one big GPU loop, which should make the system more robust (and mean that stuff like new-shader compilation doesn't freeze the whole system) (and let others extend it further in Folk modules and get access to state that's just stored in statements instead of loop-local variables), but does mean we need more locking and fences and semaphores and stuff to synchronize the GPU actions.

It's cool to think about how much draw logic can be implemented using idiomatic Folk constructs, and it's a good test of how expressive we can make Folk.

Lots of weird images like this to test drawing in an image and then drawing that image onto the screen:

(here, was also encountering a bug with line drawing that turned out to be that I forgot about triangle facing / vertex order.)

Laptop universe and safety concerns

I've been testing a lot on my laptop, which has produced some side quests:

These all have a similar energy: they both are part of a whole constellation of safety concerns I have about Folk programs. Basically, a Folk program needs to be cancelable, it needs to be hot-swappable (that is, you retract it, clean up all its stuff, then assert a new version of it that presumably wants to do a lot of the same things). That's a pretty high bar! Unix can do it with processes, but we want to share more state than Unix processes do. How do you release all the resources cleanly, do it in the right order, etc?

Another class of issue is queueing issues, where a thread (usually the sysmon thread) freezes and the rest of the system gets overrun by clock events and/or garbage and crashes:

Inter-thread shared objects

(I still don't know if we'll merge this, but) I've been noodling some more on the concept of inter-thread shared objects (see last month), so we don't have to copy and reparse (or cache-lookup) term objects every time we use a statement in a different thread.

I've been trying an approach from last month (which does feel actually-implementable) where we only /temporarily/ shimmer objects if they're inter-thread-shared, which only requires some pretty mechanical changes to all the shimmering functions in the Jim Tcl interpreter.

But there are so many trade-offs (now shimmering is just temporary for shared objects, now we have to track lots more stuff, etc) that I don't know if it's worth it compared to the current folk2 approach of just caching everything. Hmmmm.

Mason's dive into not shimmering shared objects

Mason: I've been tinkering with preventing shared objects from shimmering, so that they can be safely shared across threads. It's been fairly straightforward, though there's some edge-cases I'm still trying to figure out (programs and variable names still need to shimmer so they can do fast execution and lookup, how to deal with program transfer?). I also haven't tested the code yet, so maybe it'll all blow up. ¯\_(ツ)_/¯

Outreach and other systems

Open house

We had our usual open house on Wednesday, March 26:

Other visitors and interactions

What we'll be up to in April

Omar

Andrés