We've had a funding shortfall recently – we'd really appreciate it if you sponsor Folk on GitHub Sponsors if you can, so that we can keep doing this free and open work

We’ll be setting up at Rhizome World this Sunday, May 4 (at 161 Water Street in the Financial District). Feel free to say hello; we’ll also be running a Folk Computer workshop there at noon.

If you want to stop by the studio, our next Folk open house is in the afternoon on Saturday, May 31, in East Williamsburg, Brooklyn.

• (Andrés) I gave a small talk and workshop to Christina Huang's Learning Analytics class at NYU. I was so busy demoing that I didn't take any photos myself, but there was a photographer there from NYU's Educational Communication and Technology Wireframe Newsletter — when I get those back I'll share them here.

• (Andrés) We merged Rob Fielding's PR's to:
  • Re-activate Jacob Haip's blob-detection code, introducing a spruced up demo program:
    • This also means the laser capability is back! (Side note: I love the dog getting to play with the laser here … it makes me wonder what a Folk system specifically for entertaining / training pets could look like!)
  • Add Wish $this draws text "example" to the system, making text work like our general shape syntax:
  • Tools for creating subregions:

Omar: I spent like a day trying to add stereo camera support (was hoping to show it off at a party.)

The immediate motivation was extending the QR scanner gadget application from last month so it can actually lasso the QR codes (tightly fit projected marching ants outline to the QR code on the surface), which requires pose-estimating arbitrary QR codes in the wild, which we can't do with just one camera because we don't know the size of those arbitrary QR codes a priori (unlike AprilTags that are printed through Folk). So we need two cameras, I think.

A lot of the changes are just basic scoping of calibration and camera and tag code to actually support multiple cameras, instead of collapsing all cameras into one.

Also some changes to support multiple crop areas, so you can split the ELP stereo camera on the gadget into multiple 'virtual cameras'.

I still think stereo camera has a lot of potential to improve our baseline performance and level of calibration:

trying the stereo camera on folk0 (will also need some software patching which should be generally useful also – ability to address multiple cameras, ability to use cropping to split a single linux video device into multiple virtual cameras)


[notice the second – ELP stereo, same as in gadget – camera in the foreground on the right here:]


and then of course the multi-calibration lol (i think easiest is to just calibrate them all individually for now and then exploit the fact that we'd have extrinsics from each camera to the common projector to derive their relationship to each other)

advantages of stereo camera are we can have much more legit pose estimation and i think we might be able to run faster because we could do all the detection at lower resolution (because we wouldn't need the high resolution for pose estimation)

(also just prototyping for the gadget which has more applications for the stereo cameras)

Anyway, didn't get very far, but it's probably worthwhile for the gadget applications alone, but it could also be useful on more normal systems for the reasons above.

Omar: I haven't done much on the gadget this month. Mostly trying to get the projector to stay stable so the calibration remains valid (it swivels and wiggles around because it's only attached at one point). Added washers to the attach point for the projector because the bolt warps the plastic.

It looked like the old front panel had too short an aperture for the projector, cutting off a lot of the bottom of the image.

Printed a new front panel with a taller aperture to try to not block the projection image at all. Better, but still not tall enough!

It actually wiggles up and down as much as left and right… I wonder if I can clip it somehow. Need to think about physical changes I can make to the chassis to keep the projector locked in place.

Also still need to put this design on GitHub! I'm just not happy with it yet.

Omar: I would say work this month has been driven by three different pushes (not really separable that cleanly, but oh well):

1. Continuing the performance/anti-blink push from last month: this drove page quads / render-to-texture, trying to get rid of blinking both semantically and by improving performance

2. Run folk2 on my gadget: it quickly leaks memory and runs out of RAM (the gadget has much less RAM than desktops do), which made me try to hunt down memory leaks

3. Fix calibrate and other functionality on folk2 so we can actually merge it: render-to-texture is forcing us to make more changes to calibrate and editor at least to get them to work again

I feel pretty good about folk2 right now. Beyond fixing the memory leak, a lot of the remaining work is functionality work, honestly, where we need to get calibration and the editor running again under the new regime (as well as camera slices, animation, other smaller modules). I've been putting that off because I really want to fix the memory leaks…

I've spent a lot of this month working on the page quads / render-to-texture stuff that I started last month, which has produced huge performance improvements (almost more than everything else in folk2 combined). Really excited to get this merged now.

Weird design choice for now: I got rid of the implicit parameter that passes screen resolution into every shader/pipeline – you still need it a lot of the time, but you have to do it explicitly now. This is because pipelines can now be used both to draw to the screen and to draw to a writable image. It's also because we're really tight on push constant space for the glyph shader in particular (it's max 128 bytes and we now need to pass a 3×3 matrix into most shaders), and manually specifying and ordering all the push constants lets you save a bunch of padding space. Maybe this is fine, but we should also think about the higher-level shader notation idea again as a layer on top?

I added support for drawing text and images and AprilTags on the textures, which was hard to debug, so I added a /images endpoint that shows all textures in memory (including placeholder image 0, font atlases, page textures, projected calibration board texture, other images):

The ability to get 'the stuff projected onto this page' as an image feels really cool and useful in itself, like Folk is now a compositor like Wayland or macOS Quartz and it has internal semantically useful knowledge of what all the layers are. I think we could have some cool applications for this gpu→cpu image transfer capability, like compositing the rendered page graphics on top of a livestream of the camera, or or printing what's projected on a page.

(most writable image textures are square 1024×1024 by default, and then we just resample them when we project them onto the display, which is why they're weird and square in the debugger view here)

To cleanly do GPU texture destruction when a page is flipped over, I added support for per-statement destructors, where you can pass a -destructor parameter to Claim or Wish to run some code once it's guaranteed that no one is looking at that statement anymore.

I also use this to clean up camera images now, which feels really nice:

(instead of just timing out camera images after some fixed period of time, or doing it on match unmatch which is unsafe since it doesn't guarantee no concurrent accessors to the statement)

This involved some changes to statement acquisition behavior and some new constructs:

• When a When matches a statement, the statement is now acquired for the duration of the When block's execution, so the statement's destructor (if applicable) will not run until the When has terminated. This feels like the correct behavior.
• But Query!, which returns a static set of result statements, does not acquire any of those statements. So the destructor of any statement in the result set may run at any time once Query! has returned. So it'd be unsafe to access a camera image that you got from Query! without explicitly acquiring the owning statement first to make sure it sticks around (and that it's still around).
• So ForEach! has been added for both ergonomic and safety reasons: ForEach! is like Query!, but you pass it a code block as last arg; it iterates and binds each result statement and runs the code block. In other words, it's like a mix of Query! (imperative semantics, doesn't yield) and When! (acquires each matched statement, binds variables for you, safe to use terms in statement inside your code block)

  • set omarAdjs [list]
    ForEach! Omar is /adj/ {
      lappend omarAdjs $adj
    }
    puts "Omar is all of these: $omarAdjs"

  • (Andrés and Mason helped to name this one – I suspect we'll be using it a lot)
• Manual StatementAcquire! and StatementRelease! have also been added if you want to use those with Query! (some of your queried statements may be invalid, though, and you'd have to ignore/toss those)

It turned out that our implementation of bilinear interpolation is pretty broken if rotating at an exact 90-degree angle. (This includes on main folk1 – the bug just hasn't been coming up on folk1 because angles tend to 'almost never' be exactly 90/180/270 there, since angles there are generally in projector-pixel-space and are derived from the angle of something else on the table.)

It comes up a lot more on folk2 now that we're drawing images/glyphs in page-space. In particular, when you're calibrating, you want to rotate the projected tags 90/180/270 degrees, and that was broken.

I found a better implementation of inverse bilinear interpolation and ported it to GLSL to replace the old one from Inigo Quilez, and that fixed the problem.

(One of the other most annoying things was just getting vertex order right so that glyph/image/tag quads actually show up and have the correct triangle facing!)

Because I've been trying to run folk2 on the (RAM-constrained) gadget, I've been really wanting to fix the memory leaks in it. It's been tough. My hypothesis (from seeing Jim allocations get out of control and from manually inspecting the heap) is that it's mostly little Jim Tcl heap objects (Jim_Objs, mostly). But hard to trace causation beyond that point so far.

It's getting kind of ridiculous how many different execution modes we have for folk2:

(it is a lot easier to do all this in folk2 than in folk1, though, because folk2 is all one big process and most of its kernel is statically compiled)

I was debugging memory leaks and found them way too hard to debug because of the added complexity of the per-thread statement term cache (that I added a couple months ago). So I temporarily disabled the cache (so we're now reparsing every term on every When block execution) to see what would happen.

Turning it off fixed all the 'brownouts'/blinking left on folk2, and it actually feels very solid now! And it's not slower at all (maybe the performance improvement from page quads is so enormous that it's OK?). And I guess blinking isn't as big a deal because of the stabilizing effect of page quads, too (pages almost always get the chance to display some converged version that's already in a bitmap in RAM.)

I guess we were fundamentally running on a buggy foundation for the last month or two.

Disabling cache is pretty much what has brought folk2 back into a 'stable era':

Mason Jones has been working on zero-copy object sharing between worker threads in folk2 (by disabling shimmering for shared objects so they're actually immutable).

Mason also wrote a fuzzer to make sure that the (pretty extensive) changes to the Jim interpreter don't break anything. As a result, Mason even found some bugs in Jim Tcl itself and got patches upstreamed there! All normal tests and fuzz tests seem to pass with the new zero-copy object sharing stuff, so it might be ready to try with folk2.

It's unclear to me what will perform well in practice – all the techniques we've tried for sharing data between threads seem like they'll behave differently under different workloads, so they need to be empirically tested to see what makes folk2 fast.

For now, I guess we'll try to merge folk2 without even the cache, just string reparsing every block, and then later see if some of these other approaches can be revived and can help further.

We had our open house on Sunday, April 27:

On April 23, a group from aci-d club (art craft interface-design) here in New York came by for a special visit with tours and a private talk by Omar.

We really enjoyed this. It was a nice change of pace from the usual open house format that we've been doing for 2 years: higher-context audience + the talk at the end was a reminder of a lot of our history and let the whole audience come together. Thinking about how to make future open houses have some of this energy.

• (Andrés) I went to Christina Huang's Learning Analytics class at NYU to demonstrate Folk, it was a fun and productive class. I'll add pictures in this newsletter once they're published.

• Our next Folk open house is in the afternoon on Saturday, May 31, at our studio in East Williamsburg, Brooklyn.
• Tickets for Rhizome World workshop on this Sunday, May 4, at 161 Water Street in the Financial District
• Omar: bumping this from last month: new gadget design pass (mostly just need to stabilize projector/calibration); put up gadget design on GitHub
• Omar: folk2: finish calibration support and re-port editor; make pull request?
• also maybe/bumped from last month (will probably wait until folk2 is merged at this point):
  • dual-camera and self-calibration/refinement tech?
• Andrés: Finish video playback
• Andrés: Kosmik integration demo

• https://csshell.com
• https://paperology.openthresholds.org
• https://arstechnica.com/science/2022/01/the-most-important-computer-youve-never-heard-of/
• https://observablehq.com/@tophtucker/babylonian-numerals
• https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-reference/network/esp_smartconfig.html
• https://www.reedbeta.com/blog/quadrilateral-interpolation-part-2/
• https://ktrosenberg.github.io/drawtalking/
• https://jingyi.me/static/misuse-PREPRINT.pdf
• https://65000.blog/blog/blake-andrews-on-horse-race-tests-and-reining-it-all-in
• https://newsletter.shifthappens.site/archive/the-modifiers-vs-the-keepers/

• I've been reading about “time-based media conservation” and stumbled on this fun workshop called "Art With A Plug"
• I also really like NYU's Institute of Fine Arts' video How to Become a Time-Based Media Conservator
• Fascinating paper on Curved origami with tunable stiffness — basically making small compliant mechanism machines using scoring and folding
• A Bit About Interface Builder
• Gabrielle Merite is making a lot of fascinating post-Tufte data visualizations