We've had a funding shortfall recently – we'd really appreciate it if you sponsor Folk on GitHub Sponsors if you can, so that we can keep doing this free and open work

We’ll be set up at Rhizome World on this Sunday, May 4 (at 161 Water Street in the Financial District). Feel free to say hello; we’ll also be running a Folk Computer workshop there at noon.

If you want to stop by the studio, our next Folk open house is in the afternoon on Saturday, May 31, in East Williamsburg, Brooklyn.

• (Andrés) We merged (TODO (Andrés): merge all) Rob Fielding's PR's to:
  • Re-activate Jacob Haip's blob-detection code, introducing a spruced up demo program:
    • This also means the laser capability is back! (Side note: I love the dog getting to play with the laser here … it makes me wonder what a Folk system specifically for entertaining / training pets could look like!)
  • Add `Wish $this draws text “example”` to the system, making text work like our general shape syntax:
  • Tools for creating subregions:

Omar: I spent like a day trying to add stereo camera support (was hoping to show it off at a party.)

The immediate motivation was extending the QR scanner gadget application from last month so it can actually lasso the QR codes (tightly fit projected marching ants outline to the QR code on the surface), which requires pose-estimating arbitrary QR codes in the wild, which we can't do with just one camera because we don't know the size of those arbitrary QR codes a priori (unlike AprilTags that are printed through Folk). So we need two cameras, I think.

A lot of the changes are just basic scoping of calibration and camera and tag code to actually support multiple cameras, instead of collapsing all cameras into one.

Also some changes to support multiple crop areas, so you can split the ELP stereo camera on the gadget into multiple 'virtual cameras'.

I still think stereo camera has a lot of potential to improve our baseline performance and level of calibration:

trying the stereo camera on folk0 (will also need some software patching which should be generally useful also – ability to address multiple cameras, ability to use cropping to split a single linux video device into multiple virtual cameras)


[notice the second – ELP stereo, same as in gadget – camera in the foreground on the right here:]


and then of course the multi-calibration lol (i think easiest is to just calibrate them all individually for now and then exploit the fact that we'd have extrinsics from each camera to the common projector to derive their relationship to each other)

advantages of stereo camera are we can have much more legit pose estimation and i think we might be able to run faster because we could do all the detection at lower resolution (because we wouldn't need the high resolution for pose estimation)

(also just prototyping for the gadget which has more applications for the stereo cameras)

Anyway, didn't get very far, but it's probably worthwhile for the gadget applications alone, but it could also be useful on more normal systems for the reasons above.

Omar: I haven't done much on the gadget this month. Mostly trying to get the projector to stay stable so the calibration remains valid (it swivels and wiggles around because it's only attached at one point). Added washers to the attach point for the projector because the bolt warps the plastic.

It looked like the old front panel had too short an aperture for the projector, cutting off a lot of the bottom of the image.

Printed a new front panel with a taller aperture to try to not block the projection image at all. Better, but still not tall enough!

It actually wiggles up and down as much as left and right… I wonder if I can clip it somehow. Need to think about physical changes I can make to the chassis to keep the projector locked in place.

Also still need to put this design on GitHub! I'm just not happy with it yet.

Omar: I would say work this month has been driven by three different pushes (not really separable that cleanly, but oh well):

1. Continuing the performance/anti-blink push from last month: this drove page quads

2. Gadget: memory leak

3. Fix calibrate on folk2 (given the changes from 1)

I feel pretty good about folk2 right now. Beyond fixing the memory leak, a lot of the remaining work is functionality work, honestly, where we need to get calibration and the editor running again under the new regime (as well as camera slices, animation, other smaller modules). I've been putting that off because I really want to fix the memory leaks…

I've spent a lot of this month working on the page quads / render-to-texture stuff that I started last month, which has produced huge performance improvements (almost more than everything else in folk2 combined). Really excited to get this merged now.

I added support for drawing text and images and AprilTags on the textures, which was hard to debug, so I added a /images endpoint that shows all textures in memory (including placeholder image 0, font atlases, page textures, projected calibration board texture, other images):

To cleanly do texture destruction when a page is flipped over, I added support for per-statement destructors:

I also use this to clean up camera images now, which feels really nice (instead of just timing out camera images after some fixed period of time):

It turned out that our implementation of bilinear interpolation is pretty broken if rotating at an exact 90-degree angle. (This includes on main folk1 – the bug just hasn't been coming up on folk1 because angles tend to 'almost never' be exactly 90/180/270 there, since angles there are generally in projector-pixel-space and are derived from the angle of something else on the table.)

It comes up a lot more on folk2 now that we're drawing images/glyphs in page-space. In particular, when you're calibrating, you want to rotate the projected tags 90/180/270 degrees, and that was broken.

I found a better implementation of inverse bilinear interpolation and ported it to GLSL to replace the old one from Inigo Quilez, and that fixed the problem.

(One of the other most annoying things was just getting vertex order right so that glyph/image/tag quads actually show up and have the correct triangle facing!)

I was debugging memory leaks and found them way too hard to debug because of the added complexity of the per-thread statement term cache (that I added a couple months ago). So I temporarily disabled the cache (so we're now reparsing every term on every When block execution) to see what would happen.

Turning it off fixed all the 'brownouts'/blinking left on folk2, and it actually feels very solid now! And it's not slower at all (maybe the performance improvement from page quads is so enormous that it's OK?). And I guess blinking isn't as big a deal because of the stabilizing effect of page quads, too (pages almost always get the chance to display some converged version that's already in a bitmap in RAM.)

I guess we were fundamentally running on a buggy foundation for the last month or two.

Disabling cache is pretty much what has brought folk2 back into a 'stable era':

Mason Jones has been working on zero-copy object sharing between worker threads in folk2 (by disabling shimmering for shared objects so they're actually immutable).

Mason also wrote a fuzzer to make sure that the (pretty extensive) changes to the Jim interpreter don't break anything. As a result, Mason even found some bugs in Jim Tcl itself and got patches upstreamed there! All normal tests and fuzz tests seem to pass with the new zero-copy object sharing stuff, so it might be ready to try with folk2.

It's unclear to me what will perform well in practice – all the techniques we've tried for sharing data between threads seem like they'll behave differently under different workloads, so they need to be empirically tested to see what makes folk2 fast.

For now, I guess we'll try to merge folk2 without even the cache, just string reparsing every block, and then later see if some of these other approaches can be revived and can help further.

• TODO: Discord updates

We had our open house on Sunday, April 27:

On April 23, a group from aci-d club (art craft interface-design) here in New York came by for a special visit with tours and a private talk by Omar.

We really enjoyed this. It was a nice change of pace from the usual open house format that we've been doing for 2 years: higher-context audience + the talk at the end was a reminder of a lot of our history and let the whole audience come together. Thinking about how to make future open houses have some of this energy.

• Andrés went to Christina Huang's Learning Analytics class at NYU to demonstrate Folk, it was a fun and productive class:
  • TODO (Andrés): pictures

• Our next Folk open house is in the afternoon on Saturday, May 31, at our studio in East Williamsburg, Brooklyn.
• Tickets for Rhizome World workshop on this Sunday, May 4, at 161 Water Street in the Financial District
• Omar: bumping this from last month: new gadget design pass (mostly just need to stabilize projector/calibration); put up gadget design on GitHub
• Omar: folk2: finish calibration support and re-port editor; make pull request?
• also maybe/bumped from last month (will probably wait until folk2 is merged at this point):
  • dual-camera and self-calibration/refinement tech?
• Andrés: Finish video playback
• Andrés: Kosmik integration demo

• https://csshell.com
• https://paperology.openthresholds.org
• https://arstechnica.com/science/2022/01/the-most-important-computer-youve-never-heard-of/
• https://observablehq.com/@tophtucker/babylonian-numerals
• https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-reference/network/esp_smartconfig.html
• https://www.reedbeta.com/blog/quadrilateral-interpolation-part-2/
• https://ktrosenberg.github.io/drawtalking/
• https://jingyi.me/static/misuse-PREPRINT.pdf
• https://65000.blog/blog/blake-andrews-on-horse-race-tests-and-reining-it-all-in
• https://newsletter.shifthappens.site/archive/the-modifiers-vs-the-keepers/

• Vox graphics