• Jessie Grosen made a SuperCollider-based audio system using Folk, over at Recurse Center in Downtown Brooklyn:
  • (Jessie has been building on some of the work that Ian Clester did on music in Folk last year)
  • (also building on Cristóbal Sciutto's hands!)
• Andrés made a “container” demo. The pink and blue programs below search for neighbors that are immediately next to them or within their bounds and give those programs thick (pink||blue) outlines.
  • 
  • This is a simple example, but you could imagine putting photos into Twitter or Instagram containers and then uploading them from the table, or sorting through files on your computer by placing them into various Folk containers that correspond to your filesystem. These kinds of physical → digital interactions are something we're really excited about Folk powering.
• Andrés worked with Char Stiles to update the GPU guide, making it easier to get started with uniforms, functions, etc.
• Omar made some Switch Brancher tools: 3D calibration (discussed later in this newsletter) has gotten advanced enough that we want to swap systems in and out of the 3D calibration branch without needing a laptop to ssh & git (we still want to go back to main sometimes since compatibility isn't 100% yet)
  • 
• Omar got the 3D distance demo from his apartment last month to run on folk0 and made it more plug-and-play:
  • 
  • The printed program at the top in the middle shows the distance (in centimeters) between the program on the left and the one on the right. The target programs are specified by the program being written in the editor at the bottom. Watch the distance change as I move the program:
• Andrés has been working on more complicated shader examples:
  • A scavenger hunt in screen space for a red oval on folk0:
    • 
  • A black shader programming moving over a full-table shader – feels like there's a ghost or something in the room:
    • 
  • Playing with opacity in shaders and shapes in Folk:
    • 
• s-ol bekic did an “augmented brainstorming” session in Milan on folk-opendot – this might be the largest-scale use of Folk so far!
  • 
  • “not everything worked the way I thought but the goal was to do something a bit different and engaging as a backdrop to paper+pen work and it totally worked in that context 🙂”
    • “Performance went down to like 2fps and some things stopped working as I expected (in ways that I don't understand yet)”
    • Omar: more motivation for performance work, haha

• Andrés has been working on changing ''Commit'' → ''Hold''
  • This change is mostly done, need to do another test on more complicated programs that use Commit in Folk (n.b. if you're a Folk user, please test this branch out for bugs). Andrés is going to do some more testing August 1st and hopefully will merge the branch August 2nd.
• Andrés and s-ol worked on removing the need for passwordless sudo to use keyboards
• Various small fixes from s-ol (prep for opening GitHub repo to public):
  • Web editor can now edit folk-live and user-programs, not just core virtual-programs
  • Support for z layer in all drawing primitives
  • Wish PAGE is filled with color blue wish
• Omar has been working on merging folk-live-build into the main Folk repo
  • I think we want to get into more of a monorepo situation where everything gets versioned together (so you know exactly what dependencies a given Folk version has, what OS you need, etc). Also want to vendor the AprilTag library soon

Omar: Our goal is to merge the 3D calibration branch (osnr/checkerboard-calibration) into main, so we want it to be roughly on par with the current 2D calibration (ideally much better, though) on all systems.

We made a lot of progress this month. At the end of last month, we were trying to figure out why folk0 at Hex House doesn't calibrate that well compared to the system at my apartment. (it was like 1-2cm off compared to 1-2mm off) (maybe because folk0 is a physically big system, it has different camera and projector optics, etc).

Now we are able to calibrate folk0 to at least the level that 2D calibration is at most of the time (thanks to various accuracy and geometry fixes), and we've created the pull request and done a lot of polish so people can start to test it.

I had this theory (from reading some more projector-camera calibration papers) that our calibration board (letter-sized) wasn't big enough to capture the camera/projector variation well:

If this theory were true and our board just isn't big enough, it would be very annoying, because we can't really ask people to fabricate a big board (you have to print several pages, find a board to glue them onto, measure very carefully). We might need to use an entirely different technique like the points-of-light from Yang (2016), which would be a lot of new engineering to reimplement.

I made a big board and calibrated with it, but only got slightly better results (maybe more due to better tag detection than board size?).

So I tried to test some old projector-camera calibration papers – do any of them get better results on folk0 (presumably because we implemented something wrong) with a similarly small board, so we could keep iterating on our current technique, or is this an inherent problem with the approach / is our board just not big enough?

Lots of struggles in that replication work:

Audet's original ProCamCalib was the only one I could actually get to work.

I made a table of data from different Folk and Audet calibrations on folk0:

Notice how the camera is very good and consistent across all of them, but the projector is sketchy, so I focused more on the projector parameters over time. how can we get them to be consistent from our calibration and how do we get them to roughly match Audet's? 5000+ focal length

I even added some code to import Audet calibrations to test them in Folk, since Audet's software can't project outlines or anything like that (they're good, better than what we had at the end of June). This all made me optimistic. How can we get at least this level of improved calibration into Folk? It can't be that different.

(one major advantage we have is that the AprilTag detector that we use is a lot better than the ARToolkit detector used in Audet from 2009 or whatever, it's 10 years of progress, it doesn't require nearly as much fiddling with detector and camera options to be reliable)

Buoyed by the fact that at least Audet can get better results with a similar scheme to us (and with a similarly small calibration board), I was able to make multiple small accuracy improvements:

• Gate pose recording based on pose homography RMSE: we now require the printed and projected tags to be well-aligned to a proper linear grid (RMSE should be less than 5 pixels), or else we won't record a pose. we previously relied only on consistency over the last few frames ('fixed point') as a proxy for this, but I feel more comfortable now having the direct test
• Switched from perimeter to alternating grid, inspired by ProCamCalib
  • Old (perimeter board):
    • 
  • New (alternating grid):
    • 
  • It's a little harder on the user now, because printed & projected tags overlap more often, there isn't the wide gap in the middle of the board that you can get most sizes of projected tag into (it's also harder to initialize; in general, you need to be conscious of fitting projected tags into gaps to keep it alive now)
  • Reduced the number of tags and made them bigger, which seems to be a better overall tradeoff (lower maximum number of points but their positions are more reliably detected on camera)

Extended the 3D calibration UI with:

• “Print through Folk”
• semi-live camera preview (so you can debug focus/exposure/physical placement issues without leaving the calibration page)
  • (we're also finally building ''/camera-frame'' into the base virtual programs, since we need it to do the camera preview)
• slider for adjusting default projected tag size so it'll fit (also lets you reset state if it gets in a weird state where self-detection is broken)
• result reporting (pose status, RMSEs of poses and calibrations)
• more explicit instructions

(We probably should have a demonstration video embedded in the calibration page as well.)

The calibration UI now uses s-ol's folk.js from last month to do Folk↔Web frontend communication, which has made it hugely easier to make little additions to the Web UI.

Fixed calibration updating so it's immediately active once you finish the calibration process. (it was this horrible lexical scope bug that we should really fix, where existing variables can stomp 'shadowed' names)

Added a warning if the system isn't calibrated, which should help both with initial setup and with migration of existing 2D systems.

I spent a whole night digging into why some tags on folk0 work pretty well and others totally break in the same position (like 1047 will work a lot better than 642):

I discovered that some calibration accuracy issues on folk0 are just that certain weird tags (like keyboard tags) have been printed at different sizes! on cardstock/other paper. like they might be 32mm instead of 30mm, which totally breaks pose estimation (which starts from the assumption that tags are all 30mm).

So I added a new geometry specification step to the calibration process. This creates a default.folkgeom file that states the default tag-program geometry for that Folk system.

I also added support for .folkgeom files, which can manually override the geometry for a specific tag. (so you can make 642.folkgeom and tell Folk that that tag is a weird size and shape, like maybe it's 40mm and has a square around it). I think something like this was always going to be necessary if people want to be able to manually cut and recopy tags.

I also spent some time looking into storing geometry in annotations in each program's PDF. We might still do this – would be useful for fields and zines. Programmatically generated geometry of various kinds. But I realized it wouldn't solve our immediate problems (how do you guarantee that the PDF is printed at exact scale? and what about already-printed programs?)

(the .folktag approach I suggest in those messages morphed into the .folkgeom files that we now are using in the branch)

So far, have tested on: folk-omar-mini (excellent), folk-cwe (good), Daniel (good), folk-recurse (bad), folk0 (ok), folk-beads (bad/broken), folk-convivial (meh). More testing always welcome! Just check out the osnr/checkerboard-calibration branch and go to /calibrate.

(by the way, it would be particularly good if we could improve folk-convivial, which is the folk-cnc machine, so we can use that calibration for 3D/CNC coordinate stuff.)

I think we want to push a little more on accuracy across the various systems and improve the calibration UI, add more guidance so people know how to do calibration and get good coverage (there's still a fair amount of “tacit knowledge” involved in calibrating), but I'm reasonably optimistic

Aside from accuracy improvements, it's nice to have a lot of the polish work and have the pull request in play now, too; it feels much closer to being done.

Omar: I've been working on making the new evaluator more reliable & porting things to it. I fixed a couple of correctness issues, but I think a big push is still needed there where I think through some of the deep problems. (for example, how do we keep old statements downstream of an old camera frame from blinking out before new statements from the new camera frame are locked in?)

Made an atomicity fix to statement insertion (I ripped out a lock last month and didn't add proper new synchronization to replace it). Vendored the AprilTag library (finally! I like the idea that we have AprilTag and jimtcl vendored and can control them and change them). Added glfw support on Linux so I can test graphics stuff on my Linux laptop.

Spent a couple of days on making stacktraces actually work, which wasn't as bad as I expected, honestly. (I had been dreading it, even though it felt necessary.) It's nice that all stack trace / script provenance behavior is understandable and editable in our vendored Jim interpreter, so we can re-inject the provenance across interpreters as stuff migrates between threads. folk2 now has the cool feature that every statement has a line number and source filename attached to it, which feels like a really nice/generally useful provenance thing.

Ported the keyboard driver from the old evaluator. Now that keyboards seem to work, I want to work on porting the editor, which is a good test of language capability (maybe we want mutable variables? maybe we want event statements?) and of reliability (the editor has to maintain a lot of transient state) and performance (it slows down old Folk a lot when you have a big source file and/or multiple editors at once, and we want to fix problems like that). I'm excited to have a really smooth editor that is as comfortable to work in as a laptop editor, where you're not scared to edit a long shader or have multiple people coding on the table at the same time.

• Our July open house was the biggest we've had in a while – it was at 8pm on a Saturday (partly forced on us by lack of air conditioning at Hex House, but we also wanted to celebrate printing program 1000 last month):
  • 
  • Andrés had people sign or doodle on their Guestbook programs this time around – visitors had fun leaving their mark on their programs:
    • 
• We brought the projector gadget and crank device to the Malleable Computing dinner at the Val Town office in Brooklyn, met many people who were in town (like Lu Wilson and Orion Reed and Maxim Leyzerovich)
  • 
• Cheng Xu visited us – we talked a bunch about installation art, the advantages of embodied interaction, and using computers for fun over productivity.
  • 
• anna y lin & Zichen Yuan visited – some excitement about the portable gadget and the new mode of interaction
  • 
• Omar and Andrés visited Jessie at Recurse Center and checked out the setup there; tried to run 3D calibration; put up some signs in the hope that future Recursers will be able to use and maintain the system
  • 
  • We'd been excited to have an RC system for a while – just needed hardware and an internal maintainer at RC
  • 
• Our old colleague Weiwei Xu stopped in while she was visiting the city last week – it's been a couple of years, so it was nice to show all the changes:
  • 
• today, we met some of our very close neighbors at boshi's place and gave them a tour of Hex House and Folk!
  • 

• Our next Folk open house is on the night of Sunday, August 18 at our studio in East Williamsburg, Brooklyn.
  • Possibly do a workshop to get people familiar with making stuff in the system beforehand
• 3D calibration: one last accuracy/UI push, get a couple more people to test, then merge
  • also minor compatibility fixes: animation, mask-tags
  • Omar: if we can get it merged, I hope we can start working on some really exciting stuff with precise camera slices: OCRing fields, checkboxes, stuff like that
• New evaluator: porting the editor, doing correctness/blinking fixup (maybe we track provenance of statements?)
• Gadget: need to look into issues with power delivery and with crashing. Maybe upgrade to Pi 5? And build button?

• https://en.wikipedia.org/wiki/Extensible_Metadata_Platform#Location_in_file_types
• https://valgrind.org/docs/manual/ms-manual.html
• https://blog.trailofbits.com/2022/07/01/libmagic-the-blathering/
• https://ai.meta.com/blog/segment-anything-2/ – we gotta get on using this (the real problem is the logistics of getting it set up, do you need Python, do you need Nvidia hardware, etc. ideally there would be a self-contained C library that just calls into a Vulkan backend)
• http://vcg.isti.cnr.it/publication/2024/PPSC24-3dTouch/ – still thinking about 3D-printed computational objects with touch, projection mapping, RFID, snap magnets, etc. get away from flat planar pages
• https://arxiv.org/abs/2306.01643
• https://www.youtube.com/watch?v=_SREVKe8QhQ

• "Creating a QR Code step by step"
• Long and incredibly incredibly illuminating post on color spaces
• A manga Pentel produced about their iconic Pentel Sign Pen
• I went down a Lynda Barry rabbit hole last month — this is a collection of links I got out of that exploration.
• Collecting desks I find inspiring
• Fascinating article on the history of education television — Look, Eye on Research, and Tomorrow's World are some of the earliest educational television content and I'm deeply grateful to this little write-up for introducing them to me
• Speaking of education and technology … Andy Matuschak's "Exorcising us of the Primer" is a great explanation of what's not laudable about wanting to build the AR Primer from Neil Stephenson's Diamond Age. The whole thing is worth a read but it pairs well with this tweet: “Redditor: My ultimate goal in life is to make the primer real. Anything you want to make sure I get right? NealStephenson: Kids need to get answers from humans who love them.” (emphasis mine)
• A wild drawing editor project completely written using WebGPU
• Adam Wiggins is now at Elicit working on some fascinating tools for exploring research via living documents