Table of Contents

July 2024 newsletter

What we've been up to

Demos

System improvements

3D calibration

Omar: Our goal is to merge the 3D calibration branch (osnr/checkerboard-calibration) into main, so we want it to be roughly on par with the current 2D calibration (ideally much better, though) on all systems.

We made a lot of progress this month. At the end of last month, we were trying to figure out why folk0 at Hex House doesn't calibrate that well compared to the system at my apartment. (it was like 1-2cm off compared to 1-2mm off) (maybe because folk0 is a physically big system, it has different camera and projector optics, etc).

Now we are able to calibrate folk0 to at least the level that 2D calibration is at most of the time (thanks to various accuracy and geometry fixes), and we've created the pull request and done a lot of polish so people can start to test it.

Big calibration board

I had this theory (from reading some more projector-camera calibration papers) that our calibration board (letter-sized) wasn't big enough to capture the camera/projector variation well:

If this theory were true and our board just isn't big enough, it would be very annoying, because we can't really ask people to fabricate a big board (you have to print several pages, find a board to glue them onto, measure very carefully). We might need to use an entirely different technique like the points-of-light from Yang (2016), which would be a lot of new engineering to reimplement.

20240731-194833.jpeg

I made a big board and calibrated with it, but only got slightly better results (maybe more due to better tag detection than board size?).

Testing old papers on folk0

So I tried to test some old projector-camera calibration papers – do any of them get better results on folk0 (presumably because we implemented something wrong) with a similarly small board, so we could keep iterating on our current technique, or is this an inherent problem with the approach / is our board just not big enough?

Lots of struggles in that replication work:

Audet's original ProCamCalib was the only one I could actually get to work.

20240801-021925.jpeg

I made a table of data from different Folk and Audet calibrations on folk0:

Notice how the camera is very good and consistent across all of them, but the projector is sketchy, so I focused more on the projector parameters over time. how can we get them to be consistent from our calibration and how do we get them to roughly match Audet's? 5000+ focal length

I even added some code to import Audet calibrations to test them in Folk, since Audet's software can't project outlines or anything like that (they're good, better than what we had at the end of June). This all made me optimistic. How can we get at least this level of improved calibration into Folk? It can't be that different.

(one major advantage we have is that the AprilTag detector that we use is a lot better than the ARToolkit detector used in Audet from 2009 or whatever, it's 10 years of progress, it doesn't require nearly as much fiddling with detector and camera options to be reliable)

Small accuracy improvements

Buoyed by the fact that at least Audet can get better results with a similar scheme to us (and with a similarly small calibration board), I was able to make multiple small accuracy improvements:

Calibration UI improvements

Extended the 3D calibration UI with:

(We probably should have a demonstration video embedded in the calibration page as well.)

The calibration UI now uses s-ol's folk.js from last month to do Folk↔Web frontend communication, which has made it hugely easier to make little additions to the Web UI.

Fixed calibration updating so it's immediately active once you finish the calibration process. (it was this horrible lexical scope bug that we should really fix, where existing variables can stomp 'shadowed' names)

Added a warning if the system isn't calibrated, which should help both with initial setup and with migration of existing 2D systems.

img_0334.jpg

Default and per-tag geometries

I spent a whole night digging into why some tags on folk0 work pretty well and others totally break in the same position (like 1047 will work a lot better than 642):

I discovered that some calibration accuracy issues on folk0 are just that certain weird tags (like keyboard tags) have been printed at different sizes! on cardstock/other paper. like they might be 32mm instead of 30mm, which totally breaks pose estimation (which starts from the assumption that tags are all 30mm).

So I added a new geometry specification step to the calibration process. This creates a default.folkgeom file that states the default tag-program geometry for that Folk system.

I also added support for .folkgeom files, which can manually override the geometry for a specific tag. (so you can make 642.folkgeom and tell Folk that that tag is a weird size and shape, like maybe it's 40mm and has a square around it). I think something like this was always going to be necessary if people want to be able to manually cut and recopy tags.

Storing geometry in PDF annotations

I also spent some time looking into storing geometry in annotations in each program's PDF. We might still do this – would be useful for fields and zines. Programmatically generated geometry of various kinds. But I realized it wouldn't solve our immediate problems (how do you guarantee that the PDF is printed at exact scale? and what about already-printed programs?)

(the .folktag approach I suggest in those messages morphed into the .folkgeom files that we now are using in the branch)

Testing 3D calibration so far

So far, have tested on: folk-omar-mini (excellent), folk-cwe (good), Daniel (good), folk-recurse (bad), folk0 (ok), folk-beads (bad/broken), folk-convivial (meh). More testing always welcome! Just check out the osnr/checkerboard-calibration branch and go to /calibrate.

(by the way, it would be particularly good if we could improve folk-convivial, which is the folk-cnc machine, so we can use that calibration for 3D/CNC coordinate stuff.)

What's next

I think we want to push a little more on accuracy across the various systems and improve the calibration UI, add more guidance so people know how to do calibration and get good coverage (there's still a fair amount of “tacit knowledge” involved in calibrating), but I'm reasonably optimistic :-)

Aside from accuracy improvements, it's nice to have a lot of the polish work and have the pull request in play now, too; it feels much closer to being done.

New evaluator

Omar: I've been working on making the new evaluator more reliable & porting things to it. I fixed a couple of correctness issues, but I think a big push is still needed there where I think through some of the deep problems. (for example, how do we keep old statements downstream of an old camera frame from blinking out before new statements from the new camera frame are locked in?)

Made an atomicity fix to statement insertion (I ripped out a lock last month and didn't add proper new synchronization to replace it). Vendored the AprilTag library (finally! I like the idea that we have AprilTag and jimtcl vendored and can control them and change them). Added glfw support on Linux so I can test graphics stuff on my Linux laptop.

Spent a couple of days on making stacktraces actually work, which wasn't as bad as I expected, honestly. (I had been dreading it, even though it felt necessary.) It's nice that all stack trace / script provenance behavior is understandable and editable in our vendored Jim interpreter, so we can re-inject the provenance across interpreters as stuff migrates between threads. folk2 now has the cool feature that every statement has a line number and source filename attached to it, which feels like a really nice/generally useful provenance thing.

Ported the keyboard driver from the old evaluator. Now that keyboards seem to work, I want to work on porting the editor, which is a good test of language capability (maybe we want mutable variables? maybe we want event statements?) and of reliability (the editor has to maintain a lot of transient state) and performance (it slows down old Folk a lot when you have a big source file and/or multiple editors at once, and we want to fix problems like that). I'm excited to have a really smooth editor that is as comfortable to work in as a laptop editor, where you're not scared to edit a long shader or have multiple people coding on the table at the same time.

Friends and outreach

What we'll be up to in August

Omar

Andrés