Table of Contents

December 2023 newsletter

It's been a big year. Hard to believe that a year ago, there was exactly 1 Folk system in the world (the original one, folk0 at Hex House) & it ran about ten times slower than what we have now and could barely do anything besides display text and outlines. But there's still an enormous amount to do :-)

If you're in New York next month, we'd love to have you visit us at our next Folk open house, in the evening on Wednesday, January 31.

We hope you enjoy this final newsletter of 2023 and have a great New Year's!

What we've been up to

Applications and demos

Other stuff

Tabletop / in-system editor

Calibration

Omar: the new 3D calibration has been coming along. Here's a video of the current state of projector-camera calibration:

As you can see, calibration accuracy is actually very good in this area. Notice how it works in 3D, not just on the table plane, and how the projected outline is almost perfectly aligned with the tag. As we've said in previous newsletters, we're hoping that this (real-metric 3D space, 3D calibration) will open up a lot of new applications and sensor/actuator integrations.

Caveats:

  1. this is near the center of an almost-coplanar projector-camera pair, so it's favorable conditions for the calibration
  2. there is a fair amount of jitter each frame, which is pretty annoying and disruptive for real applications (especially if you extrapolate out from the tag itself into a larger page-sized region on its plane, as we would for labeling/outlining/other display)

Most or all of the improvement since last month comes from working on the nonlinear Levenberg-Marquardt refinement – we're still using the same kind of initial pose→linear pinhole estimate guess as before, then refining it with this iterative process.

The refinement now computes two radial distortion terms per lens (two for the projector, two for the camera, with initial guesses all 0s), which helps with accuracy toward the fringes of the projector/camera area. (I feel like the radial distortion applied is too conservative, it still gets pretty messed up at the fringe, 1cm+ error)

I learned about three-degree-of-freedom vector representation of rotations, which is a surprisingly complicated subject, with a lot of weird formulas you just have to implement.

I realized I had totally messed up how refinement deals with rotation – you have to go from 9-degree rotation matrix to 3-degree vector, refine in that representation, then go back to 9-degree matrix when you're done. So I implemented those formulas and fixed that.

Rather than using Zhang and Audet's technique of having separate 'camera calibrations' (basically calling OpenCV, incl. refinement) for the camera vs. for the projector, we now do a single joint end-to-end refinement for the entire system (including projector intrinsics, projector distortions, camera intrinsics, camera distortions, and camera-to-projector rigid motion), with the error function being the reprojection error of the training data AprilTags, like in the video above.

This end-to-end optimization produces much better results, as you might expect, although it's slow: it can take 1-4 minutes. It requires doing AprilTag pose estimation for every example tag at every iteration, and pose estimation is itself an iterative process that the AprilTag library does.

Next steps:

Going to test offline and try projecting onto pose images to iterate faster, see distribution of error among poses.

(Worst comes to worst – and this is another payoff of 3D calibration – we can always just throw more cheap cameras at the problem, like have 2 or 3 or 4 cameras pointed at roughly the same space and integrate their observations…) (that's also nice because it gives us a way out of occlusion problems)

Jitter reduction

Cristóbal and I spent a day trying some tactics to reduce jitter, which I think might be the single biggest impediment to getting this thing merged and usable. The jitter is really noticeable in person.

The actual 2D AprilTag detection of corners in the camera image is actually very stable and reliable, and the images are clear, so that's not a source of jitter in our view. (We tried some moving-average smoothing stuff on the detected tag corners, but it didn't work that well, even if we made it really aggressive. We concluded that it has to be further on in the pipeline, not the input corners.)

So you have a stable input (locations of the corners in the camera image) and a pure function (camera corners → 3D pose estimate → reprojection corners), but somehow the output projection has jitter. So where is the jitter in the function?

We found two primary sources of jitter:

  1. Instability in the AprilTag detector's tag plane→camera plane homography computation (which just uses 4 example points to compute a homography, (-1, -1), (1, 1), etc → the 4 camera corners). this surprised us, since it's such a simple computation, pure linear algebra! this jitter is all over the place, the homography is very sensitive to subpixel changes in the corner points, it's like an overfitting kind of thing.
  2. Instability in the AprilTag detector's iterative pose estimate. this is more expected, and it's more discrete, it often toggles between two distinct states where you might imagine there's some visual ambiguity; it probably ranks the possible states in different orders for different frames, leading to jitter

(See also this GitHub issue on the AprilTag repo.)

The homography jitter we were able to completely tamp down by forcing the homography to be affine (forcing the bottom-left elements to be zero), but that feels not right.

But it works pretty well in practice, even under skew conditions. Weird.

As for the pose estimate jitter… well, we have some ideas and tactics we could try, we haven't gotten to it yet.

RFID refactor

Omar: the long-awaited RFID refactor is finally underway: I've made a lot of progress on it and have kind of broken through the previous point where I was stuck.

I haven't caught up with our original branch on actual localization yet, but we now have tag identities and checksums, which is a capability we never had before. You can see above that we can carry out a back-and-forth and get that long reply from the tag (which tells us its ID).

And here we query the tag a few times in a row and get a valid, checksum-passing ID response each time (although we're not printing the ID itself here, only the expected+received checksums):

Checksum and identity are immediately useful in a few ways:

and of course, you ultimately need tag identities to do any interesting application with RFID tags – that's the point of them – so we knew we needed to do this sooner or later. It's also the hardest part of implementing an RFID reader on software-defined radio, because it's highly latency-sensitive (you need to decode entire tag packet and respond within 500 microseconds or so to get the tag ID), so it's a huge relief that we are able to do it and won't have to change approach to get it to work.

Some key things are:

Next steps:

Outreach and community

Open house

We had another in-person open house on December 13 – not long after the one at the end of November, but we like keeping up the pace. This one was more relaxed & a good opportunity to catch up with friends in the area.

img_1865.jpeg img_1881.jpeg

Cart

The open house was a good push to follow up on building folk-beads last month. This month, we built a cart for folk-beads, so we can wheel it in and out, since we only have it out during events (it's not a permanent system – it uses shared space in Hex House):

img_1804.jpeg

The cart uses a cheap utility cart from Amazon and some two-foot quick clamps to keep the ultra-short-throw projector from falling over (and some blocks of wood we had lying around Hex House to raise the projector to table height while letting the HDMI and power cords jut down underneath). It's not the most stable thing ever – you have to hold the projector while rolling it – but it works well enough, and it's so nice that it's finally a self-contained unit.

(We probably also want it to have a longer power cable so we can wheel it around the space without unplugging and replugging. Ten feet is too short, I think.)

Folk User Group

We had a “user group” meeting in Discord on the evening of December 19, where we got on a call with people in the community (mostly people with running Folk systems). It was fun to see everyone again (the last one we did was a few months ago); we explained some recent and upcoming improvements to the system & we did some Q&A:

We'll do another one in a few months, most likely.

What we'll be up to in January

Andrés

Omar