Table of Contents

June 2024 newsletter

What we've been up to

Demos

Real-world/3D calibration

Omar: A lot of progress on 3D calibration this month. As a reminder, the point of this 3D calibration project is

  1. to track programs in 3D, so you can lift them off the table and they'll projection-map properly, but also
  2. to build much more accurate tracking in general – millimeter precision – to enable new applications where you can highlight individual words or facets of objects, and
  3. to track coordinates in real-world meters instead of arbitrary projector or camera pixels, so we can start to integrate multiple projectors, cameras, and other real-world sensors and actuators (phone localization, RFID, CNC machine bed and material, spatial audio…), which will all benefit from having a shared ground coordinate system

This video (running the osnr/checkerboard-calibration branch on my home system) shows how you can accurately measure the distance between 2 pages in real centimeters, as well as how accurate the 3D projection map can get right now:

The distance is being displayed by this sort of hacked-together program using the new concept of quad, which we may change/simplify/wrap: 

When 19 has quad /a/ & 13 has quad /b/ {
  set a [quad change $a "/dev/video4"]
  set b [quad change $b "/dev/video4"]
  Wish to draw text with x 800 y 400 text [format {%.2f} [* 100 [norm [sub [quad right $a] [quad left $b]]]]]cm radians 3.14
}

This past month:

3D calibration works extremely well (1-2mm error) on my personal system in my apartment, as you saw in the video above.

It works much less well (1-2cm off) on folk0 at Hex House, and even worse on folk-beads (the cart system at Hex House), where the high distortion and shallow 3D volume of the ultra-short-throw projector make it really hard to calibrate.

I've also made it almost fully backward-compatible with all existing code – it now derives 2D projector-space region claims from the 3D calibration, so all code that uses regions still works. I deleted all the old 2D calibration and homography and region code (calibrate.tcl), so 3D calibration backs everything on the branch.

I'd like to get it to at least be on par with the existing 2D calibration in most cases before we merge it, and we're not there yet… (working theory is still that the individual intrinsics of camera and projector are derived pretty well, but the extrinsics seem off)

Arcade got the 3D calibration branch working on their system:

rn_image_picker_lib_temp_b9b82582-cbe0-4bb5-b879-589c08e7f789.jpg

(Not perfect – there's clearly that 1-2cm offset between camera and projector – but it's great to see other people get it working at some level, since we've been worried that the calibration won't generalize or that it'll be hard for people to learn to do.)

Arcade's calibration board, taped to a board game:

pxl_20240623_205104464.jpg

New parallel evaluator

Omar: I've been continuing to work on the new preemptive/low-latency/parallel Folk evaluator.

At the beginning of June, I fixed a major bug where some Hold statements would just vanish half the time (so you might lose a program if you were dragging it around and got unlucky). It turned out that Holding the same statement multiple times could get you into a weird state (it would throw out the old statement but not make a new statement). (and this happens often when dragging, since micro-events might just fire with the thing in the same place, or you might drag it back over the same territory) I feel a lot better about the evaluator after fixing this – it seems a lot more reliable.

I made the edge-lists (of children of a statement/match) garbage collect on resize/defragment, which is something I'd been meaning to do & should solve one of the big leaks (and make the GraphViz graphs a lot more comprehensible, since they won't have lots of zombie edges).

I ported the setup.folk scheme from main so that I can run folk2 on a laptop and choose a non-default projector and camera in setup.folk. Had to touch some of the camera and GPU code to pass parameters in.

Still need to fix some other memory safety / refcount issues and some leaks, like we aren't freeing clauses yet. I've started running folk2 through Valgrind (it's actually a lot more doable than running original Folk through Valgrind – new evaluator does less stuff, is more pre-compiled, etc) and will probably do more in July. Found and fixed a workqueue initialization issue (have to do it before starting any thread so they can safely try to steal from other threads right away).

After fixing the Hold bug, we got to test the new evaluator on the actual table on folk0. (It's now reliable and fast enough to be worth testing.)

It performs really well, much faster/lower latency/higher throughput than any previous Folk evaluator including the current main one.

In terms of actually being able to fully switch to the new evaluator, the main open issues are 1. memory crashes and 2. labels/outlines/etc flashing (because statement revocation can happen too early and cause stuff to blink out, it's not strictly ordered to happen after new statements come in like it is on classic Folk). Once the crashes are fixed and we have a good way of dealing with statement revocation (local convergence? marking downstream statements?), we should start moving to the new evaluator.

Portable gadget

Omar has done a couple new iterations of the portable gadget from last month (including picking up more CAD software, which as surprisingly inspirational UI). Here's what it looks like right now:

img_9297.jpeg img_9291.jpeg img_9299.jpeg

The hand grip and front look good, and the projector and camera and Pi 4 fit well. The hand grip is a little loose, so need rubber or a groove to keep it from swiveling around. (The Ultimems projector inside also swivels around and probably also needs rubber or a groove.) The back / wiring isn't fully figured out yet; that also will need a lid/cap that you screw in, I think. I still need to design and wire and do the trigger button, which I think will make a big difference to how you can interact with it.

I also have issues keeping it powered right now (and I want to figure out a USB power pack situation anyway, where you can choose to use wall power or a power pack). I wonder if the projector is too much for the Pi 4's USB port to power, or if it's overheating, or something.

There're also still a bunch of software improvements to make to Folk to make this thing usable (text rendering is broken on the Pi 4; you can't do traditional 2D stripe calibrate with this mini projector because it's too dim; etc)

System improvements

folk.js

s-ol has added folk.js, a Web client library so you can emit and subscribe to Folk statements from inside a Web page. It also lets you easily convert between Tcl and JS values.

It feels like it can handle disconnect and status reporting and other such conditions a lot better and cleaner than one-off code. It significantly simplifies the implementation of stuff like the Web editor.

WS message are executed in the context of a match that is tied to the websocket lifetime. An emit prefix msg proc is provided, and the JS code dispatches messages based on the prefix value.

Utilities for encoding and decoding Tcl strings are provided, including a tagged template literal:

const complicatedString = 'This string {contains} some {} odd "stuff"';
tcl`We ${'embed'} ${complicatedString} here`
// 'We embed {This string {contains} some {} odd "stuff"} here' 

const json = [ { name: "Sol", age: 27 }, { name: "folk", age: "-1" } ];
ws.send(tcl`Assert the entities are ${json}`);
// 'Assert the entities are {{name Sol age 27} {name folk age -1}}'
 
const tclVal = '{name Sol age 27} {name folk age -1}';
loadList(tclVal).map(loadDict);
// [ { "name": "Sol", "age": "27" }, { "name": "folk", "age": "-1" } ]

It should be very useful and lower the barrier to many phone, laptop, other traditional computing integrations which we've had to do from scratch every time until now (like the work that Arcade has done with tablet and files, that Andrés has done with canvas, & that Omar has done with calibration and CNC preview endpoints).

Friends and outreach

What we'll be up to in July

Omar

Andrés