• We printed the 1000th program on folk0 at Hex House!
• Our friend Kevin Kwok came by and reprogrammed his crank gadget to display Folk program AprilTags; we cranked through a lot of the history on folk0, which was fun
  • 
  • 
• Arcade Wise made a file upload page which lets you upload files from a phone/laptop (so you can use them in Folk):
  • 
• Arcade made a Web routes utility

  • Wish $this is titled "Routes"
    Wish $this is outlined blue
    
    When /someone/ wishes the web server handles route /route/ with handler /h/ {
        Wish $this is labelled $route
    }

• Video
  • There's two parallel branches right now:
    • Andrés has been working on loading videos using libav* (the libraries underpinning ffmpeg). That project's mid-way through, with the current state being Wish $this displays video /path/to/video.mp4 loads the first frame of the video and displays as a native image_t type in Folk:
      • 
    • s-ol bekic has been working on a branch for supporting WebRTC streaming of video and images in Folk:
      • 
    • It looks like the two branches can eventually be merged, and we might actually pick GStreamer over ffmpeg because it can also handle audio routing.

Omar: A lot of progress on 3D calibration this month. As a reminder, the point of this 3D calibration project is

1. to track programs in 3D, so you can lift them off the table and they'll projection-map properly, but also
2. to build much more accurate tracking in general – millimeter precision – to enable new applications where you can highlight individual words or facets of objects, and
3. to track coordinates in real-world meters instead of arbitrary projector or camera pixels, so we can start to integrate multiple projectors, cameras, and other real-world sensors and actuators (phone localization, RFID, CNC machine bed and material, spatial audio…), which will all benefit from having a shared ground coordinate system

This video (running the osnr/checkerboard-calibration branch on my home system) shows how you can accurately measure the distance between 2 pages in real centimeters, as well as how accurate the 3D projection map can get right now:

The distance is being displayed by this sort of hacked-together program using the new concept of quad, which we may change/simplify/wrap:

When 19 has quad /a/ & 13 has quad /b/ {
  set a [quad change $a "/dev/video4"]
  set b [quad change $b "/dev/video4"]
  Wish to draw text with x 800 y 400 text [format {%.2f} [* 100 [norm [sub [quad right $a] [quad left $b]]]]]cm radians 3.14
}

This past month:

• Many small steps to improve the accuracy of the intrinsic and extrinsic calibrations:
  • Instead of end-to-end refinement using the pose estimate, which is slow, we now separately refine the intrinsic calibrations of the camera and projector, which is the more classic Zhang/Audet technique
    • Then we refine the extrinsics (rotation and translation from camera 3D space → projector 3D space) as a third distinct step
    • so each refinement has relatively few parameters and is fairly fast
  • Disable autofocus on webcam when calibrating :' ) (refocusing changes the geometry of the camera, breaks calibration!)
  • Block out skew parameter of cam/proj intrinsics, can assume that it is always 0 (OpenCV does this so we copy them and assume it's OK)
  • Instead of averaging extrinsics (3D cam→proj rotation and translation) derived from each pose, which doesn't make sense (you can't average rotations), we accumulate a list of point pairs from camera→projector space and then use Kabsch algorithm to get a rotation and translation estimate, then do the nonlinear refinement mentioned above
  • "Virtual visual servoing" pose estimation. Almost completely fixes flicker issues with 3D poses! This was surprisingly huge, and it's a really simple technique
    • Keeps a tiny bit of state (pose from prev frame) and finds the most likely pose for current frame using tag detection + prev state
    • Can get stuck in weird poses but fixes itself if you cover & uncover tag, this is tolerable for now I think
    • I really like that this is both pose estimation and 'smoothing' at the same time, and it has some intrinsic model that this is a thing moving around in space that we're trying to estimate – wonder if same technique could work for RFID
      • Wonder if we could also teach it about the square tag instead of doing servoing for each point on its own, would that make it more robust
  • Spent a while on performance micro-optimization, went from like 300ms latency to 200ms. Didn't really help on folk0 (and folk-live at my apartment is not that much faster even though its calibration is way better), so ruling out the performance theory of why folk0 doesn't work as well
  • Spent a while on making a separate correct-extrinsics step that you do at the end of calibration, where you leave the board flat on your table and it reprojects tags and measures the offset and tries to correct the extrinsics:
    • 
    • Can't figure out how to get it to really work – I guess the underlying theory here is that extrinsic offset comes from drift in the poses during normal calibration, so having a static pose (and biasing it toward the table plane where you care more) would help. But it hasn't really helped. Maybe I'm implementing it wrong!
• Made the calibration process more user-friendly:
  • Step-by-step instructions in the HTML, kind of like in folk-cnc:
    • 
  • Calibration projects the current pose, the critical path time between pose projection and detection, and the number of poses needed in total, so you don't need to look back and forth with your laptop – you can just focus on the board when calibrating:
• Measured the page so we can derive the whole page's 3D pose from the tag's 3D pose:
  • 
• Introduced the concept of quad, which is like the 3D equivalent of regions (but more constrained, always a quad, has clearly specified top/left/right/bottom, tagged with an explicit coordinate space, to reflect the way in which we've actually been using regions in practice)

3D calibration works extremely well (1-2mm error) on my personal system in my apartment, as you saw in the video above.

It works much less well (1-2cm off) on folk0 at Hex House, and even worse on folk-beads (the cart system at Hex House), where the high distortion and shallow 3D volume of the ultra-short-throw projector make it really hard to calibrate.

I've also made it almost fully backward-compatible with all existing code – it now derives 2D projector-space region claims from the 3D calibration, so all code that uses regions still works. I deleted all the old 2D calibration and homography and region code (calibrate.tcl), so 3D calibration backs everything on the branch.

I'd like to get it to at least be on par with the existing 2D calibration in most cases before we merge it, and we're not there yet… (working theory is still that the individual intrinsics of camera and projector are derived pretty well, but the extrinsics seem off)

Arcade got the 3D calibration branch working on their system:

(Not perfect – there's clearly that 1-2cm offset between camera and projector – but it's great to see other people get it working at some level, since we've been worried that the calibration won't generalize or that it'll be hard for people to learn to do.)

Arcade's calibration board, taped to a board game:

Omar: I've been continuing to work on the new preemptive/low-latency/parallel Folk evaluator.

At the beginning of June, I fixed a major bug where some Hold statements would just vanish half the time (so you might lose a program if you were dragging it around and got unlucky). It turned out that Holding the same statement multiple times could get you into a weird state (it would throw out the old statement but not make a new statement). (and this happens often when dragging, since micro-events might just fire with the thing in the same place, or you might drag it back over the same territory) I feel a lot better about the evaluator after fixing this – it seems a lot more reliable.

I made the edge-lists (of children of a statement/match) garbage collect on resize/defragment, which is something I'd been meaning to do & should solve one of the big leaks (and make the GraphViz graphs a lot more comprehensible, since they won't have lots of zombie edges).

I ported the setup.folk scheme from main so that I can run folk2 on a laptop and choose a non-default projector and camera in setup.folk. Had to touch some of the camera and GPU code to pass parameters in.

Still need to fix some other memory safety / refcount issues and some leaks, like we aren't freeing clauses yet. I've started running folk2 through Valgrind (it's actually a lot more doable than running original Folk through Valgrind – new evaluator does less stuff, is more pre-compiled, etc) and will probably do more in July. Found and fixed a workqueue initialization issue (have to do it before starting any thread so they can safely try to steal from other threads right away).

After fixing the Hold bug, we got to test the new evaluator on the actual table on folk0. (It's now reliable and fast enough to be worth testing.)

It performs really well, much faster/lower latency/higher throughput than any previous Folk evaluator including the current main one.

In terms of actually being able to fully switch to the new evaluator, the main open issues are 1. memory crashes and 2. labels/outlines/etc flashing (because statement revocation can happen too early and cause stuff to blink out, it's not strictly ordered to happen after new statements come in like it is on classic Folk). Once the crashes are fixed and we have a good way of dealing with statement revocation (local convergence? marking downstream statements?), we should start moving to the new evaluator.

Omar has done a couple new iterations of the portable gadget from last month (including picking up more CAD software, which as surprisingly inspirational UI). Here's what it looks like right now:

The hand grip and front look good, and the projector and camera and Pi 4 fit well. The hand grip is a little loose, so need rubber or a groove to keep it from swiveling around. (The Ultimems projector inside also swivels around and probably also needs rubber or a groove.) The back / wiring isn't fully figured out yet; that also will need a lid/cap that you screw in, I think. I still need to design and wire and do the trigger button, which I think will make a big difference to how you can interact with it.

I also have issues keeping it powered right now (and I want to figure out a USB power pack situation anyway, where you can choose to use wall power or a power pack). I wonder if the projector is too much for the Pi 4's USB port to power, or if it's overheating, or something.

There're also still a bunch of software improvements to make to Folk to make this thing usable (text rendering is broken on the Pi 4; you can't do traditional 2D stripe calibrate with this mini projector because it's too dim; etc)

• s-ol added FOLK_ERROR macros that can be called from arbitrary C code to immediately return to Tcl and throw a Tcl error
  • Omar: we've wanted this for a while – we should go through C code and remove a lot of weird calls to exit() and null returns and replace them with proper Tcl errors now. we've had it at the Tcl-C arg/rtype translation layer (it uses setjmp and longjmp to set a checkpoint each time you call from Tcl into C), but s-ol's generalized it so it's callable from user C code also
• Naveen Michaud-Agrawal added support for making connections & dynamic connections from points other than the region centroid (so you can connect to left edge, top edge, etc)
• Naveen added support for the alpha channel in RGBA colors
• Merged s-ol's work on international keymaps, so keyboards other than US can work (adds console-data dep)
• Andrés fixed Alt-Esc when a keyboard hasn't been visible to the system yet (it now loads default keymap out of the box, so Alt-Esc works to restart the system)

s-ol has added folk.js, a Web client library so you can emit and subscribe to Folk statements from inside a Web page. It also lets you easily convert between Tcl and JS values.

It feels like it can handle disconnect and status reporting and other such conditions a lot better and cleaner than one-off code. It significantly simplifies the implementation of stuff like the Web editor.

WS message are executed in the context of a match that is tied to the websocket lifetime. An emit prefix msg proc is provided, and the JS code dispatches messages based on the prefix value.

Utilities for encoding and decoding Tcl strings are provided, including a tagged template literal:

const complicatedString = 'This string {contains} some {} odd "stuff"';
tcl`We ${'embed'} ${complicatedString} here`
// 'We embed {This string {contains} some {} odd "stuff"} here' 

const json = [ { name: "Sol", age: 27 }, { name: "folk", age: "-1" } ];
ws.send(tcl`Assert the entities are ${json}`);
// 'Assert the entities are {{name Sol age 27} {name folk age -1}}'
 
const tclVal = '{name Sol age 27} {name folk age -1}';
loadList(tclVal).map(loadDict);
// [ { "name": "Sol", "age": "27" }, { "name": "folk", "age": "-1" } ]

It should be very useful and lower the barrier to many phone, laptop, other traditional computing integrations which we've had to do from scratch every time until now (like the work that Arcade has done with tablet and files, that Andrés has done with canvas, & that Omar has done with calibration and CNC preview endpoints).

• June open house:
  • 

• As mentioned earlier, our friend Arcade visited us from Georgia for a day!
  • 

• Our next Folk open house is on the night of Saturday, July 27 at our studio in East Williamsburg, Brooklyn.
• Getting 3D calibration to work well on all systems, then merging it…….
• Figuring out retraction and transaction behavior for parallel evaluator; testing it more
• Gadget iteration (need to fix swiveling, fix power issues, improve software)
• Video support
• Documentation (language reference, new kits and printable worksheets for setting up and learning the Folk language), open up GitHub repo

• The Task Gallery: A 3D Window Manager
• is_computer_on
• "programming tools as art"
• Emacs Doesn't Have Plugins
• The Factory (via Cristóbal)
• Ultimems MEMS Projector raster scanning laser at Computex 2024
• Pose Estimation for Augmented Reality: A Hands-On Survey
• time
• Rotation Averaging

• Teaching myself SwissGL
• How E-Ink Developed Full Color E-paper - a great breakdown from IEEE
• printable mini e-ink displays with their own batteries
• The first few art objects in this video remind me of large, polished, but simplistic versions of Kevin Kwok's e-ink crank
• Fascinating multi-colored stamp — watching this over and over, it’s inspiring a bunch of ideas around hand stamped fiducial tools :)