Our next Folk open house will be in the evening on Wednesday, July 29, at our studio in Williamsburg, Brooklyn.

Also: Daniel Pipkin and Mason Jones will be exhibiting Folk at Craft Lake City's DIY Festival August 7-9 in Salt Lake City.

• Audrey Gu improved the language reference / README documentation (Bluetooth keyboards, Hold! example)
• Omar: Fixed bug where killed workers would lose work items on their local queue, which caused problems with ball simulation
• Omar: Use tgkill to kill threads on Linux
• Support editors separate from keyboard, now that default editor is so aligned/at risk of interference with printed code
  • An independent editor pointed at a page:
    • 
• Omar: Added /camera-slice/<id> web endpoint, so you can view the camera slice of any object on the table
  • Useful for debugging local perception models!
  • Shows how camera slices are all automatically rewarped to be properly rectangular (even if they're skewed, rotated, etc in the camera frame)
  • 

Andrés: I finally merged the Revive drawing capabilities (new and improved!) PR! So now you can do:

• Wish $this draws a circle
• Wish $this draws a circle with color gold radius 20mm
• Wish $this draws a circle with color magenta radius 20cm position [list x 20% 50%]
  • and you get this:

We enforce units (mm, cm, m, %) on everything in the new drawing system which is nice as it makes it easy to tell, using your real-world brain, where the shapes you’re drawing are going to land.

• Error reporting requires you to look at the shapes.folk log on the web dashboard right now:
  • 

• By the end of July we should have a fix that bubbles up errors to the top level reporting mechanism on the table, which should make this annoying reliance on the web dashboard for shapes.folk errors disappear.

Omar added support for shaders from ShaderToy – as long as they only use iResolution and iTime, you can just paste them into Wish $this draws toy shader { ... } and they'll work.

Some shaders (copied from ShaderToy) in action on the table:

We've had an issue about this for a while, since the internal shader/pipeline API is mostly just meant for use by the drawing primitives, not for creative coding, but this new feature was immediately motivated by talking to Karl on Discord about richer drawing APIs – Daniel Pipkin did an initial attempt, and then Omar built on the approach and figured we might as well go all the way to ShaderToy compatibility if we're going to have a user-facing API for toy shaders.

Omar: After the to-scale printing we added last month, I started working on 'fields' that are inline in the source code:

This is an interaction we've wanted for a long time, where the code becomes an interaction surface and not just documentation for people. The idea is you'd write in those green boxes and then your writing would get OCRed and returned to the caller of the Field function.

But our calibration is still not quite accurate/stable enough for the projected green fields to precisely hit the expressions in the code. How can we have more of a closed loop where we can nudge the calibration to exactly hit targets on the page?

Idea: print each program with this black ring below the line numbers:

Then, at runtime, try to project a dot into the black ring, using the existing calibration and known page geometry:

Now you can (hopefully) look at the camera slice for the black ring and see how far off the projected dot is from the center. That's the offset that your projection is from the 'correct' place to project to hit the physical page.

Nudge the projection by whatever that offset is (in pixels? meters? unclear) such that the white dot will perfectly hit the center of the black ring.

This makes sense in theory, but doesn't help much in practice yet. But I think we want some way to self-calibrate and account for these sub-centimeter inaccuracies in projection mapping, so I probably want to keep digging into this.

We've been continuing to work with Karl Rosenberg at NYU on an integration with Karl's DrawTalking system.

We started by building a 'viewport' system to act as an arena for the storytelling to happen, where you can dynamically define the viewport by moving these 2 corner pages:

Then you can print your viewport as a fixed page that locks that region of the table in, so even if you don't have the corner pages on the table anymore, the viewport is still locked in:

Karl was able to cast shapes from his iPad DrawTalking onto the table, so it seems like we'll be able to use the iPad to hold the model and then use Folk as a view.

Next step is to track physical objects and expose them into DrawTalking as entities for the drawn characters to interact with. Karl brought in this blue cup as an example object to track:

We spent a while on object tracking, which led to the contour demos that we'll detail next. Omar, Andrés, Audrey Gu, and Karl all got together to discuss object-tracking strategies:

Inspired by this paper that Karl sent over, we started by trying out pure contour tracking to track the cup in the viewport. Omar vibe-ported Lingdong Huang's PContour from Java to C (equivalent to OpenCV's findContours – we just didn't want to pull in OpenCV, and this is a self-contained function anyway, and it's easy for Folk to bind to C).

This new "CContour" library takes in a binary black-or-white image (so you have to binarize the RGB image from the camera first, and your binarization threshold makes a big difference in what gets detected and connected), then traces contours around connected dark regions. It returns a list of contours (a contour is a list of pixel coordinates in the image; we usually scale it up to meters in Folk after we get it). Our wrapper library has a tunable lightness threshold, minimum contour length (to get rid of 'confetti' contours so you can focus on bigger objects), and simplification level (CContour automatically simplifies contours, which is great to reduce Tcl and display list traffic):

One way to think about contours is that they give you geometry, not just masks/images, so you can project outlines of things, you can conceivably do geometric queries like points-at or overlaps, you can make a virtual bouncing ball collide with the things, etc. With geometry, your tracked objects can interact with the rest of the Folk object universe.

(also interesting that contours are sort of another geometry representation that is different than the quads we use for most programs… not sure how to unify this)

Our demo traces all the contours on a camera slice for a page/viewport (we have a few different variations that trace on the page itself, or draw on the side, etc, but they all are pretty similar code).

In our demos, each separate contour gets colored differently – you can see how it pretty successfully traces the drawn sketches and the blue cup:

I would say there are a few big issues to deal with here to get the object-tracking interactions that we and Karl ultimately want.

First, you can see how contours all change color from frame to frame – we can't identify that a contour is 'the same' object from frame to frame without some additional logic.

You could imagine tricks for this: we could compute an average color or color distribution for each contour, and we could use that to distinguish a blue cup contour from a red stapler contour (identity by color) (that also might be a nice visualization anyway). We could look at the contour shape (identity by shape) (although this runs into rotation and the distortion problem described below). We could find the closest contour from previous frame and identify it with new contour in the new frame (so identity by location/trajectory).

But the point is, it's not something you get for free from the contour detector (unless you assume you're looking for exactly 1 object in the frame or something).

Contours are traced from a binarized image, so all dark areas that are adjacent get merged into the same contour. You might have one object that accidentally splits into multiple contours (if it has light sections). You might have multiple objects (and shadows) that get blobbed into one contour!

The results are lighting-dependent, which makes it hard to make a turnkey solution. Sometimes it gets sunnier or darker over the course of a day. There's a risk that the whole page is dark, or all the objects are light. The thresholds would ideally continuously adjust based on lighting conditions. (You could judge by the AprilTag black/white pixels; you know a priori that white blocks on the AprilTag should be classified as light.) (In addition, it's also affected by the exposure parameter on the webcam, which we also should probably automatically control.)

Also problematic if you have a cup and its shadow and they merge into a giant blob contour:

It's hard for me to imagine a scheme that can deal with this with pure contours – you'd need to get the threshold exactly right to cleave the cup away from its shadow, and they look pretty close in luminance to me.

This issue is subtle, but it might be the hardest issue to actually fix.

For the DrawTalking demo and a lot of other purposes, we kind of want a contour that is the 'foot' of the object, the part touching the table or viewport or program underneath. But the contour you get is from the POV of the camera, which is generally overhead and off on the side, so the contour gets this weird position-dependent section of a bunch of the upper part of the object:

You can see that if we move the viewport toward the center of camera image and make its plane parallel to the camera plane, we get a much less distorted camera slice and contour:

Karl's idea is that during setup/intake, you would draw a manual foot contour to impose, given the cup center:

But how do you get the cup center to do that? You still need to go from distorted contour to center somehow.

The identity and separation problems from earlier I think are generally addressable with SAM2 (which we'll talk more about next month) or other simple or prebaked techniques. This distortion problem requires some object-specific knowledge: even if you know that it is a cup, you have to know how to undistort it, and you need probably a different policy for each object, or a 3D model to fit pose for each object.

There are a number of strategies we could use to detect / correct distortion. One that we've gone some of the way toward is supporting two cameras pointed at the same space, so you have stereo vision (which would also help with pose misestimations). You could then segment/contour on both cameras and merge the information to get an accurate contour somehow?

Tried switching from 1 NexiGo (1080p60 webcam) to 2 cameras (NexiGo and Brio 4K):

I think this is still viable, but the physical setup is annoying to switch to/from, and it pushes the USB bus hard (the Brio 4K wrecks whatever hub it's connected to), and there's this giant shadow. And the Brio 4K can't do 60fps at 4K! So shelving for now. I want to try some simpler schemes.

There'll be a lot more on this in the next newsletter, but one approach to solve the identity and contour-failure problems is to use SAM2 and then contour-trace that instead of using a raw contour.

I was out of town for a few days, so was testing SAM2 on my laptop. (I've been sort of hoping to use my laptop or some other powerful PC as a model server for stuff like SAM2 so not all Folk systems need a powerful GPU.)

But performance of SAM2 on macOS / M1 is weirdly bad, taking 400ms to process each frame (as opposed to the 60ms on the RTX 4090 on folk-hex).

It seems to be this issue with PyTorch interpolate, which is used to scale some features in the SAM2 pipeline.

Omar: I continue to be sort of frustrated with performance – I talk in particular later about how basic blocks of Folk code should not take 100+ microseconds to run – the practical payoff for this is for weaker systems (where we literally can run out of CPU time to do the frame work, leading to glitching) and general responsiveness feeling and for the code editor (which is probably the most intensive thing we run). We want it to feel comfortable to do as much as possible in-system, rather than resorting to a laptop.

For a few months, I've been meaning to try the magic-trace tool from Jane Street. For the last couple of years, we've relied on Tracy to profile Folk, which is pretty good for Folk-level issues and has a nice UI and can be pretty granular (tracks multiple threads, has low-overhead annotation), but I don't think Tracy is the right tool to profile interpreter-level issues.

I finally got magic-trace to work this past month; it was pretty annoying. Out of the box, it didn't work with the latest version of perf. (the perf traces were outputting syntax that it couldn't parse). So I had to use this random fork (and eventually just fix it myself for more syntax like that). Then I had to make a bunch of changes – to get it to compile with llvm-18, to explicitly link libraries, and to use the llvm ld instead of GNU ld.

But magic-trace seems extremely, extremely useful for debugging the Tcl interpreter itself, where individual operations can be a few nanoseconds (so don't show up on sampling and are expensive to annotate with explicit tracing spans), and the behavior isn't consistent enough to aggregate well (it's super branchy since programs do all sorts of different things). An actual trace from it is a lot more useful than squinting at Tracy.

I also had to figure out how to actually invoke magic-trace – set up the trigger symbol in Folk so you can trigger a trace (and make sure the compiler doesn't elide the call),

invoke it properly for highly-multithreaded Folk (magic-trace is not super thread-aware), change Folk build settings to produce sensible traces (you have to turn off tail call optimization!).

Disadvantages of magic-trace: it needs an explicit trigger + it only runs on literal Intel CPUs (no AMD, no ARM / Raspberry Pi / Orange Pi / Apple Silicon)

Mason Jones spent some of last year working on comprehensive changes to the Jim Tcl interpreter to support sharing objects between threads. The changes work, but weren't noticeably faster than the copying and reparsing that we do right now.

The new branch does feel more elegant and like the 'right' direction, where we can just keep everything as refcounted Tcl objects forever, instead of having trie-managed strings and interpreter-local objects and copying and reparsing between those. In theory, we could get rid of a lot of code by unifying all this stuff.

I wanted to dig this branch back up now that we have magic-trace and really go into depth about why it's slow (and why Folk Tcl blocks are so slow in general).

I merged main back with the new branch to catch it up and dealt with the moderate amount of conflicts / changed stuff / other Tcl interpreter changes we'd made since last year (Mason's branch changes some of the Tcl-C APIs for getting strings and stuff, so any new calls to those needed to be changed, too):

(It's a little ugly in some parts, where I finished the merge so that trie terms are now just Jim objects – I got rid of Folk's db Trie and Clause types – so there's more intermingling of the Jim and Folk levels:)

Talked with Mason a bit about some tactics we could use in addition to object sharing:

Here's a specific performance question we can use to compare Mason's interpreter and the main interpreter:

Why does tag masking take 100 microseconds?

I added a trace trigger at the end of this When block, which is in fill.folk and gets called every time we draw a black quad to mask out projection above an AprilTag. Each of those block executions takes 100 microseconds on folk-hex, which feels way too slow to me. (and folk-hex is pretty fast! it's even slower, more like 300 microseconds, on my home PC)

So let's look at some traces of this quad-fill block.

(These first few are a little messed up because I hadn't disabled tail-call elimination yet.)

Here are a couple of traces of that quad block on main (not on Mason's cross-thread-object branch yet):

If you zoom in, you can see that instances of SetListFromAny and SetDictFromAny toward the left side are taking a lot of the time.

• I think the first Jim_ListAppend → SetListFromAny happens in folk.c when we build up the env stack. (~12 microseconds)
• The Jim_DictMerge → SetDictFromAny → SetListFromAny to the right of that is probably in applyBlock in prelude.tcl.
• The Jim_DictKeysVector → SetDictFromAny → SetListFromAny a bit later is probably the dict getdef $colorMap inside the evaluation of dict with options block – it has to reparse the whole color map from string to Jim dict, which is slow
• Then you see two branches afterward which I think are the two draw Wishes

You'd hope that all the SetListFromAny and SetDictFromAny here are shimmering of objects that were copied from another interpreter, and that they'd just drop out on Mason's branch, because the objects are shared and already come in dict/list form (don't need to be shimmered from string representation anymore). That would, in theory, make this thing like 2x faster.

Here's a good trace on Mason's branch after I removed tail-call optimization:

You can see a few things here. Basically, a lot of the shimmering (especially the color map one) does drop out, but the rest of the trace is slower enough that the whole thing is still slower overall.

The two peaks (each has a JimInvokeCommand at top) representing the two Wishes are still present.

On the cross-thread branch, each of these Wish calls takes 40ish microseconds:

On main, it's pretty similar, but each Wish takes 24ish microseconds:

it's weird because structurally main and cross-thread have similar traces, but 1. cross-thread is completely missing the expensive string→dict because it presumably is sharing the color map and other stuff 2. everything in cross-thread is just like 30% slower
so it ends up being mildly slower on net
like the for loop also shows up on the main trace

Our theory is that the multithreaded reference counting is 2x more expensive which slows down all interpreter operations.

There are some immediate lessons here for both branches.

• You can see that the term-processing for loop in Say is actually pretty expensive (the section under Jim_ForCoreCommand). We could probably optimize that loop (use foreach, a switch statement, stuff like that) or even move it to C. Would help a lot in workloads that make a lot of statements.
• We could probably speed up the db operations a bit.
• The env and source line/file management is also surprisingly slow – need to deal with this somehow.

Mason: I was finally able to get enough of Zicl running to test its performance against the main branch, and… it's about ten times slower. Profiling it showed a lot of time spent in my custom allocator and in string generation. So I've been reflecting on what went wrong, and I'm afraid I've had a bad case of Second System Syndrome. I've realized that I overcomplicated the heap design, and made my code have an impedence mismatch with Zig's and C's conventions. So, I'm currently in the process of redoing the internals with a much simpler and hopefully faster design. I think it's working, since the current diff has net 2500 lines removed, with hopefully more removed as I continue refactoring (the abstractions are much tighter this time around). It's about 60% ported, so hopefully I can test it in a month or two.

• Andrew Le Clair visited on June 1st:
  • Andrés walked Andrew through making his first program in Folk using the table editor:
  • He also made this experimental type animation in Folk:
• Cedric Honnet (who helped organize the Scalable HCI trip in China in January) visited and played with the system and the gadget:
  • 
• Irena Wang co-worked for a day and tried out the latest updates:
  • 
• Aidan Strong visited
• Jessica Stringham visited

Omar, Andrés, and Brian went in and fixed up the system at SVA in Manhattan.

Omar and Andrés worked with Audrey Gu to fix up the Recurse Center Folk system.

We wanted to figure out why the system felt so sluggish.

JPEG decompression of 4K webcam images was taking up a lot of time!

We moved the camera down (so tags are bigger in the image) and downgraded resolution to 1080p. Even after we went down to 1080p, it was still a little slow (20ms instead of 60ms). That might be an inherent problem with that computer + libjpeg-turbo (single-threaded, each CPU is not /that/ fast on this PC).

Even on my laptop, which is pretty fast, decompressing a 1080p image takes like 5ms.

Ideas: Try to parallelize the decoding on CPU (not obvious if this is even possible)? Or decode on GPU with nvjpeg/VA-API (ffmpeg should wrap this), like how a lot of webcam software probably works? Would give us some extra CPU time to run actual Folk tasks, too, if we're not spending it on JPEG decompress.

We had a really lively open house this month. Brian brought a lot of friends from ITP Camp who were curious about Folk.

Amanda Yeh made a color spinner (program that changes hue based on angle) and put it on a dolly and made it circular:

• Our next Folk open house will be on Wednesday, July 29, at our studio in Williamsburg.
• Daniel will be exhibiting Folk at Craft Lake City's DIY Festival August 7-9 in Salt Lake City. Reach out to him on Discord if you have any questions!
• Omar: further push on performance using trace to make the in-system editor really usable
• Omar: new gadget design; our current one is falling apart, so it might finally be time
• Omar: SAM2 object-tracking system: distortion correction, make some interactions around it
• Omar: Pick RFID project back up
• Andrés: Fixing silent shapes.folk errors

• Interactive Color Map (CIE-Lab)
• Mark Weiser - Some Computer Science Issues in Ubiquitous Computing (1993)
• The Digital Antiquarian - Deep Histories of Analytical & Text-Based Games
• CIE Color Spaces & Uniformity Guidelines for Vision/UI Ergonomics
• Understanding Hall Effect Switches & Magnetic Prototyping Mechanics

• https://en.wikipedia.org/wiki/Exposure_fusion
• https://direct.mit.edu/books/edited-volume/3814/HCI-RemixedReflections-on-Works-That-Have
• https://www.are.na/omar-rizwan/linedrive
• https://www.goodreads.com/review/show/3040698985
• https://kprotty.me/2021/09/12/resource-efficient-thread-pools-with-zig.html
• https://parentheticallyspeaking.org/articles/pedagogy-recommendations/
• https://fil-c.org/fugc
• https://www.instagram.com/reels/DZGC6EChvGK/
• https://www.kmjn.org/notes/automated_assisted_design.html
• https://2026.html.energy/
• https://store.steampowered.com/app/2460350/Brave_Mouse_Cartographer_Trilogy/
• https://old.reddit.com/r/computervision/comments/1nv4d8u/basketball_players_recognition_with_rfdetr_sam2/
• https://news.ycombinator.com/item?id=41105087