• Our next Folk open house will be on TODO

• TODO: Andrés: Drawing merged
• Omar: Fixed bug where killed workers would lose work items on their local queue, which caused problems with ball simulation
• Use tgkill to kill threads on Linux
• Support editors separate from keyboard, now that default editor is so aligned/at risk of interference with printed code
  • An independent editor pointed at a page:
    • 
• Omar: Added /camera-slice/<id> web endpoint, so you can view the camera slice of any object on the table
  • Useful for debugging local perception models!
  • Shows how camera slices are all automatically rewarped to be properly rectangular (even if they're skewed, rotated, etc in the camera frame)
  • 

Omar: After the to-scale printing we added last month, I started working on 'fields' that are inline in the source code:

This is an interaction we've wanted for a long time, where the code becomes an interaction surface and not just documentation for people. The idea is you'd write in those green boxes and then your writing would get OCRed and returned to the caller of the Field function.

But our calibration is still not quite accurate/stable enough for the projected green fields to precisely hit the expressions in the code. How can we have more of a closed loop where we can nudge the calibration to exactly hit targets on the page?

Idea: print each program with this black ring below the line numbers:

Then, at runtime, try to project a dot into the black ring, using the existing calibration and known page geometry:

Now you can (hopefully) look at the camera slice for the black ring and see how far off the projected dot is from the center. That's the offset that your projection is from the 'correct' place to project to hit the physical page.

Nudge the projection by whatever that offset is (in pixels? meters? unclear) such that the white dot will perfectly hit the center of the black ring.

This makes sense in theory, but doesn't do much in practice yet. But I think we want some way to self-calibrate and account for these sub-centimeter inaccuracies in projection mapping, so I probably want to keep digging into this.

We've been continuing to work with Karl Rosenberg at NYU on an integration with Karl's DrawTalking system.

We started by building a 'viewport' system to act as an arena for the storytelling to happen, where you can dynamically define the viewport by moving pages to control its corners:

Then you can print your viewport as a fixed page that locks that region of the table in:

Karl was able to cast shapes from his iPad DrawTalking onto the table, so it seems like we'll be able to use the iPad to hold the model and then use Folk as a view.

Next step is to track physical objects and expose them into DrawTalking as entities for the drawn characters to interact with. Karl brought in this blue cup as an example object to track:

TODO blue cup

We spent a while on object tracking, which led to the contour demos that we'll detail next. Omar, Andrés, Audrey Gu, and Karl all got together to discuss object-tracking strategies:

Inspired by this paper, we started by trying out pure contour tracking to track the cup in the viewport. Omar ported Lingdong Huang's PContour from Java to C (equivalent to OpenCV's findContours – we just didn't want to pull in OpenCV, and this is a self-contained function anyway, and it's easy for Folk to bind to C).

TODO contour video

I would say there are a few big issues to deal with here.

First, you can see how the contour colors flicker from frame to frame – we can't identify that a contour is 'the same' object from frame to frame without additional logic.

There are tricks for this…

Contours are traced from a binarized image, so all dark areas that are adjacent get merged into the same contour. Problematic if you have a cup and its shadow and they merge into a giant blob contour. Also very lighting-dependent, which makes it hard to make a turnkey solution. The thresholds have to continuously adjust based on lighting conditions. There are tricks for this…

This is a subtle issue, but it might be the hardest issue.

TODO

https://github.com/FolkComputer/folk/pull/271

Some shaders (copied from ShaderToy) in action on the table:

Omar: I've been trying magic-trace. It was pretty annoying.

Out of the box, it didn't work with the latest version of perf. (th perf traces were outputting syntax that it couldn't parse). So I had to use this random fork. Then I had to make a bunch of changes to get it to compile with llvm-18, including changes to the build process to explicitly link libraries, and using the llvm ld instead of GNU ld.

But magic-trace seems extremely, extremely useful for debugging the Tcl interpreter itself, where individual operations can be a few nanoseconds (so don't show up on sampling), and the behavior isn't consistent enough to aggregate well (it's super branchy since programs do all sorts of different things).

Mason and I have started using magic-trace to figure out what is slow in the run blocks and in particular, why his cross-thread branch last year is moderately slower than main (we would expect it to be faster!)

• Andrew Le Clair visited on June 1st:
  • Andrés walked Andrew through making his first program in Folk using the table editor:
  • He also made this experimental type animation in Folk:
• Cedric
• Irena

* TODO: Omar and Andrés worked with Audrey Gu at Recurse

We had a really lively open house this month. Brian brought a lot of friends from ITP Camp who were curious about Folk.

Amanda Yeh made a color spinner (program that changes hue based on angle) and put it on a dolly and made it circular:

• Our next RSVP for the Folk open house on Thursday, June TODO, at our studio in Williamsburg.
• Daniel will be exhibiting Folk at Craft Lake City's DIY Festival August 7-9 in Salt Lake City. Reach out to him on Discord if you have any questions!

• TODO

• https://en.wikipedia.org/wiki/Exposure_fusion
• https://direct.mit.edu/books/edited-volume/3814/HCI-RemixedReflections-on-Works-That-Have
• https://www.are.na/omar-rizwan/linedrive
• https://www.goodreads.com/review/show/3040698985
• https://kprotty.me/2021/09/12/resource-efficient-thread-pools-with-zig.html
• https://parentheticallyspeaking.org/articles/pedagogy-recommendations/
• https://fil-c.org/fugc