May 2024 newsletter

• Cristóbal Sciutto has been working on a demo that uses Folk to display book covers from a digital library – under the hood, it's doing some fun new things with calibre-server:
  • 
  • 
  • Omar: Two observations about Cristóbal's library system:
    • it's always cool when we can get concrete data into Folk that has inherent user interest, rather than self-contained demos
    • this is a good example of the little nicety of http:// image url support paying off and composing, would be much more annoying to do this without that (hopefully we can parallelize the http get and calibredb calls in increasingly clean ways as we improve the interpreter)

• Vedran Budimcic has been building a Scratch-like environment in Folk. This month, he added support for multiple characters:
  • Vedran: I think I only need to do a few more things before I can let me kids try to break it: 1. Put the cards on something thicker like cardboard. Easier for little hands. Maybe laminate as well. 2. Consider character actions from left to right instead of order of appearance. Right now it only considers the actions on the right of the “play” pages, but doesn't consider their order.

• Andrés:
  • We have a booklet that shows you how to animate letters in Folk via position, rotation, and kerning. For a few months this has had an annoying deprecation warning from updating our graphics API, so I took a morning a few days ago to refactor the programs to the new graphics API (e.g. Wish $this draws a circle with color red) and add scaling animations to the final page. Here's the final result:
    • 
  • I've been looking at The Book of Shaders again recently and was inspired to try taking a few of the introductory examples and porting them into Folk. Below you can see I took the first three shaders (a solid red shader, a flashing red to white demo, and a resolution coloring example), implemented them in Folk's Vulkan shader pipeline, and then glued those onto printouts of the chapters I took the demos from.
    • 
    • I translated a ~300-line shader titled "Raymarched Hexagonal Truchet" from Shadertoy into Folk, it was a much simpler process than expected:
      • 
  • I adjusted the resolution coloring example to take up the entire screen — impractical when sharing the table with others but could be fun for presentations.
    • 

• Andrés presented their project at the final show for the NYU ITP Fellowship program for the '23-'24 academic year.
  • It was a collection of games (a word search program, pac man, pong, and a mini drawing demo) that could be run inside of Folk. People had fun making their own image markers as game controller pieces:
    • 
  • Setup at the Clive Davis Gallery:
    • 
    • Andrés and Omar disassembled the cart system from ITP (in Downtown Brooklyn) and brought it all back to the studio in East Williamsburg:
      • 

• Omar added Commit (on PAGE) so Andrés could make 'copy program source code into tabletop editor' work properly – fixes a long-standing omission where you couldn't change Commits from outside the committing program
• Omar added a GPU function wish – another thing we should have done a while ago
  • Replaces needing to manually run Gpu::fn on the GPU process
  • 
  • Slightly updated the GPU guide to reflect this
• s-ol bekic added support for paginating long programs so they print on multiple pages! (as well as A4 support) we've wanted this for a while
• s-ol made a lot of improvements to the live USB setup.folk config file, so you can set hostname and configure a printer
  • Omar added persistence of calibration and printed programs, and fixed file permissions for git pull, so the live USB is totally usable as a long-term system now!
• Omar did a cleanup of the repo, removing the pi/ and play/ folder (preparing for opening the repo more to the public)
  • Inlined hardware interface code into the virtual-programs – should make it much easier to read since all GPU stuff is in one file, all AprilTag stuff is in one file, all camera stuff is in one file, etc
  • Removed a lot of globals, publishing code/lambda through statements instead
• Andrés added support for CSS-style color formats in drawing:
  • 0xNNN or 0xNNNNNN
  • rgb(N,N,N)
  • hsl(N,N%,N%)
• Naveen Michaud-Agrawal added a ''crop'' option to image rendering:
  • 
• Andrés added a rotation slider on the web editor, so you can easily test what happens when a page is rotated before you print it
  • 
• Omar has been working a bit on the portable gadget from last month
  • Got a good hand grip, working on and off on CAD for enclosure (need to fit Pi, Pi camera, threaded inserts for hand grip and projector, and be able to physically fit my hand in to assemble all of it; think I need to break it up into panels)
  • 
• Vedran fixed a major breakage with printing AprilTags (came from changes to generate image to project AprilTag, broke printing)
• Omar did a tiny amount of work on RFID localization, fixing some misaligned RX/TX spacing issues – still trying to get it to reliably run for more than 1 round (not slow down over time, bail/reset properly on fail so it can continue):
  • 

folk-cnc is now working and released in a basic form. You can follow the link for installation and usage instructions. This feels like the absolute minimum useful thing, and we're looking forward to taking it further.

Omar finished porting Owen's gcode parser and tool calibration and preview logic from the original prototype. Here's how CNC tool calibration looks, where you move the machine to the 4 points and type their CNC coordinates in:

Tested it with a basic job – notice the projected preview in green vs. the actual cut in the wood:

Not perfect accuracy, but already useful & exciting.

Potential next steps:

• Once 3D calibration is in, and assuming the machine is stable, you might be able to get by just specifying the material plane for each session (drag a tag) and reuse most of the existing 3D-space calibration (might also make calibration more accurate on average if you have that model already)

• Make Folk drive the CNC machine (gcode sender) – should help automate/recheck CNC calibration also if it can move the machine for you

• Socializing the system so it can get used by other people in our studio space
  • Connects to making Folk drive the machine: ideally people can use the machine through Folk, not need to plug it into their laptop

• Move more of the UI into Folk / tabletop and away from the web UI (although the web UI is honestly not bad, it would be cool to have more design tools in physical space)

Prompted by the portable gadget and the CNC demo, Omar has been picking 3D calibration back up. We're hoping to get it merged next month in a backward-compatible way, so we don't have to port everything from 2D to 3D immediately.

It was in decent shape in December, so work on 3D calibration this month has involved a lot of small/polish tasks:

• brought it back up to date with main
• adding 2D compatibility in (it should be able to also do 2D calibration at the same time, if you capture some flat poses)
• support for choosing display and choosing camera (so we can reinitialize the camera at higher resolution when we calibrate – 720p usually isn't enough – and so I can test on a laptop plugged into external webcam and projector)
  • removes weird vestigial dependence on fbset, uses the proper Vulkan mode-finding stuff to get the default screen resolution!
• worked on calibration UI
  • you can now see the number of poses projected on the calibration board (so you know how far into the calibration you are + whether you should be capturing a flat table pose or an in-the-air pose)
    • 
  • little nice UI thing of pointing out the inner tag length that the user should measure in green
    • 
• maybe trying to improve accuracy a little bit
• performance stuff (actually very related to improving accuracy, it might be the main problem – the more latency you have, the more time the board position can drift in your hand, reducing calibration accuracy)

Omar has been doing a lot of work on the new folk2 evaluator (as discussed in previous newsletters, aiming for much better performance, lower latency, higher reliability, something more like Erlang w/ preemptive multitasking).

• Added basic support for Collect
• Added support for text & labelling
• Added support for images
• Fixed boolean support, array rtype support with C FFI
• Fixed some C/library concurrency issues, fixed incorrect locking on Holds
• Fixed really weird lldb issue with C FFI? Don't really understand this
  • 
• Either very clever or a terrible hack: use SIGUSR1 to interrupt arbitrary work on other threads that depends on a retracted statement
  • works for anything in Tcl, even infinite loops, etc. but doesn't interrupt C work (like a C infinite loop) because it depends on the Jim interpreter running its internal signal handler and checking its internal signal flag a lot (which is built-in behavior in Jim)
    • but could write our own signal handler, or use another signal after a few tens of milliseconds to hard-abort stuff that didn't respond, or something
  • It's not even clear if this is desirable in all cases: what if you 'livelock' because you're doing work that takes 500ms per camera frame? the behavior you want is to let each iteration run to completion, then start the next one, not just interrupt them all the time so that none of them ever finishes
    • maybe desired abort/fan-out behavior needs to be specified for each When? or could dynamically schedule?
  • useful for code that infinite-loops (like the Collect implementation right now) so it can get automatically reaped by the evaluator when the Collect goes away
• Thread-local workqueues based on Chase-Lev deque (original paper; C11 implementation; Andy Wingo's very useful note that points out some issues like this)
  • Replaces global mutex-protected priority queue; should be faster and better at natural load balancing across CPUs/threads, also lets us lock things to threads more cleanly
  • Also these slides about how to deal with dependencies, scheduling in general
• Added trace page (like the log we have in mainline Folk) for debugging specific leak or elision issues (why is this statement still around? why didn't this statement go through?)
  • 

In terms of functionality, it's surprisingly almost caught up to mainline Folk – the real issues are around performance and correctness

(Some of the changes in the rewrite are already paying off and got backported to mainline Folk: inlining of pi/ libraries into virtual programs, use of Vulkan built-in mode-finding for displays instead of fbset)

• May open house:
  • A lot of playing around with Folk programs / talking to Robert about reactivity and spreadsheets and text editing. Andrés put a program on their hat, allowing them to look at things on the table and label them “Hello”.
  • 
• s-ol set up a new Folk system at his lab in Milan:
  • 
• Forrest O. has been organizing their Folk table – it's looking really nice:
  • 

• Our next Folk open house is in the afternoon of Wednesday, June 26 at our studio in East Williamsburg, Brooklyn.
• Video support?
• We want to do a larger event as 3D calibration and/or the new evaluator and/or a new program count milestone come in. Watch this space
• Merge 3D calibration into main and start porting decorations/apps to use it natively
• Performance and correctness push on the parallel evaluator…
• Try to get RFID caught up
• Do another projector gadget iteration
• CNC + 3D calibration?
• Open up the repo? Docs for that?
• s-ol working on international keyboard support; need to review and merge

• Cilk Pousse
• The Handheld Is Dead! Long Live The Handheld! (via Leslie)
• "The angel librarian puts their reviews linked to QR codes and inside each book so they can hear their friends thoughts before checking them out"
• Opponent
• Lecture 6: Perf Optimization I: Work Distribution and Scheduling

• Attention Receipts: Utilizing Materiality of Receipts to Improve Screen-time Reflection [CHI2024] — cool to see a custom receipt printer used as a persistent utility.