January 2024 newsletter

We're really happy to announce that the tabletop editor has been merged.

We've been working on it on and off since early last year, and although there's still a lot to do, we think it makes Folk feel like an entirely different kind of system now that it's in place. Folk is now a 'real' computer, in the sense that it can be programmed in itself, without depending on a laptop (and it brings all the usual advantages of Folk to the activity of programming Folk – you can show people what you're doing, they have an ambient sense of when other people in the room are programming, we can extend the editor easily, and so on).

• Final push from Andrés and some fixes by Omar: Fixed key drop issue by using timestamps and keeping statements around; fixed keypress state machine breakage by setting the input file channel to binary translation
  • Added a Keyboards page to help get keyboards set up:
    • 
    • The web-keyboards.folk code doubles as an (ugly but functional) example of making a Web dashboard, using the WebSockets interface for live feedback from the running Folk system (streaming keypresses). Lots of ugly escaping and nested code to run in different contexts, but it does work, and we want to make it easier and do more things like it.

• Some struggles to get Bluetooth keyboards working on folk-beads (the cart) – hack to fix driver issue with Intel AX201 Bluetooth worked. Always surprising how often we run into bad Linux issues with Wi-Fi and Bluetooth, on what feels like very standard off-the-shelf PC hardware

• You'll want to get your USB or Bluetooth keyboard and set it up to get the editor to work (map your keyboard into /dev/input/by-path, then use the Keyboards page to print out a tag to attach to the keyboard, basically)

• Using the editor in production, you run into some obvious issues, and it motivates some improvements we should make:
  • Event statements. We're hacking around their absence a lot (you have to retain keypresses for a bit to avoid dropping them, so we keep keypresses around for 5 seconds, but that requires some extra care to deal with – you now get replay bugs from old keypresses if you cover and uncover the editor)
  • Jitter is very obvious and distracting on the editor, since you're so focused on it. Could boost camera resolution, do some stateful smoothing, lock editor position in place. May wait until 3D calibration comes in to deal with jitter, though

• Interesting potential future directions:
  • More WYSIWYG (“draw a circle underneath this line of code”), more print design, more inline handwriting/field creation (may also require high-accuracy calibration)
  • More autocomplete and introspection of the running system to help you with programming (what statements are available for you to match? how many results is this query returning, and what are those results?)

• Jacob Haip and Andrés have both been working on OpenCV in Folk.
  • This will be a good way to quickly prototype computer-vision-based interactions that aren't just more AprilTag stuff – important for expanding the scope and feel of Folk
  • Jacob wrote a guide about how he connected OpenCV
  • Andrés has been reading papers on techniques for template matching, and they've been prototyping using OpenCV's template matching with Folk's camera slices. They're working toward an installation experience where you can try out various board-game-inspired interactions in Folk (for a showcase at NYU ITP in May).

Omar has been hacking on a parallel evaluator.

• Implementation is much more monolithic and self-contained than standard Folk so far: plain C codebase that compiles with `make`, statically links a vendored Jim Tcl, builds into a fixed executable
  • Easier to debug, faster to boot, and nobody really touches the kernel in practice anyway, and its structure is pretty fixed (has been for months), so we might as well make it static
  • (Faster to boot because it doesn't recompile itself from scratch on boot)
  • Probably cleaner package management situation too, we don't need to depend on a system `tclsh8.6`, you can just ship the Folk binary
  • 

• Tried multiprocessing but gave up, switching to threads for now. Some problems with multiprocessing:
  • Memory allocation (how do you do memory allocation? even if you can share flat heap objects, how do you share function pointers between processes, without using fork which is unreliable on macOS so impractical when I was just hacking on laptop).
    • Sharing function pointers came up very early in my experiments because I was trying to use libpqueue in an interprocess way, and it's generic and requires function pointers for all pqueue ops, so how do you get function pointers that work across processes…
    • This is already a problem in mainline Folk, where we'd forget to use the interprocess heap and then images wouldn't work on the display process or whatever.
      • Makes it hard to use standard C libraries and idioms if they implicitly depend on the non-interprocess malloc. But if we use threads, standard malloc should produce heap objects that work on any thread on any CPU. And supporting C idioms is definitely something we want in Folk
  • How do you use the Vulkan context across processes? Does all display stuff have to get pinned to one process?
  • How do you use the video4linux2 camera binding across processes? Does all camera stuff have to get pinned to one process?
  • (Maybe/hopefully we can still do error recovery on threads instead of processes, although it won't be as good. Would be nice for a segfault in a Folk C program you're hacking on to not take down the entire system, just current worker thread)

General approach is to do naive pure-C, copy-into-shared-string, one-big-lock version of everything first, which I'm hoping will already be much faster and much more parallel than the current evaluator (which uses Tcl heap objects, isn't parallel at all, and still copies a lot, I think)

There'll be NUM_CPUS-ish worker threads; each holds a lock while it's touching the database or priority queue; they're constantly pulling off next work item (Assert/Retract/Say/Unmatch) from the shared priority queue. (This is how the mainline evaluator works, except it's just one loop pulling off the priority queue.)

(Doing this all in a parallel way kind of requires the entire database, including statements and terms of statements, to be pure C data, since Tcl heap objects can't be shared across threads or processes)

(It'll be fun to get to the point of thinking about scheduling policy – detecting When blocks that are taking a really long time, maybe setting up additional worker threads as some get blocked, and so on. We don't have any way to entertain concepts like that in mainline Folk right now. It'll also be fun to really push the purity of Folk code, where global state will all literally have to run through statements.)

To give motivation for all this, I'll quote myself from the November 2023 newsletter:

The main motivation is actually sort of a user interface motivation: I want people to feel comfortable writing programs that hang or segfault, and I want people to feel comfortable putting 30 different programs on the table, without being scared that they'll crash the whole Folk system or break their (in-system) editor. That requires Unix process isolation, I think, and it requires preemptive multitasking.

I was showing people Metrics at the open house recently – I would flip stuff over and show how much performance improved the more stuff I flipped over, until there was like 1 program out and it would fly (100fps+), and the smoothness was noticeable and impressive. And it made the system much more fun to play with. I want the system to feel like that all the time!

(in other words, why should adding a new program slow down all your other programs? it doesn't on a normal OS, up to the actual limit of your hardware)

• RFID progress: Omar: Can now do multiple hops
  • Went from old
    • 
    • to new
    • 
    • Notice how only the first 2 queries get responses from the tag in the old plot, but in the new plot, we get responses on all queries.
  • Problem was that we weren't filling in any gap after the RN16 on subsequent hops if it was shorter than the first RN16 we sent out, so you'd have some weird invalid bits/garbage from the previous RN16 in the 2nd/3rd hop most of the time, which presumably made the tag fail to respond
    • (I was wondering why the pattern is always 'it works once or twice, then it fails' – that suggests that there's some state in the system that gets broken, not that the failure is purely random)
    • (I figured this out because I zoomed into a plot, printed it out, went through each bit in the tag's RN16 by hand, checked if we were sending that bit properly, then noticed extra unaccounted garbage at the bottom:)
      • 

• Next: start reimplementing out-of-band system, now that we can do a full round of in-band transmissions
  • Also need to transmit IB bit boundaries to the OOB or to the supervising PC and make sure our sync pulse scheme works – I had plans for most of this, but they're from months ago, need to see if they all make sense in practice
  • As a reminder, the broader context here is we're trying to get more accurate and faster localization – the rearchitected RFID system will have a much more accurate idea of timing, will be able to end-to-end validate its data using the protocol checksums so we aren't using garbage data to localize, and (probably most important) we should be able to throw more antennas at it with the better timing sync scheme that it'll have, which should help narrow down localization.

• January had a bunch of friends (Daniel Pipkin, Geoffrey Litt, Steve Krouse, Charles Chamberlain) come by to see the current state of Folk, try printing programs, and messing around with new program shapes. One new idea Geoffrey came up with was using a jar as a dial — Andrés loves how the code is wrapped around the jar as if it's a label:
  • 

• Jonathan Dahan has gotten good performance from Folk running on a Pi 5:
  • Omar: I was surprised at how much better it works than the Pi 4. PCs are faster and a better deal, but Pis are nice because hardware support is good / it's standard hardware, they're small, they have GPIO pins… maybe we'll go back to recommending them at some point.

• Govind Manian's home Folk setup is making progress

• Ian Johnson has been using Folk with the camera resolution set to 4K – this is good enough to have printed programs that are the size of playing cards:
  • 

• We had our monthly open house just now, on January 31 – it was nice to roll out the downstairs cart system and to show off the new editor:
  • 
  • People also played with Cristóbal's animation system from last month:
    • 

• Our next Folk open house will be on the evening of Thursday, February 22, at our studio in East Williamsburg, Brooklyn.

• New collaboration (to be announced) on deeper phone/laptop integration and knowledge management

• Ian Clester, who we worked with on RFID and who built the music demo last year, is in town to do a workshop with Livecode.NYC, so we want to brainstorm a little bit about future audio

• Polishing Folk/CNC interface for use by other people at Hex House and in general
  • USB stick install / Linux distro (can we make it a turnkey install if you have a CNC machine and want a Folk UI for it?)
  • System reliability (can we feel sure that people can walk up to it and use it?)
  • Finishing up the 3D/high-accuracy calibration work from late last year to actually do the machine preview in Folk

• Refining the editor & follow-up work on the Folk system: jitter reduction, handling event bugs

• OpenCV: dealing with memory management bugs, making easier wrappers for template matching capabilities from OpenCV in Folk programs

• RFID: reimplement the out-of-band component of the system, get syncing and bit boundary transfer working, maybe start localizing again!

• 100 Years of Fictional UI
• Peachpuffs and Lemonchiffons — a talk about the history of CSS color names
• Multiple-Computer User Interfaces: "Beyond the Desktop" Direct Manipulation Environments
• Two papers that helped me better understand shape matching this month:
  • A new Algorithm for shape matching and pattern recognition using dynamic programming
  • Shape Matching and Object Recognition Using Shape Contexts

• Turtlestitch - Coded Embroidery
• https://github.com/pfalcon/foreign-dlopen
• Stepping over a `brk` instruction on ARM64
• QArt Codes (via Ian Clester)
• The genius tool LEGO wants to forget (via Tinnei Pang)
• The Maison Alcée high-end clock kit teaching the art of horology at home
• https://github.com/mmorearty/hardware-breakpoints