October 2024 newsletter

(Our next Folk open house is on the evening of Wednesday, November 20, at our studio in East Williamsburg, Brooklyn.)

Multiple talks and presentations this month:

• Andrés gave a keynote talk at Causal Islands in Berlin on October 4; they brought a portable gadget to Berlin and demoed it during the talk
  • When the recording is posted (most likely in November) we'll post the link here. The talk used the gadget to change the slides; Andrés is excited to have a good recording of Folk being explained using Folk itself.
  • 
  • Kept it set up and showed it off all day:
    • 
• Omar and Andrés gave a short presentation and demo of Folk (using the handheld gadget) at the South Park Commons "New Behaviors" demo night in New York.
  • 
  • Some issues with the recording, but here's the video:
    • 
• Omar talked about his background & the motivation and ideas behind Folk on the Lux Riskgaming podcast with Sam Arbesman.
  • 

• Daniel Pipkin was trying "Folktober", where he made Folk programs based on the daily Inktober prompts.
  • 
  • He ended up making 8 programs: “Backpack”, “Discover”, “Boots”, “Binoculars”, “Trek”, “Passport”, “Hike”, and “Sun”

Andrés added a “desksaver”, like a screensaver, that has a welcome message and useful metrics (Folk clock time, CPU, RAM, disk usage, uptime, git branch, and most recent commit) when there are not programs out on the table. When there are programs, the desksaver instantly goes away.

While the basic feature is implemented, next week Andrés is going to make a few updates to the implementation (migrating web pages into their own Folk pages rather than web.tcl) and add a setting so individual Folk instances can turn this feature off at will.

Andrés and Omar have been showing the gadget a lot lately, both at outside presentations and at open houses, so Omar has made a lot of small gadget-related fixes.

These all come together to make the gadget feel much more like the computational flashlight we want it to be:

(these improvements all apply to the system more generally – so the gadget is paying off for the normal ceiling systems as well – but have been particularly acute or frequent with the gadget, because it has more tags at the fringe, it moves more, it has less RAM, etc)

• Fixed crash where number of Gauss-Newton iters gets really high when tag is near edge of camera space, which was causing segfault in those cases
  • I guess that we were just blowing out the stack with tens of thousands of iterations (and associated stack-allocated matrices) when tags are too off-camera since it just may never converge in that case
  • (normally you can get pretty good in 10-20 iterations, in my experience, so we could maybe reduce this cap further)
• evaluator: Did some heap profiling and fixed several memory leaks
  • (at first heap profiled with gperftools, and then with Tcl memory profiling, because Tcl uses its own allocator for ckalloc anyway and we mostly use that outside apriltag lib)
  • (and then finally just looked at the memory map in /proc and verified that the heap is the problem)
    • 
    • (then dumped the heap with gdb and looked at it in a hex editor and saw a lot of strings repeated, leading to the fix below)
  • Mostly just fixes in evaluator.tcl – Omar had been lazy and was allocating new strings in various places that should just be allocated once and reused (for stuff like ::matchId and for other dict keys that are fixed)
  • It still leaks, but way slower (the 2GB Pi 5 used to crash in 12 minutes or so, now at least 20-30 minutes uptime, could be much more). More to do here!

These apply only to the gadget camera (or to any other system that uses Pi cameras instead of USB webcams) – they're improvements to the camera-rpi.folk driver:

• Massively sped up gadget camera (5fps→60fps) by telling libcamera to use YUV420 format directly (which already has a gray plane that we can just copy) instead of using ARGB8888 and converting to grayscale
  • Saw that tracking was slow on the gadget, looked at metrics, saw that the camera thread was only running at 5fps so that was obviously the bottleneck
  • We were literally spending 200ms each frame converting the RGB camera image to grayscale it's like 99% of the flamegraph of the camera thread:
    • 
• Hack to fix dequeue timeout crash (which sometimes makes camera stop working after a while): increase timeout in libcamera environment config
  • 
• Noticed that the Pi wide-angle camera has focus control, unlike the normal camera module, and we hadn't set the focus, so it was at the default of 2m away, which is way too far; locked focus closer, to ~30cm away from the camera which seems like a good start and seems to help with recognition stability a bit

The combination of the Pi 5, ribbon-type micro-HDMI to HDMI cable, and pico projector (Ultimems HD305D1-C1) seems to sometimes (maybe only at the Linux fbcon level? outside when Folk controls Vulkan) produce garbled output. I figured out this is because the EDID data doesn't get through properly to provide a reasonable default resolution?

Solution: hard-code the EDID to 720p in /boot/firmware/cmdline.txt.

Sometimes you only see llvmpipe (the not-useful software renderer) and not the actual Pi 5 GPU on vulkaninfo --summary. I've found that this gets fixed by doing sudo rpi-update (brings back a good Pi kernel with good GPU support?)

Omar: I wanted printing that we could take on the go (to the SPC demo night and abroad), so I revived Ian Clester's indexcard layout for printing on cat printer receipts & got cat printer printing over BLE to work.

The receipts are about playing-card width, so I've been carrying around a deck of playing cards and attach each program to a card (to make it flat and also feel more object-y).

(only the print layout changes are really part of the Folk codebase, but it's still nice to see all this come together and understand the Linux and CUPS steps needed)

Folk sees the cat printer as just another CUPS printer (that you manually configure to use indexcard layout in setup.folk); I've been installing and running NaitLee Cat-Printer which provides CUPS interface to the printer; Ian got this to work on his system last year, but we hadn't tried it ourselves & we hadn't updated the layout as the printing system has changed.

Finally, I made cards to scan for and connect the printer, so you don't need a laptop to do it (in theory):

Here's the “Connect printer” card:

Its source code:

Wish $this is labelled [exec curl -s \
  -X POST http://localhost:8095/connect -d {{"device":"MX06,DF:07:26:13:53:BB"}}]

Omar: I've been picking this up again in October because I want to get it to work.

And I have good news: I fixed the bug that had me stuck in May and June, where the in-band reader would crash after the first round completed. It turned out that the problem was that I was using the % (mod) operator in the hot transmit loop (to decide the index of the sample to send out), millions of times per second, and that operator is actually pretty slow. I added another counter and if statement instead, and that works great:

Now it can run as long as you want, which means we should be able to use it live (move a tag around and see how it responds), and (most important) we should be able to use it alongside the out-of-band radio, so we can start to actually localize again.

So I've started working on out-of-band, which does exist in the repo but has bitrotted a lot (with all the optimizations and refactoring done to in-band to make RFID reading actually work on time, all the code it shared with OOB has been broken/removed).

We need to build the OOB logic again (what samples to transmit and receive), which isn't too bad since it's much less intelligent than IB and not on a deadline, it's mostly just bookkeeping and hopping, and we also need all the server and synchronization stuff to boot OOB, timesync OOB with IB, feed the samples to the PC.

Anyway, with the IB RFID reader working, I feel optimistic that we can bring this all together and release it in some useful form.

One issue has been that reading the tags is finicky (sometimes you have to move it, or it doesn't work as well on my home system, etc) because the new fast decoder is so much less robust. (maybe continuous reading instead of one-round will help with this once we get it?)

(the reader is live & on a deadline now, so it's just using a binary threshold on amplitude, and it only uses the first few samples to compute that threshold; the old decoder looked at the whole stream and tried to decode it in the optimal way).

How to threshold something like this? So much stuff that's in the middle even on visual inspection.

You often get read failures when the tag responses are sketchy like that right now.

• In late October, we held our usual monthly open house at our studio:
  • 
    • (drawing by Shenai Chan)
  • Our friend Lia Coleman drawing an animation for people:
  • We showed our now two gadgets – one handheld (now with a battery unit so it doesn't need to be plugged in), and one on an arm clamped to the table:
    • 
• Paul of the Upstate Carolina Linux Users Group set up a new Folk 'lamp' in their space:
  • 
  • 
  • (replacing their previous system that used a much bigger, more unwieldy projector on a ladder, I think – this one uses an AAXA P400 and a boom mic stand)
• Daniel brought a small Folk system to the Ink & Switch unconference after SPLASH in Los Angeles:
  • 
  • He printed out most of the virtual programs to show:
    • 
  • Some discussion on Discord about the questions and takeaways and issues – see the next section

Hard-won experience from what things we needed (& what things broke down) (and what questions we got) at mobile demos over the last couple of months:

(at Hex House or at home, we have a little more scope to just restart things, we have laptops and other tools around, the Wi-Fi is known-good, etc, so these problems are not as acute)

• Sometimes printing of tags on receipts is a bit faded – makes detection less reliable
• Calibration gets knocked around if the front panel moves (which it will, it's only attached at 2 screw points) or the camera wiggles off its PCB (the glue seems weak? is it the heat?)
  • Solution 0 (done): superglue the camera back to the PCB
  • Solution 1: make camera attachment to gadget more rigid; put the threaded holes on the gadget chassis instead of on the separate panel
  • Solution A: make calibration fast and reliable on the go
    • or have a fast, quick calibration refinement we can do, rather than the full calibration
• Printer status not clear, pressing Ctrl-P can be uncertain
  • Solution 1 (done): Printer control cards
  • Solution 2: Ambiently show printer status, print job status once you hit Ctrl-P?
• Wi-Fi is questionable as a way to talk to Folk – you need to manually get it on the network SSID and password & sometimes local connections are blocked
  • Solution 1: use Bluetooth Low Energy
  • Solution 2 (achieved): carry portable wireless USB keyboard that can always go to a Linux terminal, enter Wi-Fi options, do other stuff
  • Solution 3: some kind of wired serial terminal that you can drive from laptop?
  • Solution 4: run an ad-hoc Wi-Fi always? haven't figured this out yet
• Reliance on power outlets is very annoying – what if you have a meeting at a cafe? what if you just want to carry the thing around the warehouse and point it at walls?
  • Solution: Waveshare makes a standalone battery unit for the Pi 5 that is perfect – it's not a hat, so it works with our existing design – Velcro it to the top of the gadget for now
    • 
    • (maybe do a respin with a built-in battery hat? unit would get significantly bigger)
    • does this break the projector?
• Calibration not reliable?
  • → “camera occlusion seems to be a big problem (i don't really feel this but it could be the position from which I was presenting)”
  • → “My arms kept getting in the way. I think more than occlusion, the tag masking wasn't working great for some of the programs I had out and that was getting misinterpreted as being an occlusion problem.”
  • → “yeah, if you're leaning hard on tag masking I think you need 1mm accuracy – often I would say that tweaking exposure is the key thing (and there's a lot we could do in the UI to support making better calibration, too)”
• Daniel: Would be good to pre-print:
  • Recalibrate card
  • Display camera frame on table card (to help adjust exposure while calibrating, debug tag detection failures)

Daniel's escape hatches: bring a laptop, bring a mobile hotspot, ssh, use the web ui

• Our next Folk open house is on the evening of Wednesday, November 20, at our studio in East Williamsburg, Brooklyn.
• New gadget design with built-in battery? with support for the off-the-shelf AnyBeam? maybe trigger button finally? second camera?
  • hope: we can actually make it turnkey to manufacture or sell or buy one, not be limited by imports
• Get RFID out-of-band radio working, start localizing again
• New evaluator scheduler needs to be finished
• Improvements to 3D calibration to make it easier to run on the fly & to get good results: exposure control, feedback on how many tags were detected, post-calibration refinement procedure
• Write in fields using the high-accuracy 3D calibration + some OCR system
  • First example: geometry setter?
• Upgrade documentation
• Create printable booklets and exercises to interactively teach people Folk programming
• Add video support
• Add more features to the drawing APIs
• Revisit making shader syntax more terse for common operations

• https://restofworld.org/series/digital-divinity/
• https://x.com/poetengineer__/status/1841198424420729055
• https://viewsproject.wordpress.com/2023/12/01/tocapus-history-and-perceptions-of-an-endangered-andean-graphical-communication-system/
• https://web.stanford.edu/class/history34q/readings/Kittler/There_is_No_Software.html
• https://okmij.org/ftp/Computation/ARPL.html
• https://electricks.info/product/spotted-dice-set/
• https://journalhistoryknowledge.org//mapping-uncertain-knowledge
• https://www.todepond.com/sky/normalise-dont-share-lol/
• https://idind201.wordpress.com/wp-content/uploads/2013/10/do_not_knock_theodor_adorno.pdf

• PaperTouch — a tangible paper interfaces that take advantage of conductive touchscreens
• "This $30 Camera Is Truly Special " (video) — A neat 4 minute video about a thermal printing instant camera designed for kids
• Programing Design Systems — a class website with a bunch of instructive writing on techniques for using programming to generate are that's encouraged to be printed using a variety of novel techniques
• Classic HCI Demos
• Ray Marching Soft Shadows in 2D
• Ditherpunk — The article I wish I had about monochrome image dithering

• https://www3.nasa.gov/sites/default/files/atoms/files/nasa_graphics_manual_nhb_1430-2_jan_1976.pdf - The NASA Graphics Standards Manual is a work of art, that I wish they would revive
• https://summereternal.com/blog/studio-architecture - A ZA/UM (Disco Elysium) successor studio with some amazing philosophy behind what they want to do.
• https://www.inkandswitch.com/patchwork/notebook/ - The I&S lab notebook for patchwork is really cool, I'm making some little tools based on it :)