November 2024 newsletter

(Our next Folk open house is at noon on Sunday, December 15, at our studio in East Williamsburg, Brooklyn.)

• Andrés added:
  • Print notice when you hit Print, both on web and tabletop editor
    • 
  • New /programs design with categories; much easier to read:
    • 

Omar: I've spent most of this month working on the new evaluator (folk2).

I've made a huge amount of progress; it honestly feels like I spent months stuck on blinking/flickering/ghosting bugs and then solved them all last week.

(Remember, the goal of folk2 is to have 60fps under bigger workloads, so we can use multiple tabletop editors at once & do other demanding stuff natively in the system. I think it's just important that the system feel really good and smooth, anyway. Need to be running programs in parallel, avoiding queueing, also just general evaluator optimization. Another goal is to make the system hard to crash or freeze just because one program misbehaves, as in any other operating system.)

There's still a lot of blinking under the new evaluator, but no ghosting, and I think the blinking all now emerges from genuine failures to meet timing (the label blinks out because the program region wasn't regenerated in time for the current frame).

Notice how the camera hits 60fps but the full-frame detection is only 20-30fps (this is using a 1080p image):

• When (serially) construct to replace infinite loops (statements retract properly)
• Tail trace so you can trace latest stuff going on
• Replaced the broken refcount scheme with gen+rc+alive in one atomic variable, which makes a lot more sense and seems to work
• Now timestamping work items, figure out elapsed time, scheduling based on that
• Fully separated match/statement 'removal' (which removes descendants) from memory freeing (which happens when rc = 0 and not alive)
• Fixed locking of destructor list
  • 
• Fixed locking of child match/statement lists
• Steal-half now actually works (fixed overflow, fixed stealing 100% instead of half)
• Fixed lsorts in infinite loops (not using now that we have serial When though)
• Now showing each statement's match children & statement grandchildren in web UI (makes debugging much nicer!)
  • 
• Ports from folk1:
  • dashed-line (want it for gadget marching ants)
  • camera-rpi throughput improvement using YUV420 format
  • implemented clock time (also want it for marching ants, and any kind of animation really)
    • now runs on an independent clock from the frames! every 2ms
  • hack for 'descriptor indexing' on Pi GPU so images & text work
  • incremental tag detection to be able to hit 60fps

Performance (CPU, stack) profiling is most of what I'm up to now, trying to eliminate the blinking.

I've already fixed a lot of weird issues using flamegraph profiling and made it noticeably smoother. Like replacing rand with rand_r to fix this:

I'm surprised at how much of the flamegraphs are just threading stuff, not Tcl-bound. Maybe I'm misreading and need different graphs (the mutex time is just time when it's scheduled off the CPU, not wasted time). Here is an average worker thread:

Now working on actually using Tracy to get a full view of what's going on, which I've been curious about for years.

There are still persistent and pretty bad memory leaks on folk2, from 10MB to 200MB per second depending on how much stuff is out. I want to get to these soon after I fix the performance stuff.

• Heap profiling with gperftools
  • Too slow, doesn't seem to detect most of the heap?
• mimalloc arenas
  • Might still be useful for images and stuff, but we'd need to migrate the arena between matches and statements as statements get reused, it's more complicated than expected
• Sealed Tcl objects
  • I had been hoping to do this forever – it seems elegant and cool and efficient – but it turned out to be much harder than expected when I looked at it – would need to change a lot of callsites in Jim Tcl (anywhere it shimmers an object) to make it work, or do horrible hacks with COW pages.
  • 
• CPU-oriented scheduling
• Spinlocks
  • Linus is probably right
  • I got too excited since we have fairly tight locking discipline in folk2
  • Look at all this spinning; it's not an improvement over mutexes:
    • 

Omar: Didn't have time to do much this month, but I did figure out how to make the IB program not crash in its new connection-dependent style. We now just drop samples from the radio until a PC connects and we set up a FIFO and RFID protocol parser then. (The current goal is to get IB and OOB both running and synced up to a reset signal over TCP from a PC, then stream info from IB and OOB to the PC.)

This project isn't particularly blocked on anything, thankfully – just needs more time to put IB and OOB together and implement localization and then release it.

Omar: Minor improvements:

• Ad-hoc Wi-Fi procedure, so you can boot the gadget and connect to its ad-hoc network from a laptop (to ssh, see web UI, edit programs, debug) without needing to get it on whatever the local Wi-Fi is
  • tried to use NetworkManager, totally failed and gave up
  • (there are downsides to ad-hoc Wi-Fi as a default, like it can't sync with other Folk systems. BLE still feels more correct… or maybe multiple Wi-Fi adapters)
• Focus/exposure adjustments
  • (this is really the only change I made to actual mainline Folk source code this month)

Right now, the biggest issues with the gadget in practice are:

1. decalibration because the camera is attached to the front panel, not the gadget chassis, and the front panel flexes
   1. solution: attach the camera to the gadget chassis! been meaning to do this for a while
   2. working on it, but haven't re-assembled and tested the gadget with the new case (I haven't figured out what to do with the front panel now):
      1. 
      2. (notice how the camera attach point – the four holes – is now on the case itself, so it should be fixed with respect to the projector which would also be rigidly attached to the case)
2. manual exposure not reliable (especially with a lot of sunlight/ambient light), tags often get washed out & don't detect when you just pull the gadget out somewhere
   1. idea: autoexposure?
   2. idea: self-autocalibration button/card that tunes exposure until it can see itself?

These are all 'minor' additions to the gadget that I want to throw into future revisions (and have ordered parts for).

I've been talking about a trigger button for a while, but been blocked by figuring out what connectors/cables to use so the handle grip with trigger button can be removable (in case you want to mount onto an arm). Realized that some commercially available camera grips just use a 2.5mm jack for a button output. This seems good. Let's use a 3.5mm audio jack (TRRS) instead. Threw that into part orders and will probably throw it into next design rev.

as for what to do with the trigger button, I'm thinking about the Super Mario Galaxy Wii remote interactions and sounds…

(I also threw a speaker into part orders.)

(And the battery system – there are a bunch of Waveshare products that can supply the needed 5V5A to power everything – I'm hoping something more hat-like will work and can be built into a future gadget case. It'll be bigger, but I'm tired of carrying around power supply, battery charger, battery module all separately, and I don't think we can easily just do USB-C anyway so the connection'll be awkward if it's an external module.)

I've been working on a new revision that uses the new off-the-shelf commercially available AnyBeam projector (you can just buy it on Amazon), so we'd actually be able to make them in quantity and give them to people. Trying to use as easy-to-get parts as possible.

The main (surprisingly big) issue is that this new AnyBeam only has a single USB-C port – no more HDMI in – which means it is incompatible with the standard Raspberry Pi 5, which can only do HDMI out.

(On the other hand, this'll be nice if we can get it to work, because it means one USB-C cable for everything, no more weird adapters and separate HDMI and power cables to fit inside the device. And it means probably more reliable power delivery to the projector through the standard USB-C cable, not running off a random USB-A port on the Pi anymore.)

So I spent a while looking at Pi alternatives.

• The Orange Pi 5 would be good if it worked (I already have at least one or two lying around, and it's easy to get), but there's almost no way to connect a decent camera to it without an additional custom board/adapter. (and if you're fabbing a custom board/adapter anyway, maybe just use the CM5 as below?) And it's a different form factor than the Raspberry Pi, so we couldn't use a battery hat.
  • It does work (but, again, the lack of a decent camera solution is a dealbreaker):
    • 
• The Radxa Rock 5B (or 5B+ if it comes back into stock) is probably what I'll try for the time being. (waiting for it to ship right now.) Annoyingly hard to get / expensive / long-term support might not be great (and it's a big board compared to the others, 100 mm x 75 mm), but it has an okay camera story, it's similar to the Raspberry Pi form factor, and it has USB-C video out.

(Both of these use Rockchip chipsets, and I have some concerns about Vulkan support on those, too..)

Anyway, I realized this week that the right thing to do might be to use the new Pi Compute Module 5 and design a new PCB carrier board for it (which would have a HDMI or MIPI-DSI to USB-C converter that could feed the projector, and proper MIPI-CSI ports for 1-2 cameras, and could fit onto a battery). Then we'd have full Pi camera and software and long-term support, plus the right outputs for the projector, plus we could probably shrink the hardware down a bit. But I'd need to learn how to do that

• Had our monthly open house:
  • 
  • Nice to see people playing around with the cube, the portable gadget (now that we have two, we can have one arm-mounted and one handheld), and editing programs on the table
  • Special guests: Paul from Greenville, SC where they've set up a Folk system, and Jessie who set up the Folk system at Recurse Center in downtown Brooklyn. Was nice to be able to talk through specific details/questions about people's Folk setups

• Our next Folk open house is at noon on Sunday, December 15, at our studio in East Williamsburg, Brooklyn.
• Fix blinking in new evaluator (that is, performance optimization so the programs resolve under 16ms)
• Create a pull request for the new evaluator
  • Port the remaining virtual programs/features
• Initial new gadget revision that uses the AnyBeam & Rock 5B
• Finish assembling new-old gadget that has fixed camera
  • Gadget trigger button
• Gadget autoadjust exposure maybe?
• RFID
• Actually revisit video support
• More convenient shader language
• Experiment with more tangible papercraft interface elements for Folk programs (using cardboard, rubberbands, etc.)
• Revisit a bunch of smaller system improvements that we've had in the backlog for a while

• https://queue.acm.org/detail.cfm?id=2513575
• https://swseng.io
• https://ayende.com/blog/187969-B/reviewing-mimalloc-part-i
• https://forum.malleable.systems/t/shouldnt-the-phone-be-our-target/235/25
• https://shop.keyboard.io/blogs/news/box-of-crap-november-2023-dubious-treasures-from-the-markets-of-shenzhen
• https://blog.archive.org/wp-content/uploads/2024/10/Vanishing-Culture-2024.pdf
• https://stackoverflow.com/a/25168942

• This exhibit showcases a variety of objects that reveal the use of touch and haptic feedback in every day objects.
• This clever cardboard pattern creates a neat holder for a variety of complex objects out of a single sheet of cardboard. In the video it's used for packaging but I've been thinking about how to use it to hold phones or small electronics on the table so they have a stable position for use in Folk installations.
• In this video a skirt transforms into a dress using multiple receipt printers
• Fold Up Toys is a great resource for papercraft template
• Leaflet is hosting an online Zine Week event
• tweet — Three.js interactive cube demo runs really well on Daylight's screen
• Generative Omnimate — “Learning to decompose video into layers”