• Andrés will be giving the opening keynote at Causal Islands Berlin on this Friday, October 4.
  • The talk will use a Folk gadget to control slides, demonstrate Folk, and explain the technology and vision behind Folk. We'll link the recording of the talk in the newsletter next month.
  • It looks like livestream tickets are also available for anyone interested
• Our next Folk open house is on the night of Monday, October 21, at our studio in East Williamsburg, Brooklyn.

• Jessie Grosen added support to the supercollider branch of Folk for the syntax Wish $this plays note A. She and Andrés and Arcade Wise got together for some crafting and SuperCollider time, made a lot of buttons, and built a paper 'piano' on the Recurse Center Folk system:
• Omar fixed up folk-cnc after someone reached out about it (so it now works with latest Folk, which has different display dimension statement and tag statement forms now that 3D calibration is in place)
• Naveen Michaud-Agrawal implemented a 'parameters' system where you can control parameters of a page by moving a controller dial nearby:
• Arcade Wise made a bar graph system (it runs entirely in a shader):
  • “it supports anyone who claims to have graphable data” (!)
  • Graph cat:
    • 

Daniel Pipkin has been working on hand tracking. (using Robert Xiao's technique to ultimately get touch tracking, so you can touch things with your fingers)

Depth and IR data from the Kinect can now be piped through folk.

• Merged Daniel Pipkin's ESC/POS receipt printer work
• Andrés added a new keyboard shortcut: Alt-F1 to stop Folk
  • Useful for configuring the gadget when Wi-Fi is broken: plug in a USB keyboard, hit Alt-F1, and then you can use sudo nmtui by hand to set up Wi-Fi without needing to outrace the Folk process reastarting
• Omar merged subprocess death detection and reporting (as part of gadget libcamera support)
• Omar merged C++ support (as part of gadget libcamera support)
• Omar added support for drawing dashed stroke including offset, so you can animate the dashes.
  • (used for outlining the border of the gadget – shown later in this newsletter)
• Naveen added a Bezier curve drawing wish
  • 
• Naveen fixed a mailbox leak when a subprocess terminates
• Omar made a new live-build .img that has a working default up-to-date setup.folk
  • (it was broken because it had no default camera/display selection statements)
  • Still need to make git pull work (its remote origin is set wrong because we copied the repo)
  • I should write a script to do this automatically…
• Omar fixed the statements graph, which was broken because the 3D-calibration-related statement node labels were too long (over 16384 chars)

Omar: Made a new gadget design that can also fit the older/more-encased Nebra AnyBeam. Built a second gadget for Andrés (to take to Berlin!) with the Nebra AnyBeam that we had lying around, so now we have 2 gadgets.

The gadget design is now on GitHub. (Not everything is specified yet – I need to measure some bolt lengths and link some Amazon stuff.) 3D print has been fairly stable for a month or two but probably needs a new revision where the front panel is more solid (doesn't wobble, camera doesn't get knocked off).

Two big camera fixes:

1. Switched to wide-angle camera
   1. (also had to expand the camera hole in the front panel to make the wide-angle camera fit)
2. Direct libcamera support in Folk instead of using libcamerify
   • + tell libcamera to configure (reduce) camera exposure time so that projected tags for calibration don't get washed out

These fixed the worst issues – it now actually calibrates! We can project onto programs! (it's still fairly annoying to calibrate, takes 20-30 minutes, but it's doable, and you ideally only have to do it once. I feel like we should ultimately be able to calibrate it really well because the projector is a laser scanning projector and has no lens distortion, also.)

First calibrated use of Omar's gadget (gadget-blue) and of Andrés's gadget (gadget-red), respectively:

Slow, but that also motivates performance improvements…

• Moving dashed line around the outside of the projector area
  • Actually makes a huge difference: now you can tell where the gadget's area of effect actually is, don't have to guess, and you can see at a glance if the system is still live or if it's blocked/frozen by whether the dashed line is still moving.
  • Code you can add to ~/folk-live/setup.folk to enable this:

    • When display /disp/ has width /w/ height /h/ {
        When the clock time is /t/ {
          Wish to draw a dashed stroke with points \
            [list [list 0 0] [list $w 0] [list $w $h] \
              [list 0 $h] [list 0 0]] \
            color white width 10 dashlength 40 \
            dashoffset [expr {fmod($t, 10)*-120}]
        }
      }

• Changed the 'page fault' mechanism to run ''curl'' in background so it doesn't block Folk awkwardly when you're pointing it at random programs

Omar has been continuing to work on the new evaluator – it really does feel like the biggest remaining issues in Folk are performance & reliability issues, so this is important.

There's a mix of little reliability/safety issues and big semantic issues with the evaluator (how do we schedule stuff onto threads? how do we make sure things don't blink out or, conversely, transiently get multiple states at once? when/how do we abort invalidated work items?)

Started working on sysmon thread which manages the size of the thread pool (too many threads looking for work? kill some; too few threads compared to number of CPUs? spawn some)

There are subtleties here where you want to avoid churning the thread pool and constantly killing and re-spawning stuff. Haven't figured out yet.

Added workqueue display on threads Web page, which made it clear that many of the random issues with folk2 are just that work-stealing was breaking down and work items would get permanently stuck on some thread that was stuck running a permanent task (instead of the work getting stolen and executed elsewhere).

Idea: sustain/TTL field where statements can get held for some number of milliseconds after their parent is retracted.

sysmon (which previously was just managing the size of the thread pool) is extended (maybe will rename to custodian) to also handle reaping sustained statements when they hit their deadline. it feels nice to do it here rather than complicating the priority queue further, since sysmon is already waking up every couple milliseconds anyway (and it doesn't need to run any more than that, fixed ticks are fine).

We had a discussion about this proposal in Discord:

Omar: when you make a When (or a statement) in parallel Folk, we might have an option for “sustain” or “time-to-live” where you can be like, I want this statement to stick around for 1ms (or 10ms or 100ms) even if its parent is retracted, in case it comes back in on the next frame

just thinking about anti-blink measures. it wouldn't surprise me if we need more little hints like this to make the system work

Is there a particular context where this came up?

it's pretty general – you'll see page outlines blink out in the new evaluator, because they get retracted because the old camera frame is retracted & that is a race against the new camera frame getting processed (and if the retract happens before the page gets re-added due to the new frame, it blinks out)

the specific one that just came up is that we were exec-ing some keymap command a lot because the keyboard page kept blinking in and out (downstream of the above)

we also do this de facto in single-threaded Folk right now by ordering retractions after new assertions in the priority queue

Why does the old frame get retracted?

retracting old statements is part of the core loop of the system – you can kind of think of the camera subsystem as a while loop that just goes like this

set prevFrame {}
while true {
   set frame [camera readFrame] ;# blocking, takes 16ms
   Assert the camera frame is $frame
   Retract the camera frame is $prevFrame
   set prevFrame $frame
}

and then when the Assert happens, the new set of active pages is calculated, and the new outlines of all pages are calculated, and then the shader display list is calculated, etc, and that's all in place

and then the Retract happens and rips out the old set of active pages (but any pages that are still on the table are now supported by the new Assert so they don't blink out)

(and then only once the single-threaded evaluator has converged and all tasks are complete, after all of this, the final display list is rendered on the GPU, so you don't see the transient retracted old statements)

And in the parallel case Retract is potentially happening before the set of active pages are reactively set as a result of the Assert?

Yes, exactly – it's a race since those operations are parallel-scheduled in the new evaluator

So the TTL could live in the camera frame claim or maybe the tag detector to make detected programs stickier. I think I got it. There is something that feels a little wrong about putting a time on it instead of declaring some sort of dependency, but I get it.

Yeah
I'm hoping that we would only need to use them internally in a couple specific places
and the end user programming experience would broadly be the same

hmm yeah i think the problem is that you don't really want to introduce a hard dependency because then programs can arbitrarily block the control loop if they're slow (which is sort of the situation we have in current single-threaded Folk)

it feels like at some level, you do actually want to say, hey, if you can't resolve within a few milliseconds, we're just gonna go ahead and continue with the next frame

anyway this (blinking) remains an open problem so i'm gonna try it when i get the chance and see how it feels

This is a really good point. We would want a timeout on a lock anyway.

Fixed some keyboard issues where the keyboard process was erroring or exec-ing stuff all the time (because of lack of persistence / because of blinking), and where the grabber was broken because it wasn't inter-thread-safe.

• Our open house this month was well-attended, and we got to show off the gadgets and eink device:
  • 
  • Andrés practiced his talk for Causal Islands Berlin:
    • 

• Our next Folk open house is on the night of Monday, October 21, at our studio in East Williamsburg, Brooklyn.
• Andrés will give their talk about Folk at Causal Islands Berlin
  • Maybe some tasks afterward around merging the slides system they built and following up with people
• Andrés continuing docs and desksaver projects
• Andrés and Omar will work with our new partner Kosmik on knowledge management and phone/laptop integrations and demos with Folk
• Daniel continuing to work on hand tracking now that the Kinect data is in Folk
• Omar wants to respin the gadget to make the front panel more solid, maybe add a second camera, & is meeting with Ultimems (the mini projector supplier)
  • Maybe also see if there is low-hanging fruit on performance
• Maybe also add a second camera to folk0
• Omar continuing to work on new evaluator, work out thread pool management and sustain/retract behavior (and fix general instability)
  • Maybe finally implement statement or match arenas so we can unify memory management
• Omar needs to get back on the 3D calibration refinement and UI (I think even adjusting exposure, without an additional refinement step yet, could make a big difference, & reporting tag detect results during calibration so you can see if it's iffy)
  • could also include livestreaming the camera feed to the browser

• https://teaching.ellenmueller.com/walking/2021/10/10/stanley-brouwn-this-way-brouwn-1962/
• https://www.hillelwayne.com/post/alan-kay/
• https://grafematik2024.sciencesconf.org
• https://x.com/rsnous/status/1540793581337604097 (gadget ideas)

• Send emails from Cloudflare Workers
• How Swift Achieved Dynamic Linking Where Rust Couldn't
• What is an application binary interface (ABI)?
• User guide for Borland C++ editor
• Print your digital reading list as a magazine, shipped monthly
• Newspaper Club (print your own newspaper as a service)
• Supervision, an interesting looking computer vision library
• Book design lecture from Ben Denzer via the Parsons Communication Design Lecture Series