Our next Folk open house will be in the afternoon on Sunday, November 29, at our studio in Williamsburg, Brooklyn.

Karl Rosenberg brought this midi keyboard in, and Omar generated a very small tidal synthesizer/driver for it, so he could play it:

Here's the program:

# KOMPLETE KONTROL A49 -> Tidal/SuperDirt.
# Blocking MIDI input; note-offs cut the corresponding key.

Wish $this is outlined green
Wish $this is labelled "MIDI piano"

fn tidal {code} {
    set fd [open /home/folk/music/tidal-input w]
    puts $fd $code
    close $fd
}

set midi [open [list |stdbuf -oL aseqdump -p "KOMPLETE KONTROL A49"] r]
set midiPids [pid $midi]
On unmatch [list apply {{pids} {
    foreach p $pids { catch {kill SIGTERM $p} }
}} $midiPids]

try {
    while {[gets $midi line] >= 0} {
        if {![regexp {Note (on|off)\s+([0-9]+), note ([0-9]+), velocity ([0-9]+)} $line \
                -> state channel key velocity]} {
          continue
        }
        Hold! Wish $this is labelled $line
        set down [expr {$state eq "on" && $velocity > 0}]
        set note [expr {$key - 60}]
        set cut [expr {1000 + 128 * $channel + $key}]
        set gain [format %.3f [expr {$down ? 1.8 * ($velocity / 127.0) : 0}]]
        set sustain [expr {$down ? 10 : 0.05}]
        tidal [subst {once $ note "$note" # sound "superpiano" #\
                      octave 5 # gain $gain # sustain $sustain # cut $cut #\
                      room 0.1 # nudge (-0.28)}]
    }
} finally {
    close $midi
}

One nice thing is that you can swap out the superpiano synth for any of the other built-in Tidal synths. (and this generalizes to all the properties of the synth! you could physically swap them or have some other UI to control them)

Omar: I was chatting with Ben Zweig about object detection models. He was excited about RF-DETR, so I decided to spend a few minutes trying it.

(If you know YOLO or models like that, RF-DETR seems like sort of their modern descendant, plus segmentation. So it has a few dozen known object classes out of the box, it's pretty fast, etc. Not that similar to the open-ended embedding and SAM2 and contour tracing stuff we've tried for object detection under the DrawTalking project, which I realized after this experiment.)

Tracking a keyboard and my hands and the cup (notice how hand ['person'] tracking works by far the best, and how the model is generally very fast, probably 30fps-ish):

Here're the RF-DETR engine and RF-DETR on-viewport displayer programs. (The locking bit in there is ugly, and there are still issues with slow boot time / lack of persistence, so if you cover it up it's annoying to wait for the model to re-load, but it generally works well.)

(also, this detector + switching folk-hex to 4K led me to add configurable camera slice width so the 4K image doesn't kill performance)

Tom Wildenhain, who we met at an open house recently, came by and made this laser reflector demo:

The laser beam shoots out of a source program and reflects off any other program that does Claim $this reflects. I think this demo is nice because it's a 'whole table' application of larger scale than a single page and it has its own 'language' of reflector objects that you can make many of. It's not just a set of a couple fixed programs filled with code that act as screens.

Later, Eric Chen and Karl cut out & attached a reflector program to Karl's hand, making the reflection even more direct:

(maybe we should have a hand or body tracking model so you can just ask for things like this and don't need to cut out the tag, make sure it's planar, angle it properly, etc)

One issue with this laser application has been flickering/blinking – I have a more cleaned-up version that uses -atomically in the right places to fix it, but ideally it would be more automatic.

• Paul, who has been working on OSC in folk, has also been working on demo using MIDI controllers with Folk:

• Small bug fixes:
  • Much better handling when a worker dies (Tcl interp cleanup, C modules retained, GPU structures cleanup, Python socket cleanup) so worker churn doesn't inevitably cause system exhaustion
  • Use POSIX locking instead of relying on pid not being reused (fixes bug where system thinks it's still alive after reboot)
  • Calibration info is keyed by projector and camera name, so we can hold multiple calibrations in theory (although the projector calibrations may fight if it's same projector)
  • Destructor race/UAF fix, where we now establish destructors before publishing/reacting statements, so that there's no risk of a subgraph spawning too early before destructor inheritance in the parent, parent going away, & then subgraph (with empty destructor set) accessing mistakenly-freed data that should've been destructor-guarded
  • Handle JPEG decode failures better, reject those frames if possible, don't crash
  • Fixed a race in Collect where an older query could complete after newer results come in and overwrite them with stale old results: fixes were to increment sequence number before the query isntead of after & advance stored sequence number even if set is unchanged
  • Fixed a Firefox (and other browsers?) memory leak with image objects in /camera
  • (Motivated by MIDI controller pushing it harder) Keep Tidal FIFO open even on worker cancellation, which should keep music from breaking until restart which used to happen a lot
  • Used -atomically to stop toy shaders from blinking under load
• Andrés: I've been working on adding an autocomplete feature to the editor. It's satisfying because it draws from the statements actually in the database, making it easy to collaborate based on what other programs exist in the system:
• Merged Mason's points API

Omar's been setting up the RTBW CNC Folk system in Vancouver.

This month, 2 major fixes to get that system running:

• Figure out how to focus long-throw 4K cameras so we can actually calibrate it (projector and cameras are mounted 15-20ft away from bed)
  • Needed spacer rings to get the right focal distance:
    • 
• Automatically turn projector on remotely (so we don't burn bulb hours and power by leaving it on 24/7)
  • Use PJLink – the projector there has an Ethernet port, send PJLink command over TCP in Folk systemd service startup
• Implement multi-camera calibration
• Implement CNC preview demo

Omar made a big board and successfully calibrated a giant 6ft x 10ft Folk instance (normal-sized board for size comparison):

103mm tags:

Surprising and impressive that normal-size tag detection works here (4K camera for 6ft x 10ft bed is not a lot of dpi!), although we probably won't use it much on this system:

Next, using SAM2 and the stereo cameras' calibrations (I calibrated the 2 cameras separately, then manually tweaked the hold file so all the statements could be present together) to automatically find and projection-map workpieces on the CNC bed:

CNC preview demo (runs autonomously right now):

Powered by a AutoHotKey script that reports (using curl to POST to Folk / endpoint) from controls on the CNC controller PC to Folk statements.

The script:

#Requires AutoHotkey v2.0
#SingleInstance Force

SetTitleMatchMode(3)

loop {
    hwnd := WinWait("Shop Sabre Controller")

    RunWait("ssh [email protected] -- sudo systemctl start folk", , "Hide")

    lastPath := ""
    lastPos := ""
    DetectHiddenWindows(true)

    while WinExist(hwnd) {
        try {
            path := Trim(ControlGetText("Edit1", hwnd))
            x := Trim(ControlGetText("Static1", hwnd))
            y := Trim(ControlGetText("Static3", hwnd))
            z := Trim(ControlGetText("Static5", hwnd))

            if !IsNumber(x) || !IsNumber(y) || !IsNumber(z)
                throw Error("Invalid CNC position: " x " " y " " z)

            pos := x " " y " " z
            body := ""

            if !(path == lastPath) && path != "" {
                content := FileRead(path)
                body .= "Hold! -on {" A_ScriptName
                    . '} -key "cnc gcode" Claim the CNC gcode is {'
                    . content "}`n"
            }

            if !(pos == lastPos) {
                body .= "Hold! -on {" A_ScriptName
                    . '} -key "cnc position" Claim the CNC position is {'
                    . pos "}`n"
            }

            if body != ""
                Post(body)

            ; Only remember changes after a successful POST.
            lastPath := path
            lastPos := pos
        } catch as err {
            OutputDebug("CNC upload: " err.Message)
        }

        Sleep(500)
    }

    DetectHiddenWindows(false)
    RunWait("ssh [email protected] -- sudo systemctl stop folk", , "Hide")
}

Post(body) {
    tempPath := A_Temp "\folk-cnc-" DllCall("GetCurrentProcessId") ".txt"

    try {
        file := FileOpen(tempPath, "w", "UTF-8-RAW")
        try {
            file.Write(body)
        } finally {
            file.Close()
        }

        exitCode := RunWait(
            'curl.exe --fail --silent --show-error --max-time 10'
            . ' -X POST "http://cnc-folk.local:4273/"'
            . ' -H "Content-Type: text/plain"'
            . ' --data-binary "@' tempPath '"',
            ,
            "Hide"
        )

        if exitCode != 0
            throw Error("curl exited with code " exitCode)
    } finally {
        if FileExist(tempPath)
            FileDelete(tempPath)
    }
}

A few different fixes for multi-camera calibration, mostly motivated by the CNC project where we have stereo cameras.

• The breaking change here is that /setup now has an explicit “Detect AprilTags” checkbox. It should be checked by default and auto-checked by migration on existing systems.
  • 
  • If “Detect AprilTags” is not checked for a given camera, you won't get tag detection on that camera. (We want it to be opt-in since many systems will have secondary cameras for laser detection, workpiece-finding, depth estimation, documents, etc.)
• 4K cameras get high decimation (we should probably make this configurable) to make performance better
• Moderate calibration bug fixes: we weight pixel errors properly instead of squaring, we increase maxiters from 200 to 400, and we normalize homographies so one pose can't throw rest off. This should make calibration behave better and more reliably (helped on rtbw cnc folk)

After learning we could use those cameras successfully on rtbw-cnc, upgraded to dual 4K60 cameras with a longer throw, mounted further up near the projector.

Old camera setup – notice how close it is to the desk, how low the mount is (a few feet below the projector):

New dual camera and mount – see how the camera is now not too far down below the projector, so more out of the way of users, less likely to be bumped, easier to mount all of it together, less warping on the image at the outer fringe since it's more straight-down view:

The new cameras have 4K60, not just 1080p60; have tried running at 4K and works most of the time. Excited to do more fields OCR and fine-grained detection stuff (snapping to text in books, reading drawings, etc).

Omar, Karl, and Audrey have been working on integration with Karl's DrawTalking project. This month, we got DrawTalking entities to render in Folk, and we started reporting tracked physical objects (from all the tracking work from the last few months) back to DrawTalking.

• Audrey has been working on getting realtime interaction working in Folk, where you draw an entity in DrawTalking on iPad and it renders on the viewport object in Folk, and then as the entity moves/interacts with stuff, it moves live in Folk:
  • First time demo of getting movement translated from DrawTalking to Folk shapes:
  • This video shows Audrey getting transforms and scale factors translating between DT and Folk in realtime:

Example interface point where a JS module in DrawTalking claims a DT object into Folk:

Early prototype that just did quads:

On folk-hex:

Some of the latest renderer code:

Then: how do we report detected physical objects (like a cup or a toy) back from Folk to DT? DT needs to subscribe to the detected object claim from Folk, then control the DT entity based on it.

Cup is controlling DrawTalking entity, which you can see drawn on top of it:

Not much TODO

Mason: this month has been pretty slow for Zicl, since I haven't had a ton of time to dedicate to it. But! Something exciting happened. Enough of the the system is functional that I was able to get all the core libraries to compile and evaluate, and so I was able to benchmark it relative to Folk's current interpreter. And…

2x faster to get core libraries running, and half memory utilization!

There's still a lot of work before I can say it has parity with the main codebase, including a long tail of broken features/incompatibilities, as well as cleaning up the code, but it's come a long way from the lowly tokenizer I started on a year ago.

We recently publicly announced the opening of the new studio space we've moved into, 87R; to celebrate, we had a six-month anniversary party. People played with the Folk systems at the party:

• Ben Zweig stopped by – he was excited about RF-DETR, so Omar made the RF-DETR demo, and we also talked about sliders where you hold it from the sides, so the fiducial marker is still visible from the top:
  • 
• Eric Chen, a friend of Karl, came by and saw the system
  • 
• Esen K. Tütüncü also visited via Karl
  • 

Tracey Le continued to bring guests from Recurse Center to work on Folk stuff on Fridays.

Playing with Marble Run:

• Our next Folk open house will be in the afternoon on Sunday, November 29, at our studio in Williamsburg, Brooklyn.
• Andrés: Working on a lamp gadget form factor at home
• Andrés: Adding video capability
• Andrés: Working on a static folk server format so programs can publish to the internet easily
• Omar: Still need to ship RFID tracker & do projection demo of distance to confirm visually that it works
• Omar: Fix CNC preview issues (flip, 90 degree rotation, workpiece height correction)
• Omar: Try filter on object tracker
• Omar: Hack on scheduler; try to make 4K input not mess up the system badly
• Omar: Maybe make a table harness?

• Touchscreen Archaeology
• Haptic pen
• Using a regular pencil and a battery to light up an LED
• Rolling computer set up at home
• mail art
• A Guided Tour of Sketchpad

• https://www.are.na/block/50966062
• https://melonland.net/surf-club
• https://www.youtube.com/watch?v=vm13_oXG2OA
• https://lwn.net/Articles/960913/
• https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit#pretrained-checkpoints