Folk Computer

This is an old revision of the document!

htmlmetatags

Our next Folk open house will be in the evening on Wednesday, September 30, at our studio in Williamsburg, Brooklyn.

What we've been up to

General system improvements

editor: Only target the smallest program the editor points at
- Fixes this issue where an editor pointing at two programs at once results in two unusable/wire-crossed editors:
  - (especially annoying when on a viewport, since the editor would always point at both the viewport and any program inside the viewport)
Fixed how bare claims totally broke Folk by triggering everything
- (for example, if you just put Claim on a line on its own in a program, it would break the whole system)
prelude: Fix function replacement when a new in-scope ^function should stomp the old one
- Should finally fix the random space changer failures which would fill up the log! (was happening when space changer initialized to null changer and then filled later, but original changer function was not actually replaced)
/program/<whatever> Web endpoint: Show the edited version of the program if it's been edited

Atomically

dbInflightDecr on the exit path where provisional statement is immediately thrown away
- Should fix “(edited)” titles sticking around randomly / after program revert.
Multiple inheritance
One-way hasConverged (latch convergence)
Timeout by parent removal

Graphics speedup

Omar: As long as I can remember, the editor has been the slowest thing in the system, where you point an editor at a program and the whole system slows down noticeably. It's especially bad on programs that have a whole page of code (that's like 2,000 glyphs).

I benchmarked it this month and found that a 2,000-glyph editor slows the system down to 8fps (and this is on the global GPU thread, so it slows everything else down too).

Simple fixes: cache the C encoding of all Tcl draw calls and turn off the Vulkan validation layers. That immediately brings us to 40-50fps. Much more usable.

Next, I want to do more editing and terminal I/O in the system. I think we can also push draw call batching further and maybe then bring back the validation layer.

Object tracking / DrawTalking

DINOv2 (embedding model)

Embedding-based tracking has a number of advantages.

It lets you do re-identification.

It's fast – you can compute an embedding of a pose or viewport in like 10ms – and is pretty fast on almost any hardware, even my low-end home Intel N95 GPU (which can run DINOv2 in 100 or 200ms but takes 2 seconds to run SAM2).

It lets you get these really nice intepretable heatmaps of what the most 'cup-like' or 'army-toy-like' 'earbuds-case-like' (or whatever you want to find) regions of the viewport are:

DINOv2 with normalized cross-correlation

Here's DINOv2 with NCC but no rotation tracking, which is remarkably fast and accurate for a radially symmetric object like this cup:

You can get pretty impressive fine centers:

Rotation tracking

Harder to deal with asymmetrical objects. Need to record several poses with known rotations, find the best cross-correlation, use that to give angle.

folk-cnc

Omar: As part of the RTBW project, Brian and I have been mounting and bringing up the CNC system. Mounted steel strut to ceiling, hung 2 cameras and a projector (cantilevered on wood block).

Weirdly, both of our Intel N95-based mini PCs seemed to have fried SSDs when I tried them. I ran into this issue on my home system after about a year, too. Replaced the SSD and put a 3D-printed spacer in to make the new SSD the right length:

[Expect] and rewindability

Mason: Omar and I (among others) have been continuing to discuss what an [Expect] command would look like. The idea is you could use [Expect] like:

Expect Omar likes color /omarColor/
Expect Mason likes color /masonColor/
if {$omarColor eq $masonColor} {
  Wish $this is labelled "Mason and Omar like the same color"
} else {
  Wish $this is labelled "Mason and Omar like different colors"
}

This doesn't look too different than the current [Expect!] command, but it's more powerful, since it will rewind any effects it caused, then rerun the code. For example, if I changed my favorite color to blue, but Omar's stayed red, it would undo the “same color” wish, then run again, wishing the “different colors” branch the second time through. This would make a number of things easier to work with, but the biggest benefit in my mind is that you can have [Expect] inside a function. This means something like point conversion would become

proc pointProject {fromPoint toSpace} {
  Expect tag [lindex $fromPoint 1] has quad /fromQuad/
  Expect tag $toSpace has quad /toQuad/
  # Conversion code here
  return $converted
}
set projected [pointProject [point 0.1 0.1] 123]

Now “projected” would update any time either quad changed. It becomes almost like Excel, where I can just say “A1+B2”, and it'll automatically update as dependencies change. Except now it's just normal code! There's a number of finicky details though: [Expect] introduces some nasty race conditions, because it can be triggered at any point a new value comes in. This is particularily difficult with things like files:

Expect the filename is /filename/
set fd [open $filename]
set contents [read $fd]
# What if Expect gets a new value right here, and rolls back? [close] will
# never be called, so we'll leak the fd.
close $fd

So, you might say, just close every resource that's still open, right? Well, that opens up a different can of worms (though this is more contrived):

Expect the filename is /filename/
set fd [open $filename]
Hold! the fd is $fd
# What if Expect gets a new value here? It will roll back, closing fd in
# the process, even though fd stays visible.

I think in general I'd prefer the second option, because it means that things are closed instead of leaked.

Another (more cursed) idea I had that stems from rewinding is propagating errors by rewinding to the problematic statement, and having the error raised there. For example:

Wish to draw a circle onto 123 with cnter [list 0.2 0.2] ;# Has cnter typo
Wish $this is labelled "here"
# Code has now run both wishes

Later…

When /someone/ wishes to draw a circle onto /p/ with /...options/ {
  # Error is raised here, since no center was provided
}

Now, the error would propagate up to the original wisher, and the code in that block would rewind back to the problematic statement. That means the “labelled” wish would undo itself, the error would occur at the circle wish, and a stack trace would contain each intermediate statement that led to the error.

Also for those who are interested in how we'd handle this in the interpreter:

This is pretty closely related to multi-resumable effect handlers, so I've been looking at what they do to implement the second resumption. Essentially what is needed is to be able to snapshot the interpreter and stack at a particular point in time, and to be able to restore that state. That means a non-recursive engine groan. Currently both Jimtcl and Zicl are what are called recursive interpreters, which means when you call a function inside of Tcl, it literally calls the “evaluate” function recursively. So if you have a function call in a function call, it becomes three deep of “evaluate” calling itself. It's a really elegant way to make an interpreter, but alas, it does not allow snapshotting the interpreter state.

A non-recursive engine on the other hand doesn't recursively call itself. It always has a C stack depth of 1, and it manages the Tcl program's stack separately (usually in the heap). This makes implementing a Tcl stdlib function way more painful though, because every time you want to evaluate a piece of code you have to return on the C stack. For example, [for] returns for the init stage, every time it runs a comparison, and every time it runs the body. This is because it has to return to the evaluator to do the next step, so the evaluator is never called by itself. It also means that each call frame has a life cycle for creation and teardown, which is just a royal pain.

But, with an NRE, we could suspend entire programs to multiplex interpreters, we could do program rewinding, we could do coroutines or effect systems (though I don't think either of those are a good fit for the time being). It opens up a lot of cool options, but it's a massive undertaking, touching pretty much every part of the interpreter.

Oh, one other detail, specific to rewinding: there needs to be some way to undo every side effect caused by the scope(s) during rewind. I'm thinking if we have an interpreter log of every side effect that happens, you could run that log in reverse when rewinding, up to the point of the Expect. This would avoid any side effects from being missed. The cool thing about this log is when running in reverse it might even go back up stack frames, so your Expect could be five function calls deep, and the log would just run all the way through the stack back to the Expect.

Open Sound Control API

Paul: I am in the process of building out a Folk music toolkit, and one of the core components needed for this system is the ability to send OSC messages from my Folk machine to arbitrary OSC servers to control synthesizers. I managed to implement simple OSC send functionality which allows Folk users to send OSC messages using the native Folk `Notify:` syntax. A Folk environment first registers the address of the OSC server using `Claim the NetAddr is <ip>:<port>` then once an endpoint is registered Folk programs can send OSC messages using the syntax `Notify: OSC at /path/ with args /values/`.

Next I will add support for OSC bundles, which will allow us to write proper tidalcycles-esque pattern languages native in Folk!

Outreach

Interface Studies podcast

TODO: Andrés or Daniel

Recurse Center and SVA visitors

Tracey Le has been bringing groups of people from the current Recurse Center batches.

Open house

Had a pretty big open house (we've been having big open houses for the last few months). Tracey brought a bunch of people from Recurse Center; Alan Laidlaw stopped back in after a while; Paul came up from Philadelphia.