If you'd like to see the latest updates on Folk in person, our next Folk open house is all afternoon (1pm-7pm) on Friday, September 26, in East Williamsburg, Brooklyn. You're welcome to come just to cowork, or use the afternoon to hack on your own project on our Folk system.

Omar: The big news from July was that (after like 18 months of work) folk2 became stable enough that we could make a pull request for it (which is also a good overview of all the changes).

In August, the pull request has been under review: Rob Fielding was testing it a bit, and Mason Jones also got it to work and is now going in and reviewing code and submitting patches:

(Feel free to test the osnr/folk2 branch yourself if you have a Folk system! It should be mostly pretty usable now.)

Mason found and fixed some serious race condition and safety bugs in the folk2 runtime. After those fixes, the runtime actually feels pretty solid now. There's 1 minor memory leak (1MB/sec?) that I want to hunt down, but overall, it feels like the folk2 runtime is now generally doing what it was intended to do in a predictable way; there aren't chaotic bugs where pages duplicate or get stuck or blink out permanently like there have been in the past.

So a lot of the things we did this month were semantic improvements, where it's not that we had a bug in implementation, it's just that the runtime design leads to some undesirable effect or can't express something we want, so we have to add new options to When/Wish/Claim, or add delays or waits or timeouts, or add a new concept (destructors) to manage the resources properly.

As of Aug 30, folk2 is now the default branch on the main system we have at Hex House, which feels like a major step (and it'll force us to really seriously use it).

Been trying bringing back the per-thread/per-interpreter value cache that we disabled a few months ago.

The idea here is that you have a LRU cache on each thread (Tcl interpreter) that maps string → Jim object. Then every time you take in a statement on that thread (when it runs a When block), you look up each term in the statement in the thread-local cache, so if you already have a shimmered Jim object that's a parsed list or code expression or something, you can reuse it and save yourself the effort of parsing the term string. To avoid leaking RAM forever, you discard any cache entries that haven't been used recently – cache capacity is arbitrarily set to like a few hundred. (it's always safe to discard cache entries, since you can always reparse from the string term in the statement)

(My theory is that we are spending most of our time in folk2 reparsing stuff; every individual When block evaluation seems to take at least 50-100 microseconds, sometimes up to 500 microseconds, which feels like a lot.)

(I got it to work this time by making it a more proper LRU cache with a linked list for eviction, as opposed to the hacky double-table scheme I had last time.)

The cache hasn't helped as much as I expected, although it does help. Mason reports that calibration cycle time improves from like 200ms to 160ms.

The thread-local object cache does seem to successfully prevent re-parsing of code on every eval:

I'm not sure why it's not helping more. (I have the same question about Mason's shared-objects branch, actually.) We might need to infect more of the codebase to be cache-aware (to look in the cache instead of taking straight from the statement, or to put things into the cache when they're also going into a statement) or to prevent shimmering / maintain objects in the most optimized format.

Mason has been trying to revive his branch to share Jim Tcl objects.

Really good bug find where we only had one global Jim_ObjType variable per struct for all of Folk, and it would churn as you reloaded the module in new interpreters, so you'd get unnecessary shimmering.

Since the spring, we've had this concept in folk2 of 'writable textures', or 'writable images', or (renamed this month, and I think this is now the final name, by analogy with HTML5) canvases. Canvases are GPU textures that you can use as a draw target for a shader, and that you can draw onto the screen (or onto other canvases) (just like any other GPU texture in Folk).

Canvases exist so that we can accurately transform page-local drawings as you lift up and skew the page, and so that we don't have to redraw page-local content when the display list doesn't change (we just use the existing texture contents). The explicit draw targets in page-local coordinates (as opposed to drawing everything onto the screen) also cut down on statement traffic, since they don't change even when the page moves, and they allow per-canvas display list settlement to work, reducing blinking/glitch states.

One problem, though, is that drawing to canvases has been a parallel process to drawing on the screen, which means that you need a way to synchronize so you don't draw a canvas to the screen while the canvas is itself being drawn onto. Until this month, we were double-buffering canvases, but this was really annoying to coordinate (either you have independent locking which can mess up the scheduler and/or renderer and/or cause deadlock, or you coordinate through a statement [a claim about what the active/inactive buffer are] which introduces blink risk).

I simplified everything (canvas render & screen render) to just run in a single loop on the GPU thread and render to canvases before then rendering to the screen. Then we can just have 1 buffer (GPU texture) per canvas, since textures are guaranteed to never be drawn to the screen in an inconsistent state. We have a decent amount of headroom on the GPU thread, so I'm hoping this is good enough for now.

I was trying a pretty basic program on folk2:

When the clock time is /t/ {
  Wish $this is labelled $t
}

If you're familiar with folk2, you might anticipate blinking problems here, because there's a lacuna where the old clock time goes away, the new clock time comes in, but this When block hasn't been evaluated on the new clock time and so the new label statement hasn't appeared. The display subsystem might sample the DB during that lacuna, leading to a visible blink-out of the time:

You could have a bridge period here, where you Wish (keep 10ms) $this is labelled $t, or use a Hold! instead, but then you have a different problem: you might see 2 clock times displayed, because the display subsystem happens to sample the DB in a period where both old and new label statements are in place.

But I don't want to resort to system-wide convergence where you don't see anything until everything in the universe has converged, so I'm trying this concept of per-canvas settle time:

Basically, the Collect statement above, which figures out the display list to render onto each canvas every frame, won't re-fire/re-Collect until the display list has been stable for 3ms. This prevents the user from seeing those transient/glitch states with 0 or 2 label statements.

TODO: Sign error

Camera slices were blinking out to magenta because their GPU texture wasn't pinned all the way until rendered to the screen.

The solution was to have the Collect statement which collects the display list also inherit all the destructors from the individual display statements.

Here is the image statement that has a destructor attached:

• Subprocess hang fix
• Mason base64 fix
• Multi-keyboard fix
• Made rendered canvases see-through

folk2 has still been missing a lot of features/virtual-programs that are standard in folk1, so I spent some time this month porting things I missed:

• Error reporting, titles
• Line drawing
• Points-at
• Music
• Animation demo

Here's the current state-of-the-art animation demo, ported to folk2:

set COLS 3
Wish tag $this is stabilized
Wish $this-display has a canvas
When $this has resolved geometry /geom/ &\
     the animation toy's frame count is /N_FRAMES/ {

    set frameGeom [list width [* $geom(width) 0.3] \
                       height [* $geom(height) 0.3]]
    Claim $this-display has resolved geometry $frameGeom

    for {set i 1} {$i <= $N_FRAMES} {incr i} {
        Wish $this-frame-$i has a canvas
        Wish $this-frame-$i is outlined red
        Claim $this-frame-$i has resolved geometry $frameGeom
    }
}

When the quad library is /quadLib/ & $this has quad /q/ {
    set displayQuad [$quadLib move $q down 270%]
    # set displayQuad [$quadLib scale $q 30%]
    Claim $this-display has quad $displayQuad
}
When the quad library is /quadLib/ &\
     $this has quad /q/ &\
     the animation toy's fps is /FPS/ &\
     the animation toy's frame count is /N_FRAMES/ {

    Wish $this is labelled "FPS: $FPS"
    Wish $this is labelled "Frame count: $N_FRAMES"
    for {set i 1} {$i <= $N_FRAMES} {incr i} {
        set col [expr {($i - 1) % $COLS}]
        set row [expr {($i - 1) / $COLS}]

        set frameQuad [$quadLib move $q right 100%]
        set frameQuad [$quadLib scale $frameQuad 30%]
        set frameQuad [$quadLib move $frameQuad left 120% up 120%]
        set frameQuad [$quadLib move $frameQuad right ${col}00% down ${row}00%]
        Claim $this-frame-$i has quad $frameQuad

        When the clock time is /t/ & $this-frame-$i has camera slice /slice/ {
            if {round($t * $FPS) % $N_FRAMES == ($i - 1)} {
                Hold! -keep 6ms {
                    Wish $this-frame-$i is outlined green
                    Wish $this-display displays image $slice
                }
            }
        }
    }
}

It totally works:

But the code is a little… ugly, because of the paperwork you need to go through to make synthetic objects (the display panel and the red drawing frames) in folk2 right now:

to make a synthetic object you need to establish a bunch of stuff – GPU canvas, resolved geometry, surface-to-clipspace projection, quad – which is quite annoying
and some of it is permanent and some of it likely to be transient (the quad, mostly) and that split is easy to mess up
you also have the option to draw directly to the screen [and sidestep creating a canvas] but then you probably have to downproject from meters to pixels which is also sort of annoying

Since folk2 already breaks a lot of stuff, I've wanted to bundle in some breaking changes that we've been meaning to do for months/years anyway. Did some of those this month:

Hold! now has a reasonably nice unified syntax, where both of these work (and there are more options you can use, like -on, -non-capturing, -keep):

Hold! -key wow \
  Claim wow this works

Hold! -key wow {
  Claim this works
  Claim I can make 2 statements too
}

so we don't need the ugly HoldStatement! wow [list $this claims wow this works]

Finally made this change in folk2 this month:

(https://github.com/FolkComputer/folk/commit/db156c60901f3af533ccbf8da616db6256e02867)

Omar: I've been updating the documentation for gadget2 as Daniel Pipkin has worked on putting one together (he's needed clarification on what parts I've been using, how to set things up, etc).

Mason has been working on a proposal for a type system in Folk, for things like pixels, percentages, millimeters, stuff like that.

• Rob Fielding created a full-fledged system for presenting slides that are fully controlled by programs in Folk:

• Omar hosted his friend Guille Angeris's colleague grjte, who was in New York for a bit and has been interested in setting up a Folk instance:
  • 
• Andrés went to Taiwan and visited Ultimems, the firm that develops the ultra-small laser projectors (the Ultimems Up projector). They were excited to hear about what we've been doing with the gadget. We had a great two-and-a-half hour talk walking through the use cases for Folk and our need (and gratefulness) for their small laser projectors for our work. We're excited to continue having a friendly relationship with them and maybe find space for a partnership some day.
  • 
• Andrés went to the artist collective Akiya Collective in Komoro, Nagano Prefecture, Japan and brought the gadget. Over the next few months, as Akiya grows, Andrés will be guiding them on making their own Folk installation and offering it as one way people can make art or work on science projects in their space
  • 
    • The Akiya collective has a workspace where people work on arts, crafts, painting, and electronics projects of all sorts. Bringing the gadget there, it was fun to imagine various ways that Folk could be incorporated to creative workflows in the makerspace.

• The Fractal community here in Brooklyn had a “barn-raising” event to set up their Folk system. They had some trouble calibrating with their ultra-short-throw projector, but there seems to be a lot of excitement there
• Mike Subelsky set up a Folk system in Baltimore
• Hamish Todd has been continuing to work on his Cambridge, UK installation and was asking questions about geometry and C API in Discord

• Our next Folk open house is all afternoon (1pm - 7pm) Friday, September 26, in East Williamsburg, Brooklyn.
• Omar: catch up remaining functionality, test with programs on the Hex House Folk system, get more reviews in, merge folk2?
• Omar: improve gadget calibration, maybe start making more
• Andrés: Experiment with dots as an interface again
• Andrés: Merge Mason's big editor improvements
• Andrés: Finalize video implementation

• https://groups.io/g/loopantennas/topic/nasa_split_lock_washers_are/95936117
• https://old.reddit.com/r/3Dprinting/comments/1g96lf9/loctite_damaging_petg/
• https://stackoverflow.com/questions/26145267/how-do-i-force-a-target-to-be-rebuilt-if-a-variable-is-set
• https://tclementdev.com/posts/what_went_wrong_with_the_libdispatch.html
• https://beej.us/guide/bgc/html/split/chapter-atomics.html
• https://erich.realtimerendering.com/ptinpoly/
  • https://wrfranklin.org/Research/Short_Notes/pnpoly.html
• https://suberic.net/~dmm/projects/mystical/README.html

• Graph Theory in State-Space — a beautiful exploration of state space visualizations for a deceptively simple shape puzzle game
• A 90° corner stapler
• A clever under-monitor mounting system using 3D printed brackets and a few hardware store dowels
• Serving 200 million requests per day with a cgi-bin
• How Does Reverse Image Search Work?
• The most insane sound design tool nobody's heard of (is free)

• Rust Atomics and Locks, Chapter 3—this finally made atomics click for me. Understanding memory ordering finally made writing atomic code possible!
• ThreadSanitizer: this has been a massive help to debug race conditions (along with AddressSanitizer and LeakSanitizer for other issues)
• Hex to ASCII: helped me figure out that my pointer was in fact the contents of a string
• Annotating atomic fences: helped with ThreadSanitizer false positives in the workqueue
• Two handy GDB breakpoint tricks: great comments on working with GDB

• Hamish: So here's a rather lovely thing: https://www.youtube.com/watch?v=ImKcqjqJNzw — Definitely feel like the gadget might fit well with this, particularly the pose where you lie on your back and the projection comes from your chest. The website sells these books for $15 each and in our case the shadow puppets would have April tags