Table of Contents

October 2023 newsletter

What we've been up to

Applications and demos

Outreach and new systems

CNC and calibration

Owen's projection-mapped CNC 'print preview' prototype

We've been working with Owen on this project to use Folk to help with CNC, and he put together a prototype that gives an impressively accurate 'print preview' of the CNC cut just using a local homography.

Here's his mini-test on his AxiDraw pen plotter:

Here's what it looks like on our CNC machine after cutting a tag; notice how accurate the projection is to the real cut:

Granted, this is a pain to calibrate each time (you have to calibrate camera, projector, and jog the CNC machine itself), and a reusable general-purpose calibration may not do as well as this (which is calibrated with a checkerboard only on the actual material), and we see some of that inaccuracy in Folk. But I think this shows that we can be very optimistic about technical accuracy, and the real open questions are about the user interface (and we can hopefully answer those questions with Folk).

Other system work

Andrés's tabletop editor

Omar's proposal for reactive Variables

(People really struggle with Commit and state management, and I think it'll be increasingly important as we build up more complex applications and simulations)

Display

The new shader language

Folk now implements a new 'GPU FFI'1) which allows you to compile 'pipelines', which consist of a vertex shader and a fragment shader, which tell the GPU how to draw something.

There aren't great helpers for this yet, like shading a region or anything like that, so you need to work in projector coordinates by hand for now. Let me know what you think would be useful to have.

Here is a very basic shader that you can just paste as a virtual program (or print). It will color a triangle underneath itself green-blue. 

Wish the GPU compiles pipeline "$this's triangle" {
    {vec2 p0 vec2 p1 vec2 p2} {
        vec2 vertices[4] = vec2[4](p0, p1, p2, p0);
        return vertices[gl_VertexIndex];
    } {
        return vec4(0, 0.1, 0.1, 1);
    }
}

Wish $this is outlined blue
When $this has region /r/ {
    set r' [region move $r down 100%]
    Wish the GPU draws pipeline "$this's triangle" with arguments [list {*}[lrange [region vertices ${r'}] 0 2]]
}

Notice how you compile the pipeline once (when the program is put down), and then it gets executed repeatedly (whenever the program's region changes).

The pipeline object that you pass after Wish the GPU compiles pipeline NAME has the form {ALL-ARGS VERT-SHADER [FRAG-ARGS] FRAG-SHADER}. See the GPU guide for more information about these pipeline parameters.

There's a nice layering and consistency and comprehensibility to how graphics works now, all the way down. See the ladder:

  1. Object-centric: Wish $this draws a circle with radius 3 ...
  2. Natural-language wish: Wish to draw a circle with radius 3 center {100 200} ...
  3. Vulkan pipeline wish: Wish the GPU draws pipeline "circle" with arguments {3 {100 200} ...}
  4. Tcl imperative: Gpu::draw $circlePipeline 3 {100 200} ...
  5. C FFI: Gpu::drawImpl $circlePipeline [encodeArgs $circlePipelineArgs]
  6. Vulkan/GPU driver: vkCmdDraw(...)

Side quests on the way to Vulkan display

Had to rewrite calibrate to use shaders instead of plotting pixels. Made its print output slightly friendlier

The Folk interprocess heap has been extended to allow freeing of heap allocations (it's now based on dlmalloc instead of just being a bump allocator) and to tag each heap slot with a random 64-bit 'version', as a hacky way to evict stale images that happen to reuse the same heap address (so we know to recopy them to the GPU)

This implementation is pretty inefficient and unsafe (it walks a capped-256 array of all allocations and has a single machine-wide lock) and we may want to replace it with an interval tree or something at some point & introduce finer synchronization

C FFI improvements: struct getter functions, some errors actually abort and throw to Tcl (using longjmp)

dict_getdef has been added

Subprocess code deployment has changed. You boot a subprocess now with Start process NAME, like: 

Start process cool {
  Wish $::thisProcess shares statements like [list /someone/ wishes /z/ is labelled /x/]
  Wish $this is labelled "Hello!"
}

You can send more code to an already-existing subprocess with On process NAME: 

On process cool {
  Wish $this is labelled "Wow"
}

This is used to deploy code from individual programs to the GPU process, so all GPU process lgoic doesn't have to be in one monolithic file anymore.

There are some catches / ugly bits: process names are global & you now have to explicitly opt into all sharing (either share or receive); there's no sharing by default anymore.

A note on the graphics rewrite

I wanted to talk a little about the motivation for this graphics project. Why did we rewrite Folk's display subsystem away from fbdev/libdrm/pixel-plotting/software-rendering and create this whole GPU-based system? (where everything needs to be a shader, and where we have to talk to Vulkan which then talks to the GPU on the Folk PC)

In a sense, it's like every other optimization we've done – we did it the easy way at first, and then we made the system do more stuff and it became too slow, so we had to rewrite it the hard way.

(Many parts of Folk began as some 100-line Tcl prototype, then became a few hundred lines of faster Tcl or mixed Tcl/C, and now are 1000-2000 lines of C. I was talking to Cristóbal about how this workflow seems surprisingly essential: I doubt that we could have started with the C, it would have been too much to bite off at once, we probably had to bootstrap through the working Tcl prototype to stay motivated and to plan.)

In this case, we added images and image rotation over the summer, and re-rotating like a 2000×2000 image or even a 500×500 chunk of text every frame (as a page's angle continually varies by fractions of a degree) was just very slow, milliseconds or tens of milliseconds per frame. So we switched to the GPU, which is built to do this; all kinds of draw operations are now virtually free. Workloads that used to bring the system down to 20fps now sit solidly at 60fps.2)

(it's surprising how often 2D drawing stuff is not actually hardware-accelerated, even today. we have it easy, because we only have like 3 draw operations anyway; we don't have to support arbitrary font paths or Bezier curves or rasterizing SVG or PostScript or whatever. so why shouldn't we just implement drawing ourselves on the GPU?)

And once the built-in drawing primitives use the GPU, we have to expose the same power – to compile and invoke arbitrary shaders – to the end user. I think it would be a violation of the spirit of Folk to fence it off.

(Jacob asked me about this a while ago, whether it's really idiomatic for people to be doing complex drawing, and if it's not idiomatic, do we actually need to expose the shader language. I think on average we don't necessarily want people rendering complex scenes, but it feels spiritually important for that capability to be available. Maybe you have some craggly public transit shapefile and you want to draw it without making 5000 drawLine calls. Maybe you want your pages to have a glowing halo around them. It feels important to give users the freedom and power to do things like that, without them needing to squeeze through some limited user-facing drawing API.)

Hmm

"the most important thing is containing faults and not interrupting the entire system, instead of correctness, because you can fix correctness over time but if you hard crash then you're done"

What we'll be up to in November

Omar

Andrés

Arcade

1)
There's an obvious analogy to our 'C FFI': both systems have you embed arbitrary foreign (C or GLSL) code in a Folk program and make it easy to call from Folk with native Tcl-object parameters.
2)
We also had all kinds of weird issues with fbdev and libdrm: Charles's system would run slow, stuff like that. Vulkan just seems much more consistent and reliable for this type of work.