Folk Computer

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== October 2023 newsletter ======

===== What we've been up to =====


==== Applications and demos ====

  * Andrés: [[guides/button|New guide on how to make a paper button:]] [[guides/button|{{guides:3_6_yayyyyy.gif?300}}]]
    * Really want to do more of these very specific technical/physical guides -- that's kind of what's nice about this newsletter, it can be very concrete and not explicitly philosophical but still communicate the underlying philosophical points

  * [[https://cristobal.space|Cristóbal Sciutto]] has been working on a series of iterations of the concept of [[notes:tableshots|'tableshots']] ([[https://twitter.com/tobyshooters/status/1710395304669843548|tweet]])
    * {{:notes:cristobal_tableshot.gif?0x200|}} {{newsletters:img_8594.jpg?0x200}}
    * {{newsletters:img_8617.mp4}}
      * (Omar: You can start to see the real motivation for this work on tableshots: building a hybrid digital-physical 'reading desk' where you can take 'screenshots' of passages of real books. It's a project I've been excited about for years, so I'm glad it's starting to happen.)
      * {{:newsletters:pasted:20231101-040656.png?400}}

  * [[http://twitter.com/naveenma|Naveen Michaud-Agrawal]] made [[https://discord.com/channels/956758212152025098/956758650700046366/1160573888451854366|dials]] on cardboard tokens:
    * {{newsletters:pxl_20231008_1344382492.mp4}}
    * The physicality of the dial -- that it's this round cardboard token -- is surprisingly important. it would be so much worse for it to be an 8.5x11 sheet of paper, it would be much less of an 'object', you couldn't palm it and hold it by the edge, stuff like that

  * Naveen also made an observability demo where you can point at a performance metric and see a live plot of that metric:
    * {{newsletters:pxl_20231015_1434173842.mp4?400}} 
    * (also check out how his system uses [[https://gist.github.com/nmichaud/2a43c15a7c1810170c88e4ff37625239|dual tags]] and a different tag family -- it's always a nice example of the flexibility of Folk)

  * Naveen also made warping/portals:
    * {{:newsletters:pasted:20231101-224529.mp4?400}}

  * Andrés: [[https://x.com/cwervo/status/1716172470858760462|Pumpkins.]] Happy Halloween!
    * {{newsletters:img_2392.jpg?200}}

  * Andrés made an [[https://discord.com/channels/956758212152025098/956758650700046366/1159570264670085250|embroidered AprilTag]]:
    * {{newsletters:apriltag-embroidery.gif?0x250}} {{newsletters:img_1765_large.jpeg?0x250}}

==== Outreach and new systems ====

  * [[https://twitter.com/djmicrobeads|djm]] & [[https://hellowelcomeback.online|Tinnei Pang]] set up [[https://twitter.com/localmaxim/status/1714043265463304375|their system in London]] using a Beelink SER5 and tension rod:
    * {{newsletters:whatsapp-image-2023-10-16-at-5.30.32-pm.jpeg?200}}

  * [[https://wwj718.github.io|Wenjie Wu]] set up a new system in Sanya, China!
    * {{:newsletters:pasted:folk-wwj.jpg?300}}

  * A group of visitors from the current [[https://www.recurse.com|Recurse Center]] batch came by, and we ended up doing a good amount of programming with them:
    * {{:newsletters:pasted:20231031-094733.jpeg?0x300}} {{:newsletters:pasted:20231031-094744.jpeg?0x300}} {{:newsletters:pasted:20231031-094821.png?0x300}} {{newsletters:img_2544.mp4?200x300}}
    * [[https://owentrueblood.com|Owen Trueblood]] made a shader:
      * {{newsletters:img_2542.mp4?200x100}}

  * We had a small open house on October 29 -- it was fun to bring new people in and show off the system
    * {{:newsletters:pasted:20231031-082535.jpeg?300}} {{:newsletters:pasted:20231031-082547.jpeg?300}} {{:newsletters:pasted:20231031-082458.jpeg?300}} {{:newsletters:pasted:20231031-082446.jpeg?300}}


==== CNC and calibration ====

  * Omar and Andrés and friends finished building the CNC! (special shout-out to [[https://www.rileynwong.com|Riley Wong]] for their help)
      * Built a heavy tripod Folk system to cover it, including a bright 1080p projector
        * {{newsletters:img_1928.jpg?250}} {{newsletters:img_1925.jpg?250}} {{newsletters:img_1927.jpg?250}}
        * The tripod is a lot more practical here than usual -- we have a high ceiling in the workshop, so don't want to use a tension rod, and we can spare the floor space, and we want to mount a heavy projector so we have enough brightness to show up on the material
      * We cut some stuff and immediately felt the motivation to projection-map and use Folk here. There are so many pitfalls with the normal way of doing things, from running into clamps to getting depths and sizes wrong to having to fiddle with random software. See the discussion of the project/prototype below; we're looking forward to doing more here

  * Omar has been working on 3D/real-world/high-accuracy calibration, something we've wanted for a long time (but immediately motivated by the CNC project)
    * A lot of research into various calibration schemes for cameras and projectors. I think we have a [[http://www.ok.sc.e.titech.ac.jp/res/PCS/research/procamcalib/|really promising]] strategy for standalone end-to-end projector/camera calibration now! ([[https://social.omar.website/@omar/statuses/01HDAR2AZBS1RNKZR39E8SXHKH|thread]])
      * Implemented [[https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr98-71.pdf|the classic Zhang camera calibration technique that pretty much everyone uses]] on a branch inside Folk/Tcl/using AprilTags, which is a good sign (it's the building block for every end-to-end calibration scheme I can find)
      * There's an interesting set of choices here. We want to use the parts we have (AprilTag emitter and detector, projector, camera, Web server, printer) and not bloat the system with new computer vision pipelines...
    * Talked about this in previous months -- this is exciting because full intrinsic projector/camera calibration will finally break us out of the 2D planes / page paradigm, and it'll let us have mobile systems (including phones that understand the world map), and high accuracy will break us out of coarse moving-pages-around -- will allow checking boxes, handwriting in text fields, etc
    * Here is the 'calibration rig' right now... (the calibration virtual program sets up a Web endpoint where you get a calibration board PDF to print; then you'll measure its real-world metric and trigger end-to-end calibration in Folk, which will project more AprilTags into the gaps on the board) {{:newsletters:pasted:20231031-214225.png?500}} {{:newsletters:pasted:20231031-085430.png?300}}
    * It's inspiring to look at papers on projector-camera calibration, because the //baseline// from like 2009 -- not even the state of the art -- is error of like 0.7 pixels, while Folk still has error of like an entire inch.
      * [[http://www.ok.sc.e.titech.ac.jp/res/PCS/publications/procams2009.pdf|{{:newsletters:pasted:20231101-011921.png?300}}]]

=== Owen's projection-mapped CNC 'print preview' prototype ===

We've been working with [[https://owentrueblood.com|Owen]] on this project to use Folk to help with CNC, and he put together [[https://github.com/jmpinit/projected-toolpath-preview|a prototype]] that gives an impressively accurate 'print preview' of the CNC cut just using a local homography.

Here's his mini-test on his AxiDraw pen plotter:

[[https://vimeo.com/876037161/8e5478f0e3|{{:newsletters:pasted:20231031-231914.png?500}}]]

Here's what it looks like on our CNC machine after cutting a tag; notice how accurate the projection is to the real cut:

{{newsletters:img_0872-2.mp4}}

Granted, this is a pain to calibrate each time (you have to calibrate camera, projector, and jog the CNC machine itself), and a reusable general-purpose calibration may not do as well as this (which is calibrated with a checkerboard only on the actual material), and we see some of that inaccuracy in Folk. But I think this shows that we can be very optimistic about technical accuracy, and the real open questions are about the user interface (and we can hopefully answer those questions with Folk).
==== Other system work ====

  * Omar: Spent a few hours [[https://github.com/FolkComputer/folk/compare/osnr/jim?expand=1|looking into]] what it would take to use the [[https://jim.tcl.tk/index.html/doc/www/www/index.html|Jim Tcl]] interpreter instead of standard Tcl.
    * (Jim Tcl's C API is a little different from standard Tcl, which is most of the work of adapting Folk to it.)
    * I'm by no means sold on switching, but I think it's very close to being usable to try, and it may have some fun benefits.
      * It may be //faster// for highly dynamic workloads like ours; it doesn't do bytecode compilation where you need a full lambda (which we can't provide good stability for), it just caches the parse, I think (which is probably stable for us).
      * It's only 15,000 lines of C or so, not much bigger than Folk itself -- we would probably just vendor it into the Folk repository and take it over, so it really would be our language, and we could start changing it in serious ways (if we want to keep more data in the heap value struct, if we want a multiprocess heap, etc).
      * It has some built-in niceties: better error reporting, some lexical scope support, a first-class foreign reference type.
    * This would have been a harder sell a year ago, when we were bootstrapping off a lot of Tcl features like the event loop, Tk, and threading, but we mostly are doing our own stuff now (Web + Vulkan UIs, zygote/subprocesses, synchronous mailboxes)

  * Cristóbal made a number of [[https://github.com/FolkComputer/folk/pull/108|small quality-of-life improvements]]: image scaling now works (easy to do now that we're on the GPU, just change the shader vertices); you can now do ''When $this has neighbor /n/'' without needing to explicitly wish for neighbors; he also papered over [[https://github.com/FolkComputer/folk/issues/105|a nasty collect bug]] for outlines

=== Andrés's tabletop editor ===

  * Andrés has been working on the editor. It supports:
      * Writing Folk code into a text area; as you type code is evaluated in realtime
      * ''Ctrl + r'' — reset editor text to ''Wish $this is outlined green'', a basic piece of code to outline the active program.
      * ''Ctrl + p'' — print out code to 
      * ''Ctrl + a/e'' — Move cursor to the beginning/end of the line
      * ''Ctrl + u/k'' — Delete text from cursor to beginning of line
      * Bugs to be worked out:
        * Standardize on connecting to the keyboard via: ''Claim $this is keyboard $MACaddress''
        * Drawing a green cursor-line object — debugging pic of this below
        * Document setting a keyboard via Bluetooth
  * {{newsletters:andres_editor_multi_line.jpeg?400}} {{newsletters:andres_editor_cursor_debugging.jpeg?300}}

=== Omar's proposal for reactive Variables ===

{{:newsletters:pasted:20231031-090134.png?500}}

(People really struggle with Commit and state management, and I think it'll be increasingly important as we build up more complex applications and simulations)

{{:newsletters:pasted:20231031-090233.png?500}}

  * Looked into it a little bit. It can probably be implemented in terms of ''Observe'', which I proposed a while ago, but implementing ''Observe''/''Given'' is also not that easy
    * Idea: {{:newsletters:pasted:20231031-090518.png?400}}


==== Display ====

  * Merged [[https://github.com/FolkComputer/folk/pull/93|the Vulkan branch]] and the [[https://github.com/FolkComputer/folk/pull/103|follow-up draw refactor branch]], which together basically completely reimplement Folk's display system
    * All Folk drawing is now GPU-accelerated! Drawing generally runs at 60fps now
    * All Folk drawing now happens through wishes with keyword/optional arguments -- feels really clean and extensible. **All Display:: calls have been deprecated and will visibly warn on use.** All drawing primitives now live in [[https://github.com/FolkComputer/folk/tree/main/virtual-programs/display|cleanly separated files in virtual-programs/display/]] (which also serve as examples of how to use the GPU)
      * [[https://github.com/FolkComputer/folk/blob/e21bb5eac49dfa9ad1ee5646a76fa6638253bf11/virtual-programs/display/circle.folk|Example: circle.folk implements circles]]: {{:newsletters:pasted:20231031-005316.png?400}}
        * Notice how the rest pattern ''/...options/'' is used to take the tail of the statement as the ''options'' object, and then parameters can just be pulled out of that object
        * A program can draw using ''Wish to draw a circle with radius 3 center [list 200 300] thickness 2 color green'', for instance
    * Any (virtual or paper) program can now compile and run a custom shader; the built-in draw 'primitives' aren't privileged in any way :-)
      * On the to-do list: We [[https://github.com/FolkComputer/folk/issues/106|want to make a cleaner Shadertoy-like interface]] for writing pixel shaders, where can just shade a region or object and don't have to give clip vertices and stuff
        * {{:newsletters:pasted:20231031-005723.png?500}}
    * Instructions (to install headless Vulkan on Linux / to install Folk) still need work but seem reasonably OK, except for Nvidia GPUs (which have a different Vulkan implementation)

  * [[https://arcades.agency|Arcade Wise]] added [[https://github.com/FolkComputer/folk/pull/97|support for custom fonts]] and wrote a guide, so you can [[guides/createafont|import pretty much arbitrary TrueType and OpenType fonts]] by generating an SDF atlas, and they automatically become available inside Folk

  * Naveen [[https://github.com/FolkComputer/folk/pull/98|contributed]] a drawing primitive for [[https://discord.com/channels/956758212152025098/956758650700046366/1160023903251476591|arcs]]
    * {{newsletters:pxl_20231007_0119124392.mp4}}
 
  * Andrés made a few programs for fonts:
    * {{newsletters:andres_font_big_if_true.jpeg?0x300}}
      * A tool for viewing installed fonts on the Folk system.
    * {{newsletters:andres_font_sizes_demo.gif?0x350}}
      * A tool to preview fonts at different sizes
    * {{newsletters:andres_folder_font_book.jpeg?0x300}}


=== The new shader language ===

Folk now implements a new 'GPU FFI'((There's an obvious analogy to our 'C FFI': both systems have you embed arbitrary foreign (C or GLSL) code in a Folk program and make it easy to call from Folk with native Tcl-object parameters.)) which allows you to compile 'pipelines', which consist of a vertex shader and a fragment shader, which tell the GPU how to draw something.

There aren't great helpers for this yet, like shading a region or anything like that, so you need to work in projector coordinates by hand for now. Let me know what you think would be useful to have.

Here is a very basic shader that you can just paste as a virtual program (or print). It will color a triangle underneath itself green-blue.

<code>
Wish the GPU compiles pipeline "$this's triangle" {
    {vec2 p0 vec2 p1 vec2 p2} {
        vec2 vertices[4] = vec2[4](p0, p1, p2, p0);
        return vertices[gl_VertexIndex];
    } {
        return vec4(0, 0.1, 0.1, 1);
    }
}

Wish $this is outlined blue
When $this has region /r/ {
    set r' [region move $r down 100%]
    Wish the GPU draws pipeline "$this's triangle" with arguments [list {*}[lrange [region vertices ${r'}] 0 2]]
}
</code>

Notice how you compile the pipeline once (when the program is put down), and then it gets executed repeatedly (whenever the program's region changes).

The pipeline object that you pass after ''Wish the GPU compiles pipeline NAME'' has the form ''{ALL-ARGS VERT-SHADER [FRAG-ARGS] FRAG-SHADER}''. See [[guides/gpu|the GPU guide]] for more information about these pipeline parameters.

----

There's a nice layering and consistency and comprehensibility to how graphics works now, all the way down. See the ladder:

  - Object-centric: ''Wish $this draws a circle with radius 3 ...''
  - Natural-language wish: ''Wish to draw a circle with radius 3 center {100 200} ...''
  - Vulkan pipeline wish: ''Wish the GPU draws pipeline "circle" with arguments {3 {100 200} ...}''
  - Tcl imperative: ''Gpu::draw $circlePipeline 3 {100 200} ...''
  - C FFI: ''Gpu::drawImpl $circlePipeline [encodeArgs $circlePipelineArgs]''
  - Vulkan/GPU driver: ''vkCmdDraw(...)''


=== Side quests on the way to Vulkan display ===

----

Had to rewrite calibrate to use shaders instead of plotting pixels. Made its print output slightly friendlier

----

The Folk interprocess heap has been extended to allow freeing of heap allocations (it's now based on [[http://gee.cs.oswego.edu/dl/html/malloc.html|dlmalloc]] instead of just being a bump allocator) and to tag each heap slot with a random 64-bit 'version', as a hacky way to evict stale images that happen to reuse the same heap address (so we know to recopy them to the GPU)

This implementation is pretty inefficient and unsafe (it walks a capped-256 array of all allocations and has a single machine-wide lock) and we may want to replace it with an interval tree or something at some point & introduce finer synchronization

----

C FFI improvements: struct getter functions, some errors actually abort and throw to Tcl (using longjmp)

----

''dict_getdef'' has been added

----

Subprocess code deployment has changed. You boot a subprocess now with ''Start process NAME'', like:

<code>
Start process cool {
  Wish $::thisProcess shares statements like [list /someone/ wishes /z/ is labelled /x/]
  Wish $this is labelled "Hello!"
}
</code>

You can send more code to an already-existing subprocess with ''On process NAME'':

<code>
On process cool {
  Wish $this is labelled "Wow"
}
</code>

This is used to deploy code from individual programs to the GPU process, so all GPU process lgoic doesn't have to be in one monolithic file anymore.

There are some catches / ugly bits: process names are global & you now have to explicitly opt into all sharing (either share or receive); there's no sharing by default anymore.

----

=== A note on the graphics rewrite ===

I wanted to talk a little about the motivation for this graphics project. Why did we rewrite Folk's display subsystem away from fbdev/libdrm/pixel-plotting/software-rendering and create this whole GPU-based system? (where everything needs to be a shader, and where we have to talk to Vulkan which then talks to the GPU on the Folk PC)

In a sense, it's like every other optimization we've done -- we did it the easy way at first, and then we made the system do more stuff and it became too slow, so we had to rewrite it the hard way.

(Many parts of Folk began as some 100-line Tcl prototype, then became a few hundred lines of faster Tcl or mixed Tcl/C, and now are 1000-2000 lines of C. I was talking to Cristóbal about how this workflow seems surprisingly essential: I doubt that we could have started with the C, it would have been too much to bite off at once, we probably //had// to bootstrap through the working Tcl prototype to stay motivated and to plan.)

In this case, we added images and image rotation over the summer, and re-rotating like a 2000x2000 image or even a 500x500 chunk of text every frame (as a page's angle continually varies by fractions of a degree) was just very slow, milliseconds or tens of milliseconds per frame. So we switched to the GPU, which is built to do this; all kinds of draw operations are now virtually free. Workloads that used to bring the system down to 20fps now sit solidly at 60fps.((We also had all kinds of weird issues with fbdev and libdrm: Charles's system would run slow, stuff like that. Vulkan just seems much more consistent and reliable for this type of work.))

(it's surprising how often [[https://blog.mecheye.net/2019/05/why-is-2d-graphics-is-harder-than-3d-graphics/|2D drawing stuff]] is not actually [[https://raphlinus.github.io/rust/graphics/gpu/2019/05/08/modern-2d.html|hardware-accelerated]], even today. we have it easy, because we only have like 3 draw operations anyway; we don't have to support arbitrary font paths or Bezier curves or rasterizing SVG or PostScript or whatever. so why shouldn't we just implement drawing ourselves on the GPU?)

And once the built-in drawing primitives use the GPU, we have to expose the same power -- to compile and invoke arbitrary shaders -- to the end user. I think it would be a violation of the spirit of Folk to fence it off.

([[https://haiperspace.com|Jacob]] asked me about this a while ago, whether it's really idiomatic for people to be doing complex drawing, and if it's not idiomatic, do we actually need to expose the shader language. I think on average we don't necessarily want people rendering complex scenes, but it feels spiritually important for that capability to be available. Maybe you have some craggly public transit shapefile and you want to draw it without making 5000 drawLine calls. Maybe you want your pages to have a glowing halo around them. It feels important to give users the freedom and power to do things like that, without them needing to squeeze through some limited user-facing drawing API.)


===== Hmm =====

{{:newsletters:pasted:20231024-195706.png?500}}

----

[[https://social.omar.website/@omar/statuses/01HDEYF5XMCQYRADHSAPCVMGF1|"the most important thing is containing faults and not interrupting the entire system, instead of correctness, because you can fix correctness over time but if you hard crash then you're done"]]
----

{{:newsletters:pasted:20231024-195146.png?500}}

----
===== What we'll be up to in November =====

  * We're having [[https://partiful.com/e/3RtkNpS4KTFUj2tZNXbX|our next open house on the evening of Wednesday, November 29]], at our studio in East Williamsburg, Brooklyn. Come by!

  * Tabletop editor

  * Still need to complete RFID refactor

  * Continuing work on new calibration (followed by refactoring a lot of the system to use real-world 3D coordinates, maybe followed by super-accurate camera slicing and OCR)

  * Want to do some follow-up work on Vulkan: user-facing shader language, resource disposal, increase the image limit maybe
===== Links we've enjoyed =====

==== Omar ====

  * [[https://www.dwbowen.com/telepresent-wind|tele-present wind]]
  * https://bulbapedia.bulbagarden.net/wiki/Options/Frames
  * [[https://github.com/msteveb/jimtcl/blob/d0458b9068d60d89c590594509443b824520acb9/README.taint|Taint support for Jim Tcl]]
  * https://wiki.tcl-lang.org/page/lcomp
  * [[https://glammr.us/@overholt/111239305560706783|🦪ities]]
  * [[https://agraef.github.io/pure-lang/|The Pure Programming Language]]
  * https://ccache.dev

==== Andrés ====

  * [[https://www.youtube.com/watch?v=cBYudbaqHAk&t=6704s%20|Cellpond - Spatial programming without escape (1:51:44 timecode)]] I've been thinking about this talk for hours this week. If you watch it let me know what you think: cwervo@folk.computer
  * [[https://drfreckles42.itch.io/youre-the-os|You're the OS]] — //Do you sometimes get frustrated at how slow your computer's operating system can be? Now is your chance to prove that you could do a better job!//
  * [[https://www.baeldung.com/linux/bluetooth-via-terminal|Connect to a Bluetooth Device via the Terminal]]
  * [[https://www.youtube.com/watch?v=NEaUSP4YerM| Clearly Explained]]
==== Arcade ====

  * [[https://zine-machine.glitch.me/|Zine Machine: CSS that makes your website a printable zine!]]
  * [[https://creativecommons.org/2004/04/05/woodyguthriefreeculture/|Woodie Guthrie on Free Culture]] one of my favorite folk musicians :>
  * [[https://www.robinsloan.com/notes/home-cooked-app/|An app can be a home-cooked meal]]