Folk Computer

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== March 2025 newsletter ======

(Our next Folk open house is **[[https://partiful.com/e/khO9CQ55Phn6FEsdK8dd|in the evening on Sunday, April 27]], at our studio in East Williamsburg, Brooklyn.** Come by!)

===== What we've been up to =====

==== Demos and applications ====

  * (Andrés) Animation binder
    * I took the animation programs that [[https://cristobal.arquipelago.org|Cristóbal Sciutto]] made and put them in a binder, added a binder clip for stacks of paper, and added a wheel for people to adjust the number of frames
      * {{newsletters:img_1070.gif}}
      * {{newsletters:folk-anim-tinnei.gif}}
  * (Andrés) Frog folder
    * I've been trying to explore making more didactic Folk programs; the first of these is the **origami frog folder**, which uses a mix of shapes and text to guide you through making an origami frog. If you have ideas for other things Folk might be useful for teaching, feel free to email me at **[[mailto:cwervo@folk.computer|cwervo@folk.computer]]**
      * {{newsletters:folk_origami_frog_march_14_2025-1-.gif}} {{newsletters:frog-binder-cover.jpeg?200px}}

==== General system improvements ====

  * (Andrés) Video progress
    * All the frames load and play at the correct size now — some bugs to squash:
      * video frames not clearing
        * {{newsletters:video-bunny-large.jpeg?300px}}
      * sound playback (this will be another PR, probably involving an overhaul to the overall sound engine in Folk)
      * stuttering playback after many iterations (possible sign of a memory leak?)
        * {{newsletters:video-loop.gif}}
      * Omar: I've been really excited about video -- I think making a physical video book or diorama is a really simple and really compelling application of Folk (that you can't do without a system like Folk), and it's not even that computational or programming-y
  * (Andrés) [[https://github.com/FolkComputer/folk/pull/201|Shapes refactor]] — I fixed ''shapes.folk'' so that all shapes are drawn relative to ''$this'':
    * {{newsletters:folk-shapes-tuneup.gif?0x200}} {{newsletters:folk-shapes-tuneup-text-large.jpeg?0x200}}
  * (Andrés) Dot detector — not much progress this month, will get to it early April
  * ([[https://github.com/smj-edison|Mason Jones]]) [[https://github.com/FolkComputer/folk/pull/198|Basic program grouping]] — a program that runs a list of programs, even if they're not visible (it helps with saving table space as programs accumulate, I like stacking the programs below the group!)

==== Folk gadget & QR scanner ====

Omar: as a side project for [[https://qrshow.nyc|QR Show]] ([[https://qrshow.nyc/retrospective.html|see retrospective]]), I wrote a [[https://github.com/osnr/folk-qr|QR scanner program in Folk]] for the handheld gadget.

> [[https://bsky.app/profile/omar.website/post/3lkm3yv4ups2c|QR Show was my excuse]] to (*extremely* last-minute) hack together a prototype that I've been thinking about for years…
>
> point at the QR code, press trigger button to scan, then get output next to it; here, the QR code just says "Hello world!"
>
> {{newsletters:img_8396.mp4}}
>
> It kind of half-seriously answers the critique of, like, restaurant menu QR codes, where scanning a QR code locks you into your phone & isn't a process that people around you can participate in. Here, you scan the QR code in a 'social' way, and 'in situ', and other people can see/participate too

This is an interaction I'd had in mind for a few years, since I first got the Nebra AnyBeam in like 2021, so it's great to get to play it out now.

And it was [[https://bsky.app/profile/omar.website/post/3lkm3yv4ups2c|not hard to write]] -- came up with the idea and implemented it in like 100 lines of code in a couple of hours before showing up. (This is also a consequence of all the work we've put into making stable gadget hardware over the last few months; the gadget is a pretty good hardware platform for this in its current state as is!)

Here, I'm scanning a URL that Owen put up for his project (https://owentrueblood.com/experiments/autonomous_response/), and that web page gets rendered (using ''exec wkhtmltoimage'') and projected by Folk on the right:

{{youtube>2yj2y554O5g?}}

Since the show, I've been making small improvements. It now loads & renders the web page in the background instead of blocking on wkhtmltoimage, it lets you clear by holding the trigger button, it decodes continuously while you hold the button so you get more chances to capture the QR code...

Some future aspirations here:

  * I really want a 'lasso' interaction where the marching-ant-selection-rectangle actually lassos the QR code when you press the trigger button
    * like how that operation would go on a computer screen
    * this requires good 3D calibration of the camera and projector (the stereo camera may help a lot here; we may even /need/ the stereo camera to be able to do this without prior metrics of how big the QR code is)
    * (I want this to generalize so you can ultimately select/copy ghosts/drag around arbitrary objects and drawings/text that you point the gadget at)
  * Make the web page rendering faster and/or better status display
  * More and cooler QR scan behaviors beyond web page rendering:
    * Wi-Fi network QR -> connect to that network (this would be genuinely useful)
    * Folk program QR (either literal code in the QR or a URL with a Folk program) -> hold it in the evaluator, run it

The QR scanner is very different from most of the other stuff we've done, and it's a great test for Folk as a system:

  * fun to show off to people in the wild outside the lab (I can just take it to a meetup or party or whatever)
    * would ideally be usable and 'computer-y' without a laptop or keyboard or anything like that
    * the feedback cycle feels a lot faster, where I don't have to wait for the next open house to show off a random change I make to QR stuff to new people
  * ties into existing/familiar/relatable computational systems (the Web, QR codes) instead of being a de novo universe
    * doesn't require people to know or care about Folk itself at all; it's something even I wanted to do anyway
  * non-tag-bound input (QR computer vision, trigger button) and output (project on whole left/right half of display)
  * slow processes (synchronous-ish QR detect, asynchronous-ish rendering Web page)
  * uses the network, have to get the gadget on local Wi-Fi or whatever

=== Other gadget stuff ===

== Wi-Fi issues ==

I mentioned this last month -- I've had such trouble getting the gadget (with its Orange Pi 5 and USB Wi-Fi dongle) connected in wild environments (where it doesn't know the Wi-Fi upfront). The self-access point mode on the dongle just doesn't seem to work.

== Successful calibration ==

Following up on [[newsletters/2025-02#calibrating-the-new-gadget|last month]], I did successfully calibrate this (new, ELP dual-camera) gadget. 

(it hasn't been a priority, but I figured I might as well since I've been bringing it out so much for QR, and everything was in place already to try it)

wasn't too bad! the calibration is a bit off -- a couple of centimeters -- when trying to use it on top of folk0, but I hope some of the physical improvements below will help.

=== Upcoming ===

There's still a lot to do on the gadget. Bringing it out so often for QR demo has helped clarify what the outstanding issues are, at least:

  * Very annoying to get on Wi-Fi -- need to preprogram it at home to have the right SSID and password in netplan file, or (once on site) take off back panel and plug in Ethernet-to-USB-C to laptop and do it there
    * ideally, it would just host its own ad-hoc or access point Wi-Fi, but I literally can't get this to work with the dongle I use
      * or BLE or Bluetooth PAN or something to get equivalent shell access to the Pi? but that's novel territory
    * a cool hack (that QR scanning just opened up) would be to scan Wi-Fi QR code to connect
    * display Wi-Fi status like how we display battery status? (can we do both of these in a nicer/more ambient way?)
  * Back panel is getting looser over time and one bolt never really stuck; probably should use heat inserts to screw it in. it comes on and off a lot, too...
    * have a good way to expose a USB port or two on the back panel too? they're just generally handy
  * Projector swivels around still (which could be very bad, breaks calibration).
    * Maybe use elastics and slits in the top?
  * Front panel cuts off part of the projection a lot of the time (120px or so too short vertically)
    * Related to projector swivel a bit also, projector droops or tilts and gets cut off, but I think projector aperture is genuinely a bit too small also
  * Button is fraying over time as it gets bent outward. Needs a cover that you press on top, like in the original grip design (I made a basic casing for our button but not a cover)
  * microSD card sometimes totally fails either before boot or while the system is running, need to power cycle
    * use M.2 SSD instead? I ordered a couple


==== folk2 ====

Omar: Still a lot of work left on the new interpreter -- I've been biting off more and more as I continue to encounter blinking problems and dropped frames -- but I'm also increasingly excited about the potential frontier of performance (making Folk as responsive as possible), and I think the new system will be semantically better and cleaner, too (and more flexible for people to do new things). A lot of the changes are things we've wanted to do for years.

A few different branches are in play right now:

=== C collect ===

I noticed in the Tracy profiles that we were spending a lot of time doing recollections, which is implemented in this both-elegant-and-hacky pure-Folk way in folk2:

<code>
When when the collected matches for /clause/ are /matchesVar/ /body/ with environment /e/ {
    Wish to collect matches for $clause
}
When /someone/ wishes to collect matches for /clause/ {
    When /someone/ wishes to recollect matches for $clause with timestamp /ts/ {
        Say the collected matches for $clause are [Query! {*}$clause]
    }

    # the Hold! should hopefully dedupe
    set Recollect! [list apply {{clause} {
        HoldStatement! [list recollect $clause] \
            [list virtual-programs/collect.folk wishes to recollect matches for $clause with timestamp [clock milliseconds]]
    }} $clause]

    When {*}$clause {
        {*}${Recollect!}
        On unmatch ${Recollect!}
    }
    {*}${Recollect!}
    On unmatch {
        HoldStatement! [list recollect $clause] {}
    }
}
</code>

Notice how each recollection has to do multiple passes through the workqueue and Tcl and the database (new statement for the collection triggers the When, which triggers the HoldStatement!, which triggers the Wish, which triggers the Query!, which finally gets dispatched to collectors by doing the Say).

(Even in folk1, Collect is implemented natively in C, so it's not surprising that this would be a step backward in performance.)

So I made a new folk2 branch where Collect is implemented in C, which should cut almost all of that database traffic.

It feels a lot more responsive compared to main folk2. The major bug (and why we haven't merged it into main folk2) is that it seems to introduce long periods where pages just blink out and vanish, like this:

{{:newsletters:pasted:20250331-105912.png?400px}}

Still need to debug this.

=== Page quads ===

This is the most important branch and probably the biggest potential performance improvement / hold-up to merging folk2. I've mostly been working on this for the last couple of weeks. 

The goals here are to 1. render page graphics to textures in memory and composite those instead of drawing directly to screen buffer and 2. always represent draw coordinates in 'meters with respect to the page', all the way until the draw op hits the GPU. That means that almost all draw statements will be stable across frames, since the outline of a page doesn't change in that coordinate frame. That means we can hopefully get rid of like 80% of database traffic every frame, because we don't have to recompute every draw statement when a page wiggles slightly.

So this is both a performance improvement and a breaking user-facing API change (one we've badly wanted to make since we introduced 3D calibration late last year).

(//and// it's a correctness improvement -- it means we should be able to draw text non-distorted in the real world, not relying on the projector pixel grid as we do now.)

Broke up the Vulkan gpu.folk more cleanly into gpu.folk, gpu/images.folk, and a new gpu/writable-images.folk. (Had to improve C import/extend technology so we could access the gpu C code from the other modules -- you can even use global variables from another module now, at least by a ''_ptr()'' getter I added. Had to figure out the right Vulkan render pass and image format stuff so we can still have single pipelines that we reuse across both image-target and screen-target draw operations.)

Made more of the operations run concurrently (and in arbitrary threads) in normal Whens instead of one big GPU loop, which should make the system more robust (and mean that stuff like new-shader compilation doesn't freeze the whole system) (and let others extend it further in Folk modules and get access to state that's just stored in statements instead of loop-local variables), but does mean we need more locking and fences and semaphores and stuff to synchronize the GPU actions.

It's cool to think about how much draw logic can be implemented using idiomatic Folk constructs, and it's a good test of how expressive we can make Folk.

Lots of weird images like this to test drawing in an image and then drawing that image onto the screen:

{{:newsletters:pasted:20250401-021250.png?350px}}

(here, was also encountering a bug with line drawing that turned out to be that I forgot about triangle facing / vertex order.)

=== Laptop universe and safety concerns ===

I've been testing a lot on my laptop, which has produced some side quests:

  * Brought back [[https://github.com/FolkComputer/folk/blob/4c96d35cf1e6fa35aaac5bf37cebb3126ef78331/virtual-programs/fswatch.folk|hot swap]] so you can live-edit even pretty deep virtual programs on laptop (needed this to debug GPU texture stuff more easily, turn off draw operations, mess around with coordinate transforms until it was correct)
    * (isn't it cool that this can just be a blocking infinite loop now, because folk2 supports that? no need to think about the event loop or use ''readable'' or anything)
  * New version of ''exit()'' -- I'm calling it ''__conclude'' -- so we can write tests that terminate cleanly, so I can test the whole writable-image flow more easily.
  * New finalizer concept so we can release GPU resources in statements safely?
    * {{newsletters:imag11e.png?300px}}

These all have a similar energy: they both are part of a whole constellation of safety concerns I have about Folk programs. Basically, a Folk program needs to be cancelable, it needs to be hot-swappable (that is, you retract it, clean up all its stuff, then assert a new version of it that presumably wants to do a lot of the same things). That's a pretty high bar! [[https://social.omar.website/@omar/statuses/01JQM63EE1NKSZM5R7W8SCW4G1|Unix can do it with processes]], but we want to share more state than Unix processes do. How do you release all the resources cleanly, do it in the right order, etc?

Another class of issue is queueing issues, where a thread (usually the sysmon thread) freezes and the rest of the system gets overrun by clock events and/or garbage and crashes:

{{:newsletters:pasted:20250401-021357.png?450px}}


=== Inter-thread shared objects ===

(I still don't know if we'll merge this, but) I've been noodling some more on the concept of inter-thread shared objects (see [[newsletters/2025-02#shared-immutable-immortal-objects|last month]]), so we don't have to copy and reparse (or cache-lookup) term objects every time we use a statement in a different thread.

I've been trying an approach from last month (which does feel actually-implementable) where we only /temporarily/ shimmer objects if they're inter-thread-shared, which only requires some pretty mechanical changes to all the shimmering functions in the Jim Tcl interpreter.

{{newsletters:c66f2b55-c5e0-4d2d-8560-12ec42f861bc-22511-0000170b2cd6a9d0.png?400px}}

But there are so many trade-offs (now shimmering is just temporary for shared objects, now we have to track lots more stuff, etc) that I don't know if it's worth it compared to the current folk2 approach of just caching everything. Hmmmm.

== Mason's dive into not shimmering shared objects ==

Mason: I've been tinkering with preventing shared objects from shimmering, so that they can be safely shared across threads. It's been fairly straightforward, though there's some edge-cases I'm still trying to figure out (programs and variable names still need to shimmer so they can do fast execution and lookup, how to deal with program transfer?). I also haven't tested the code yet, so maybe it'll all blow up. ¯\_(ツ)_/¯

==== Outreach and other systems ====

  * Mason updated the [[https://folk.computer/supercollider|SuperCollider instructions]] to make booting up more consistent:
    * {{newsletters:screenshot-2025-03-31-at-5.17.34 pm.png?400px}}
  * Mason and [[https://x.com/_rpf|Rob Fielding]] discussed wanting scrolling in the editor, how to edit printed programs, and physical debugging tools.
  * Paul from [[https://www.synergymill.com|SynergyMill]]: "pics from our meetup last week, had someone at the meetup talk about a 'chemistry computer' which would allow users to manipulate cards that represented atoms and the computer would simulate their interactions, so we messed around with a (very) rough prototype"
    * {{newsletters:march-5-meetup.jpg?0x250px}} {{newsletters:folk-chemistry.gif}}
  * Mason tested out the portability of ''folk-peridot'':
    * {{newsletters:pxl_20250322_040211551.jpg?0x250}}{{newsletters:pxl_20250321_203203331.jpg?0x250}}
  * Rob got an AAXA 4K1 projector and used it to fashion his own Folk lamp:
    * {{newsletters:img_9691.jpg?0x250}}{{newsletters:img_9685.jpg?0x250}}{{newsletters:iggmg.jpg?0x250}}
  * Rob, Naveen, Paul, and Andrés talked about how to talk about Folk (if you have another way of describing Folk to people, let us know over email or in our Discord):
    * {{newsletters:screenshot-2025-03-31-at-5.42.51 pm_discord_convo_1.png?500px}}
    * {{newsletters:screenshot-2025-03-31-at-5.43.03 pm.png?500px}}

=== Open house ===

We had our usual open house on Wednesday, March 26:

  * {{newsletters:img_8671.jpeg?0x250px}} {{newsletters:img_8673.mp4?170px}} {{newsletters:img_8688.jpeg?0x250px}} {{newsletters:img_8684.jpeg?0x250px}}
  * Playing with the new origami instruction binder:
    * {{newsletters:img_8674.jpeg?0x250px}} {{newsletters:img_8677.jpeg?0x250px}}
  * Our friends [[https://www.carollei.com|Carol Lei]] and [[https://tinnei.com|Tinnei Pang]] trying out the new animation binder:
    * {{newsletters:img_8690.jpeg?250px}} {{newsletters:img_8694.mp4?200px}}


=== Other visitors and interactions ===

  * Christina Huang visited to talk about Andrés doing a guest workshop at her Learning Analytics class at NYU
  * [[https://leomcelroy.com|Leo McElroy]] visited to chat about education, CAD, and be the first to test out the origami binder:
    * {{newsletters:folk-leo-frog-bouncing.gif}} {{newsletters:folk-leo-hearts.gif}}

===== What we'll be up to in April =====

  * **Our next Folk open house is [[https://partiful.com/e/khO9CQ55Phn6FEsdK8dd|in the evening on Sunday, April 27]], at our studio in East Williamsburg, Brooklyn.**
  * Omar: do another gadget design pass, put up gadget design on GitHub
  * Omar: finish render-to-texture and make folk2 pull request
  * Maybe start working on dual-camera and self-calibration/refinement tech?
    * Omar: I'm bothered by the baseline-mediocre state of calibration and think we need some new approaches to keep the system calibrated over time -- maybe self-calibration by periodically projecting a tag somewhere on the table; maybe a lightweight form of calibration that just adjusts extrinsics and is easy; maybe an iterative calibration that continuously adjusts (as opposed to the 10-pose batch status quo) so you can really tune the fringes of the projector-camera volume
  * Andrés: finish video playback
  * Andrés: get dot detector demo working in 3D coordiantes
  * Andrés: Kosmik integration demo
  * Andrés: guest lecture at Christina Huang's Learning Analytics class

===== Links we've enjoyed =====

==== Omar ====

  * https://blogs.loc.gov/maps/2025/03/globe-gores/
  * https://thereader.mitpress.mit.edu/emily-dickinsons-playful-letterlocking/
  * https://www.youtube.com/shorts/puxHXjszzbE
  * https://mastodon.thi.ng/@toxi/114183742485579955
  * https://sioyek.info/
  * https://mikeestee.com/projects/illuminated-artifacts/project.html
  * https://poetical.day
  * https://boofcv.org/index.php?title=Performance:QrCode
  * https://www.gofundme.com/f/rescue-and-restore-the-aesthedes-1

==== Andrés ====
  * [[https://wattenberger.com/thoughts/boo-chatbots|"Why Chatbots Are Not the Future"]] — A really thoughtful piece about how limiting chatbot interfaces are
  * [[https://wattenberger.com/thoughts/our-interfaces-have-lost-their-senses|"Our interfaces have lost their senses"]] — A sequel-of-sorts to the above piece that feel like a very Folk-endorsed idea: computers used to be big and tangible, and as they've gotten smaller, and more convenient, we've also lost all the texture and ability that comes with large, room-scale physical interfaces.