Folk Computer

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== February 2025 newsletter ======

  * Andrés’s talk from October of last year, at Causal Islands in Berlin, is now up on YouTube: {{youtube>o3kIzvCYWdE?}}
  * Omar has been promising people a [[/links|Folk links/related-work/bibliography page]] and finally started on it
  * Our next Folk open house is [[https://partiful.com/e/8zlZKFFAvyeFbngr9PUe|in the evening on Wednesday, March 26]], at our studio in East Williamsburg, Brooklyn.

===== What we've been up to =====


==== Demos and applications ====

  * [[https://x.com/_rpf|Rob Fielding]] has an [[https://youtu.be/a0Xn_YK3WzY|orbital simulation]]:
    * {{youtube>a0Xn_YK3WzY?}}
    * "My solar simulation was a way in to learn folk. The kids keep coming back to my office to play with it - hoping this increases the interest in actually doing something. A 'kit' sounds more versatile and ultimately something I want to collect a library of so they can start to build."
    * See also [[https://arcades.agency|Arcade Wise]]'s [[folkarc:orbitalkit|ORBITALKIT]]
    * (see [[#rob-s-test|below]] for more comments on folk2)
  * [[https://github.com/smj-edison|Mason Jones]] has started working on a linear algebra tutorial:
    * {{newsletters:pxl_20250213_230719945.jpg?400px}}
    * (some issues off the edge of the table:)
      * {{:newsletters:pasted:20250228-203142.jpeg?200px}}

==== General system improvements ====

  * Omar and Andrés added a tunable exposure parameter to the calibration UI
    * Exposure is usually what people need to adjust if they're having trouble calibrating -- it's been annoying to have to tell people to go back and forth to an ssh session, run ''v4l2-ctl'', manually refresh the preview -- so it's great to reduce that to just dragging the slider inline
    * Andrés also added a general autorefresh option
    * Calibration UI now looks like this (changes highlighted):
      * {{:newsletters:pasted:20250227-175958.png?450px}}
  * Mason Jones [[https://github.com/FolkComputer/folk/pull/193|added opt-in stabilization to tag tracking]]
    * You can just ''Wish tag 373 is stabilized'' and that tag will stabilize (note that you can do this from a table editor, or make a tool, or build it into particular programs)
    * "Stabilization is based on simple distance thresholding: if a tag has moved a significant amount, it is unfrozen. After a short time of no more deviance, it freezes again."
    * The keyboard editors are stabilized by default, which seems to make a //huge// difference to editing ergonomics (no more jitter while editing) and to performance (since we don't have to recompute the editor every frame if it doesn't move)
      * (stabilization originally applied to all tags, but we [[https://discord.com/channels/956758212152025098/1336937333898870784/1337449081194283018|decided]] to make it opt-in by default to not change behavior of the system radically -- keyboards seem like a clear win, since you're not moving them that often anyway)
  * Mason [[https://github.com/FolkComputer/folk/pull/199|added a 'bidirectional editor' endpoint]], mainly so you can copy/paste between your laptop and a tabletop editor -- changes to the textarea on the Web page are automatically reflected on the tabletop editor and vice versa:
    * {{:newsletters:pasted:20250227-181017.png?450px}}

==== Discussions ====

  * Mason has been writing [[notes:internals|a guide to Folk internals]] on the wiki as he develops his own understanding of how stuff works 
    * {{:notes:process-notes.jpg?300px}}
    * (at least as it is now in folk1 -- some of this is changing in folk2)

=== Snapshots / state-saving / program-saving ===

>> Is there a way to Hold something on disk so a page has permanent information?
>> Or is snapshotting the goal?
>
> we don't have this right now but i think it would be great to have
>
> 1. subsume awkward hacks where we use files in ad-hoc ways for persistence (.folk, .meta.folk, calibration)
>
> 2. crash resilience (your editor state can get preserved across crashes, etc)
>
> i think of snapshotting as a way to preserve/version programs and persistent Hold as a way to preserve data i guess, i don't know if that distinction makes sense but that's how it is in my head

(snapshotting being "where you have virtual thumbnails on the printed page to represent all the programs, so they're still pointable & have regions etc… 
the thumbnail regions could all be captured from table session when the snapshot is taken (& shrunk down to fit on page), or i guess they could just be a grid if you have a statement that is just a straight list of program numbers like you propose?")

>> Could you have a separate process that journals the holds and writes to disk on crash or quit?

=== Intercept discussion ===

Intercept feature for the database so you can 'mixin' more stuff without modifying the original program to explicitly check for hooks?

{{:newsletters:pasted:20250228-193350.png?500px}}

(Omar: "side note: it would be cool if (maybe if we had statement blocking) you could implement smoothing and freezing without needing to modify the virtual program, like you could just impose your own mixin 'filters' to set smoothing policy for individual pages")

Mason proposal to have Intercept block:

<code>
Intercept tag $tagOfInterest has quad /quad/ {
    # behavior that overrides current effect
}
</code>

> So when that statement would enter the DB it traps it instead
> In fact in the body it could just forward the statement on so the system continues like nothing happened
> It feels like it would be more reactive this way as well

then some [[https://discord.com/channels/956758212152025098/956765077883744266/1341096730695041095|further discussion]] of how multiple Intercepts on the same statement would interact. Naveen brought up CLOS method combinators? (:before, :after, :around)

=== How to improve quad/space/drawing syntax discussion ===

we need to get rid of regions eventually and make 2D and 3D more idiomatic, regions just remain in for backward-compatibility -- but what should we replace them with?

Omar thoughts on what operations should be easy:

> basically, i guess we want it to be… easy to do things that we'd want to do…
> 
>  * i think drawing a line between centroids is a good example
>  * getting the distance in meters between centroids (or between the closest points on quads) also a good example
>  * outlining quads is another good example
>  * drawing points-up pointer
>  * drawing/constructing an offset quad (this is something that's a lot easier to do in tag-space than in camera/display-space right now)

Some general proposals:

  * Mason: <code>
When /tag1/ is tag1 & /tag2/ is tag2 &\
     tag /tag1/ has centroid /centroid1/ on display /disp/ &\
     tag /tag2/ has centroid /centroid2/ on display /disp/ {
    Wish to draw a stroke on display $disp with \
         points [list $centroid1 $centroid2] width 2 color white
}</code>
  * Mason: <code>
Draw on plane $tag1 {
  a line from {0 0} to {0.5 0.2}
}

set posTag1 [centroid $tag1]
set posTag2 [centroid [projectOnto $tag2 $tag1]]
Draw on plane $tag1 {
  a line from $posTag1 to $posTag2
}</code>

==== Portable gadget ====

Omar: Got a [[https://www.amazon.com/dp/B0DQ4R9S6W|narrower-angle ELP stereo camera]] for the latest revision of the gadget. (This camera has horizontal field-of-view of 75 degrees and vertical field-of-view of 60 degrees, much narrower than the [[https://www.amazon.com/dp/B0D9793XZN|previous ELP stereo camera]] I was trying which has 120 degree HFOV. That was way too wide, wasted a ton of pixels outside the area covered by the projector, distorted the image a lot to make it impossible to calibrate.)

{{:newsletters:pasted:20250228-205341.jpeg?200px}}

Designed a new case that is longer (so we can fit a back panel on the back and fully enclose it), plus the back panel (with holes for speaker & power button & power jack). 

{{newsletters:img_7928-copy.jpg?200px}}

(Annoyingly, the very first gadget took like 8 hours to print -- no battery, smaller projector and computer and camera, and the gadget now takes like 21 hours to print...)

Next: calibrate it and figure out how to make it produce its own Wi-Fi so we can connect laptop to it to calibrate/control it in outside environments.

=== Calibrating the new gadget ===

Some issues with calibration still, but I think we're close to fixing it. 

At first, the issue is that you only see a small band of the projection in the camera frame, I guess because it's a scanning projector and the exposure time of the camera (a couple ms?) is lower than the total scan time of a projector frame (and maybe also because the ELP camera is global shutter?):

{{newsletters:dd70fe6a-b31b-4263-b6cb-5aff53674fd2-6265-0000033d5969f697.png?400px}}

So I added the setting to crank exposure time up to a few ms or even more (tens or hundreds of ms, although I don't know if I believe that). Then I ran into this (and had to patch over by hard-coding 30Hz):

> funny issue from yesterday:
>
> due to it having a global shutter & the pico projector being a scanning projector (so you don't see the whole image in most frames), the ELP stereo camera on the new gadget can calibrate only if you turn exposure wayyy up (300,000 us?)
>
> (although maybe they just scale it differently and it's actually 30,000 us, which makes more sense since it's long enough for 2 whole projector frames) 
> {{newsletters:49770d09-5843-4771-b60c-7255e8112e3b-6265-0000033cf830a44c.png?400px}}
> but you get these bizarre artifacts on the right-eye image where the bottom 1/4 of the image is totally messed up
> if you turn up exposure that high
> it turns out this doesn't happen if you go down from 60Hz to 30Hz on the camera so i'll probably just add a setup.folk camera param for that 
> (it was weirding me out because still-image captures from the terminal looked fine, which fits with this -- the raw color JPEGs from the camera are also corrupt in the same way, which also fits with it being a camera-side problem -- I expect if it were an issue on our side, it wouldn't just affect the right image lol) 

I think it might be fine now but need to actually test calibration of the gadget.
==== New Folk evaluator ====

Omar: have made a lot of progress on the new evaluator. It's starting to feel like a real competitor to folk1. Including:

  * Fixed [[https://github.com/FolkComputer/folk/commit/ba72061b844206bb07f5f29472b303f38fdf8fdc|horrible sign error bug]] that made ''keep ms'' not actually work -- this fixed most of the outline blinking!  (because we keep the tag detection for ~16ms, until the next frame's detection comes in, and it actually keeps now)
  * [[https://github.com/FolkComputer/folk/commit/d9f0f154639c92f883a1bfaa003daf17e83270e2|New scheduling]] that uses [[https://github.com/FolkComputer/folk/commit/2f91d34d87a5a7b506dded892e90c36c50a52bdb|I/O-boundness]] (by checking /proc for each thread) to figure out if new threads need to be awakened to stay responsive. Maintain a pool (10-20) of threads that are mostly asleep so we don't have to literally spawn new threads (slow) for this.
    * Seems to work well in practice! Utilization of CPUs in Tracy looks pretty good
  * Inlined a lot of operations (Claim, Wish, When) so we don't have to trampoline to the workqueue (slow) to do them. Can do this because we don't lock the db anymore for those so is ok to just do them right away. The workqueue is pretty much just for queueing Tcl evals now
  * Per-thread object caching 
    * Lots of little improvements to object reuse (don't set source info because that stomps the whole object representation)
  * Much improved source line info / stacktraces, should be line-accurate even if your When is multiline (it uses the provenance info for the body string instead of the line where the When starts)
  * Added join support to Query!
  * Implemented ''Every time'' to get the keyboard to work (including the ability to trigger an Unmatch); made it use ''When'' for outer patternand ''Query!'' for all inner patterns instead of nested Whens
  * Overall fixed keyboard so you can actually edit stuff; reap keyboard events off-thread
  * Reap AprilTags when flipped over (fixes issue Rob has below)
  * Fixed stdout printing -- this has been a minor annoyance in folk2 for ages
  * Fixed repeated reloading of C modules which was a big performance drag
  * ./folk PROGRAM.folk now runs the PROGRAM in a When context, instead of a bare context, so you can do Wish/Claim/When normally inside your program. Great for writing quick tests. It also just feels nice to have Folk able to act as a 'normal' self-contained programming language runtime that can run scripts if needed
  * Failed project (for now): the 'environment store' which was meant to reduce the amount of captured data from lexical scope around functions/When bodies that we're shipping around, dedupe scopes that are actually the same by giving them pointer IDs and keeping environments in a global multithreaded store
    * Burned a lot of time just getting the reference counting of these envs to be correct (given destructors have envs, functions may outlive their original When, etc)
    * Broke too much stuff: it breaks a lot of When deduplication because those pointers are always unique, so two Whens with identical captured variables no longer merge
    * Also would create awkward JS-like bugs if you call Hold! in a loop (the environment ID & code is always the same, so the Hold! doesn't run more than once)

Most important, I've reconciled folk2 so it now lives in a real branch on the Folk repo (and not a separate repo that can't actually merged), can be tested by other people and ultimately PRed and merged. Deleted a lot of extra files from folk1 while doing that reconciliation, so the repo looks pretty clean.

Anyway, immediate next steps are to fix editor blinking while you type, then make a draft pull request.

=== Editor blinking ===


Notice the blinking of the editor -- this is extreme (other slow stuff is happening on the table), but it happens occasionally even while just normally typing/backspacing:

{{newsletters:img_7894.mp4?300px}}

Tracy view when I type a bunch of stuff (see the blips on the keyboard thread for when I'm pressing keys) and the editor blinks out (see the dips on the plot at the bottom):

{{newsletters:263550d2-1c28-4792-9e18-60b3d280c982-6265-0000034beb6ac656.png?450px}}

I did optimize the editor a bit by [[https://github.com/FolkComputer/folk/commit/a88d925f541efde46169e55ad9828620232d5f5f|decoupling cursor state from rendering]], so it only blinks when you actually edit the code, not when you're moving the cursor. (It's cool how just by separating your logic into different independent Whens, your program feels like it gets more and more parallel and reliable...)

=== Rob's test ===

Rob Fielding in New Zealand [[https://www.youtube.com/watch?v=a0Xn_YK3WzY&feature=youtu.be|tested his orbital simulation on folk2]] -- looks like it pretty much works! (pleasantly surprised me)

His comments were that programs don't vanish when flipped over (which has since been fixed) and that it burns a lot of CPU / makes a lot of heat on his system (it's summer in NZ). Worth thinking about how to make the workers not just spin on work stealing (especially for mobile systems that are battery-constrained):

{{:newsletters:pasted:20250228-194037.png?500px}}

=== Shared immutable (immortal) objects ===

Omar: Mason and I spent a while throwing around ideas in Discord about how to share objects between threads (separate Jim interpreters) in folk2 without copying/reparsing (which is what both folk1 and folk2 do now, although it's a more acute problem in folk2 because stuff comes fresh out of the DB for every When execution)

('immortal' is [[https://peps.python.org/pep-0683/|a term from Python]] for what I think is a similar concept)

I think we have an OK plan for this, actually, and I [[https://github.com/FolkComputer/folk/tree/osnr/folk2-transients|started hacking on it]], but haven't really fully implemented it yet. Something to keep in pocket if we need it, maybe. (for now, the aggressive per-thread object cache mitigates the copying and reparsing a lot)
==== Friends and outreach ====

=== Columbia GSAPP guest lecture ===

Our guest lecture to the [[https://www.arch.columbia.edu/courses/11379-5166|Design in Action class]] at Columbia GSAPP went well. We brought a table-scale Folk installation rig by packing up the projector-camera-computer combo system on a tripod (normally set up above our CNC machine at Hex House).

We walked the students through our motivations for building Folk Computer, the unique capabilities of tangible spatial computing that it allows, and showed off a few demos. We rounded out the 90-minute lecture by taking questions and talking about our paths to doing independent research on spatial computing.

  * The lecture room:
    * {{newsletters:img_0919-large.jpeg?0x300}} {{newsletters:img_0971-large.jpeg?0x300}}
  * Calibrating the system:
    * {{newsletters:img_0377-large.jpeg?0x300}} {{newsletters:img_0959-large.jpeg?height=0x300}}
  * Demo programs:
    * {{newsletters:img_0987-large.jpeg?0x300}}{{newsletters:img_0985-large.jpeg?0x300}} {{newsletters:img_0908-large.jpeg?0x300}}
    * {{newsletters:img_0923-large.jpeg?0x250}}{{newsletters:img_0904-large.jpeg?0x250}}{{newsletters:img_0386-large.jpeg?0x250}}{{newsletters:img_0928-large.jpeg?0x250}}{{newsletters:img_0975-large.jpeg?0x250}}

(Omar: a fun moment was live-stabilizing the animation program from the tabletop editor using Mason's new stabilization stuff. was also happy that we got the cat printer to work & people actually printed and ran programs on it, although they had to hold them close to the camera : )


=== Open house ===

We had a lively open house on Monday, February 24:

  * Animations:
    * {{newsletters:smiley.gif}}{{newsletters:egg.gif}}
  * {{newsletters:group_overhead.jpeg?0x250}}{{newsletters:table_pic.jpeg?0x250}}{{newsletters:group.jpeg?0x250}}
  * {{newsletters:20250224_193715.jpg?0x250}} {{newsletters:20250224_200025.jpg?0x250}}

(Omar: a fun moment was when we were rolling Azlen's die, but its title text was too small for people to see, so I live-coded a Folk program to subscribe to its value and project it way bigger in the corner of the table)

A group from Fractal Tech (nearby in NYC) came by the open house and are excited to set up their own system and do a launch event and hack day:

> We were blown away. I felt my eyes well up at one point when you guys were writing a program in your own language. This is how it must have felt back at DARPA and Xerox Labs. I have not ever experienced such a computing paradigm. 
>
> Afterwards we went to Veselka to crush some pierogis and ideate. It sounds like we are planning both a folk computer launch party and a hackathon. Both will be announced here for all to enjoy at our space just down the block: http://fractaltechhub.com/
> 
> For now, time to build and test!

=== SynergyMill ===

Paul, who previously gave [[https://www.youtube.com/watch?v=jP1FiJDB-tg|a demo of Folk to the South Carolina Linux User Group]], gave a general intro and programming demo to some of the members of [[https://www.synergymill.com/|SynergyMill]], a community DIY workshop.

  * They appreciate the newly stabilized editor, "it makes the projected code easier to read."
  * 3D calibration of their system is benefitting them a lot, since it means they can move their 'lamp head' around arbitrarily without needing to recalibrate it (calibration keeps working as long as the camera and projector are still positioned the same relative to each other)
  * Paul was used to cutting out the AprilTags and getting rid of the code to save table-space, but when explaining Folk programs to other people he realized the utility of being able to point to printed-out-code next to the corresponding AprilTag. A good quote from someone at the meetup: “it’s like we’re going back to punchcards” after Paul explained how programs are represented by their physical pieces of paper with the Folk system.
     * Omar, in our Discord, responding to this point from Paul: "idly wonder if having bare tags vs. leaving the code attached will reflect a single-person system vs. shared system divide (if it's just you, you can just have a one-line title under bare tags; if you're showing it to other people, you might prefer full source code)"
     * Andrés: it's interesting to imagine these sort of "physical dialects" shifting with Folk systems by the community's interests or physical constraints

Photos from the event:

{{newsletters:folk-meetup1-large.jpeg?0x300}}{{newsletters:folk-meetup0-large.jpeg?0x300}}

=== Other visitors and interactions ===

  * Our friend [[https://timhwang.org|Tim Hwang]] and his friend [[https://helentseng.com|Helen Shewolfe Tseng]] came by -- Helen visiting from SF. Talked about zines and books in Folk
    * {{newsletters:untitleddd.jpg?280px}} {{newsletters:6dda9538-0e95-4cec-bc97-ed4b2cff2821img_7386.mp4?140px}}
  * Started work with [[https://www.kosmik.app|Kosmik]] to help them set up a Folk system (have been stuck on calibrating it for a while, had to mess with cameras and set up projectors). Excited to get to do the demos/apps
  * Violet Whitney and William Martin from [[https://spatialpixel.com|Spatial Pixel]] came up from Philadelphia and spent an afternoon with Omar in the studio
    * Discussed how we could do some events/demos together, maybe integrate the 2 systems in some lightweight way since they have different strengths and capabilities
      * Folk can do full 3D camera-projector calibration which has been a struggle for Procession, maybe put segmentation/edge-finding on top too
      * Folk plugin for Procession that tells the LLM how to input/output stuff

===== What we'll be up to in March =====

  * **Our next Folk open house is [[https://partiful.com/e/8zlZKFFAvyeFbngr9PUe|in the evening on Wednesday, March 26]], at our studio in East Williamsburg, Brooklyn.**
  * Omar: Calibrate gadget; put up gadget design on GitHub
  * Omar: Fix editor blinking (performance improvements & maybe semantic changes); make pull request for folk2 evaluator
  * Andrés: Work on video
  * Andrés: Dot detector port to work under 3D calibration
  * Andrés: Potentially speaking at NYU's Learning Analytics class
  * Continuing to work with Kosmik on setting up their Folk system & doing a demo/integration

===== Links we've enjoyed =====

==== Omar ====

  * https://graydon2.dreamwidth.org/313862.html
  * https://www.cambridge.org/in/universitypress/subjects/computer-science/computing-general-interest/calendrical-calculations-ultimate-edition-4th-edition
  * https://www.publicbooks.org/cooking-monasteries-arithmetic-lorraine-daston-on-the-history-of-rules/
  * https://codingrelic.geekhold.com/2008/12/printf-acular.html
  * https://languagehat.com/peper-and-solt/
  * https://jcs.org/bluescsi
  * https://hforsten.com/homemade-polarimetric-synthetic-aperture-radar-drone.html
  * https://nullprogram.com/blog/2023/09/30/
  * https://weirdgloop.org/blog/why-were-helping-more-wikis-move-away-from-fandom
  * https://gladdendesign.com/collections/paper-apps

==== Andrés ====
  * [[https://www.instagram.com/reel/DFdi2U9tgEA/|Liquid paper sculptures]]
  * [[https://endtimes.dev/why-you-should-dither-images/|Why Your Website Should Use Dithered Images]]
  * [[https://www.greendesign.io/dithering-images|Dither images]]
  * [[https://x.com/kevintwohy/status/1867964091769430262?s=12&t=AUhNsAKs8G7beopf-_iD3Q|Projected wayfinding in Newark Airport]]