Differences

This shows you the differences between two versions of the page.

--- newsletters:2024-07 [2024/07/30 21:09] – osnr
+++ newsletters:2024-07 [2024/08/01 02:56] (current) – [Friends and outreach] osnr
@@ Line 1: / Line 1: @@
-====== July 2024 newsletter (WIP) ======
+====== July 2024 newsletter ======
 ===== What we've been up to =====
@@ Line 5: / Line 5: @@
 ==== Demos ====
-  * [[https://jessie.grosen.systems|Jessie Grosen]] made a [[https://supercollider.github.io|SuperCollider]]-based audio system using Folk over at Recurse Center
+  * [[https://jessie.grosen.systems|Jessie Grosen]] made a [[https://supercollider.github.io|SuperCollider]]-based audio system using Folk, over at [[https://www.recurse.com|Recurse Center]] in Downtown Brooklyn:
-    * TODO: video?
+    * {{newsletters:initial_demo.mp4}}
-    * (building on [[https://github.com/FolkComputer/folk/blob/main/virtual-programs/music.folk|some]] of the work that [[https://ijc8.me|Ian Clester]] did on [[https://www.youtube.com/watch?v=9Jik_CfcaGA|music in Folk]] last year)
+    * (Jessie has been [[https://github.com/FolkComputer/folk/commit/91306dde05d98ea6e114812f23afb3f03560d674|building]] on [[https://github.com/FolkComputer/folk/blob/main/virtual-programs/music.folk|some]] of the work that [[https://ijc8.me|Ian Clester]] did on [[https://www.youtube.com/watch?v=9Jik_CfcaGA|music in Folk]] last year)
     * (also building on [[https://x.com/tobyshooters/status/1818017799626793456|Cristóbal Sciutto's]] [[notes:tableshots|hands]]!)
-  * Andrés made a "container" demo. The pink and blue programs below search for ''neighbors'' to be immediately next to them or within their bounds and give those programs ''thick (pink||blue)'' outlines. This is a simple example but you could imagine putting photos into Twitter or Instagram ''containers'' and then uploading them from the table, or sorting through files on your computer by placing them into various Folk ''containers'' that correspond to your filesystem. These kinds of physical → digital interactions are something we're really excited about Folk powering.
+  * Andrés made a "container" demo. The pink and blue programs below search for ''neighbors'' that are immediately next to them or within their bounds and give those programs ''thick (pink||blue)'' outlines.
     * {{newsletters:img_0767.gif}}
-    * Andrés worked with [[http://charstiles.com|Char Stiles]] to [[https://folk.computer/guides/gpu|update the GPU guide]], making it easier to get started with uniforms, functions, etc.
+    * This is a simple example, but you could imagine putting photos into Twitter or Instagram ''containers'' and then uploading them from the table, or sorting through files on your computer by placing them into various Folk ''containers'' that correspond to your filesystem. These kinds of physical → digital interactions are something we're really excited about Folk powering.
-  * TODO: Switch brancher
+  * Andrés worked with [[http://charstiles.com|Char Stiles]] to [[https://folk.computer/guides/gpu|update the GPU guide]], making it easier to get started with uniforms, functions, etc.
-  * TODO: 3d demo
+  * Omar made some Switch Brancher tools: 3D calibration (discussed later in this newsletter) has gotten advanced enough that we want to swap systems in and out of the 3D calibration branch without needing a laptop to ssh & git (we still want to go back to main sometimes since compatibility isn't 100% yet)
+    *  {{newsletters:img_0600.jpg?250}} {{:newsletters:pasted:20240731-194334.jpeg?250}} {{:newsletters:pasted:20240731-194420.jpeg?250}}
+  * Omar got the 3D distance demo [[newsletters:2024-06#real-world3d-calibration|from his apartment last month]] to run on folk0 and made it more plug-and-play:
+    * {{newsletters:img_0609.jpeg?400}}
+    * The printed program at the top in the middle shows the distance (in centimeters) between the program on the left and the one on the right. The target programs are specified by the program being written in the editor at the bottom. Watch the distance change as I move the program:
+      * {{:newsletters:pasted:20240801-023436.mp4?400}}
+  * Andrés has been working on more complicated shader examples:
+    * A [[https://x.com/cwervo/status/1817341598788636800/video/1|scavenger hunt in screen space]] for a red oval on ''folk0'':
+      * {{newsletters:folk_shader_oval_window_to_fullscreen.gif}}
+    * A black shader programming moving over a full-table shader -- feels like there's a ghost or something in the room:
+      * {{newsletters:folk_shader_sliding_black_shader.gif}}
+    * Playing with opacity in shaders and shapes in Folk:
+      * {{newsletters:folk_red_circle_opacity_test.gif}}
   * [[https://mmm.s-ol.nu|s-ol bekic]] did an "augmented brainstorming" session in Milan on folk-opendot -- this might be the largest-scale use of Folk so far!
     * {{:newsletters:pasted:20240730-185341.jpeg?300}} {{:newsletters:pasted:20240730-185353.jpeg?300}} {{:newsletters:pasted:20240730-185347.jpeg?300}}
@@ Line 19: / Line 31: @@
       * "Performance went down to like 2fps and some things stopped working as I expected (in ways that I don't understand yet)"
       * Omar: more motivation for performance work, haha
-  * TODO: opacity
-==== Other system improvements ====
+==== System improvements ====
-  * Andrés has been working on Commit -> Hold. TODO: Need backward compatibility (?)
+  * Andrés has been working on [[https://github.com/FolkComputer/folk/pull/161|changing ''Commit'' → ''Hold'']]
-  * TODO: s-ol web editor
+    * This change is mostly done, need to do another test on more complicated programs that use ''Commit'' in Folk (n.b. if you're a Folk user, please test this branch out for bugs). Andrés is going to do some more testing August 1st and hopefully will merge the branch August 2nd.
-  * TODO: s-ol layer support in all drawing primitives
+  * Andrés and s-ol worked on [[https://github.com/FolkComputer/folk/pull/159|removing the need for passwordless sudo to use keyboards]]
-  * TODO: small updates for opening GitHub repo to public
+  * Various [[https://github.com/FolkComputer/folk/pull/160|small fixes from s-ol]] (prep for opening GitHub repo to public):
+    * Web editor can now edit folk-live and user-programs, not just core virtual-programs
+    * Support for z ''layer'' in all drawing primitives
+    * ''Wish PAGE is filled with color blue'' wish
+  * Omar has been working on merging [[https://github.com/FolkComputer/folk-live-build|folk-live-build]] into the [[https://github.com/FolkComputer/folk|main Folk repo]]
+    * I think we want to get into more of a monorepo situation where everything gets versioned together (so you know exactly what dependencies a given Folk version has, what OS you need, etc). Also want to vendor the AprilTag library soon
 ==== 3D calibration ====
-Omar: Our goal is to merge the 3D calibration branch ([[https://github.com/FolkComputer/folk/pull/162|osnr/checkerboard-calibration]]) into main, so we want it to be roughly on par with the current 2D calibration (ideally much better, though) on all systems. We made a lot of progress this month. A the end of last month, we were trying to figure out why folk0 at Hex House doesn't calibrate that well compared to the system at my apartment. (it was like 1-2cm off compared to 1-2mm off) (maybe because folk0 is a physically big system, it has different camera and projector optics, etc).
+Omar: Our goal is to merge the 3D calibration branch ([[https://github.com/FolkComputer/folk/pull/162|osnr/checkerboard-calibration]]) into main, so we want it to be roughly on par with the current 2D calibration (ideally much better, though) on all systems.
+We made a lot of progress this month. At the end of last month, we were trying to figure out why folk0 at Hex House doesn't calibrate that well compared to the system at my apartment. (it was like 1-2cm off compared to 1-2mm off) (maybe because folk0 is a physically big system, it has different camera and projector optics, etc).
+Now we are able to calibrate folk0 to at least the level that 2D calibration is at most of the time (thanks to various accuracy and geometry fixes), and we've created the pull request and done a lot of polish so people can start to test it.
 === Big calibration board ===
@@ Line 39: / Line 59: @@
 {{:newsletters:pasted:20240730-204821.png?450}}
-If true, this would be very annoying, because we can't really ask people to fabricate a big board (you have to print several pages, find a board to glue them onto, measure very carefully). We might need to use an entirely different technique like the points-of-light, which would be a lot of new engineering.
+If this theory were true and our board just isn't big enough, it would be very annoying, because we can't really ask people to fabricate a big board (you have to print several pages, find a board to glue them onto, measure very carefully). We might need to use an entirely different technique like [[https://www.youtube.com/watch?v=Npd0rKHBH_w|the points-of-light from Yang (2016)]], which would be a lot of new engineering to reimplement.
-TODO: photo of big board?
+{{:newsletters:pasted:20240731-194833.jpeg?300}}
-I made a big board and calibrated with it, but only got slightly better results (maybe more due to better tag detection than board size?)
+I made a big board and calibrated with it, but only got slightly better results (maybe more due to better tag detection than board size?).
 === Testing old papers on folk0 ===
-So I tried to test some old projector-camera calibration papers -- do any of them get better results on folk0 (presumably because we implemented something wrong) or is this an inherent problem with the approach / is our board just too small?
+So I tried to test some old projector-camera calibration papers -- do any of them get better results on folk0 (presumably because we implemented something wrong) with a similarly small board, so we could keep iterating on our current technique, or is this an inherent problem with the approach / is our board just not big enough?
 Lots of struggles in that replication work:
@@ Line 53: / Line 73: @@
 {{:newsletters:pasted:20240730-155724.png?450}}
-[[http://www.ok.sc.e.titech.ac.jp/res/PCS/research/procamcalib/|Audet's]] original [[https://github.com/bytedeco/procamcalib|ProCamCalib]] was the only one I could actually get to work. I made a table of data from different Folk and Audet calibrations on folk0:
+[[http://www.ok.sc.e.titech.ac.jp/res/PCS/research/procamcalib/|Audet's]] original [[https://github.com/bytedeco/procamcalib|ProCamCalib]] was the only one I could actually get to work.
+{{:newsletters:pasted:20240730-213641.mp4?0x350}} {{:newsletters:pasted:20240801-021925.jpeg?0x350}}
+I made a table of data from different Folk and Audet calibrations on folk0:
 {{:newsletters:pasted:20240730-155857.png}}
@@ Line 65: / Line 89: @@
 === Small accuracy improvements ===
-Buoyed by the fact that at least Audet gets better results with more or less the same scheme (and with a similarly small calibration board), I was able to make multiple small accuracy improvements:
+Buoyed by the fact that at least Audet can get better results with a similar scheme to us (and with a similarly small calibration board), I was able to make multiple small accuracy improvements:
   * Gate pose recording based on pose homography RMSE: we now require the printed and projected tags to be well-aligned to a proper linear grid (RMSE should be less than 5 pixels), or else we won't record a pose. we previously relied only on consistency over the last few frames ('fixed point') as a proxy for this, but I feel more comfortable now having the direct test
@@ Line 81: / Line 105: @@
   * "Print through Folk"
   * semi-live camera preview (so you can debug focus/exposure/physical placement issues without leaving the calibration page)
+    * (we're also finally [[https://github.com/FolkComputer/folk/blob/9f4726917b56c29e74b61b430e05c3c64c864fe1/virtual-programs/web/web-camera-frame.folk|building ''/camera-frame'' into the base virtual programs]], since we need it to do the camera preview)
   * slider for adjusting default projected tag size so it'll fit (also lets you reset state if it gets in a weird state where self-detection is broken)
   * result reporting (pose status, RMSEs of poses and calibrations)
   * more explicit instructions
-(We probably should have a demonstration video here, honestly.)
+(We probably should have a demonstration video embedded in the calibration page as well.)
 {{:newsletters:pasted:20240730-143708.png?400}} {{:newsletters:pasted:20240730-143725.png?400}}
@@ Line 108: / Line 133: @@
 So I added a new geometry specification step to the calibration process. This creates a default.folkgeom file that states the default tag-program geometry for that Folk system.
-TODO: image of geometry step
+{{:newsletters:pasted:20240731-200200.png?400}}
 I also added support for .folkgeom files, which can manually override the geometry for a specific tag. (so you can make 642.folkgeom and tell Folk that that tag is a weird size and shape, like maybe it's 40mm and has a square around it). I think something like this was always going to be necessary if people want to be able to manually cut and recopy tags.
@@ Line 117: / Line 142: @@
 {{:newsletters:pasted:20240730-195953.png?600}}
+(the .folktag approach I suggest in those messages morphed into the .folkgeom files that we now are using in the branch)
 === Testing 3D calibration so far ===
@@ Line 141: / Line 168: @@
 Ported the keyboard driver from the old evaluator. Now that keyboards seem to work, I want to work on porting the editor, which is a good test of language capability (maybe we want mutable variables? maybe we want event statements?) and of reliability (the editor has to maintain a lot of transient state) and performance (it slows down old Folk a //lot// when you have a big source file and/or multiple editors at once, and we want to fix problems like that). I'm excited to have a really smooth editor that is as comfortable to work in as a laptop editor, where you're not scared to edit a long shader or have multiple people coding on the table at the same time.
 ==== Friends and outreach ====
   * Our July open house was the biggest we've had in a while -- it was at 8pm on a Saturday (partly forced on us by lack of air conditioning at Hex House, but we also wanted to celebrate printing program 1000 last month):
-    * [TODO: Andrés pix]
     * {{:newsletters:pasted:20240730-144817.jpeg?250}} {{:newsletters:pasted:20240730-144842.jpeg?250}} {{:newsletters:pasted:20240730-144857.jpeg?250}} {{:newsletters:pasted:20240730-144906.jpeg?250}} {{:newsletters:pasted:20240730-144913.jpeg?250}} {{:newsletters:pasted:20240730-144919.jpeg?250}}
+    * Andrés had people sign or doodle on their Guestbook programs this time around -- visitors had fun leaving their mark on their programs:
+      * {{:newsletters:pasted:20240731-195056.jpeg?350}}
   * We brought the [[https://mas.to/@TodePond/112850475064168079|projector gadget]] and crank device to the Malleable Computing dinner at the [[https://val.town|Val Town]] office in Brooklyn, met many people who were in town ([[https://www.todepond.com|like]] [[https://x.com/cwervo/status/1816652260459577436|Lu Wilson]] and [[https://x.com/OrionReedOne/status/1817812600136102114|Orion Reed]] and [[https://x.com/round/status/1816949552639086621|Maxim Leyzerovich]])
     * [[https://x.com/round/status/1816949552639086621|{{:newsletters:pasted:20240730-193140.png?400}}]]
   * [[https://floating.pt/|Cheng Xu]] visited us -- we talked a bunch about installation art, the advantages of embodied interaction, and using computers for fun over productivity.
-    * {{newsletters:andre-s_demoing_opencv_whatsapp-image-2024-07-06-at-00.14.52-medium.jpeg}}
+    * {{newsletters:andre-s_demoing_opencv_whatsapp-image-2024-07-06-at-00.14.52-medium.jpeg?250}} {{newsletters:cheng_pressing_button_whatsapp-video-2024-07-06-at-00.14.35.gif}}
-    * {{newsletters:cheng_pressing_button_whatsapp-video-2024-07-06-at-00.14.35.gif}}
+  * [[https://annaylin.com|anna y lin]] & [[https://www.yuanzichen.com|Zichen Yuan]] visited -- some excitement about the portable gadget and the new mode of interaction
-  * TODO: Anna & Oliver
+    * {{:newsletters:pasted:20240801-022827.jpeg?200}} {{:newsletters:pasted:20240801-022840.jpeg?200}} {{:newsletters:pasted:20240801-022858.jpeg?200}}
   * Omar and Andrés visited Jessie at Recurse Center and checked out the setup there; tried to run 3D calibration; put up some signs in the hope that future Recursers will be able to use and maintain the system
+    * [[https://x.com/FolkComputer/status/1818003340631961661|{{:newsletters:pasted:20240731-164323.png?400}}]]
     * We'd been excited to have an RC system for a while -- just needed hardware and an internal maintainer at RC
-    * {{:newsletters:pasted:20240730-201833.jpeg?250}} {{:newsletters:pasted:20240730-201840.jpeg?250}} {{newsletters:img_0544.jpeg?250}}
+    * {{newsletters:gtrvenfwoaanh0y.jpeg?0x250}} {{:newsletters:pasted:20240730-201840.jpeg?0x250}} {{newsletters:img_0544.jpeg?0x250}}
   * Our old colleague [[https://weiwei.place|Weiwei Xu]] stopped in while she was visiting the city last week -- it's been a couple of years, so it was nice to show all the changes:
-    * {{:newsletters:pasted:20240730-202458.jpeg?250}}
+    * {{:newsletters:pasted:20240730-202458.jpeg?200}}
+  * today, we met some of our very close neighbors at [[https://www.instagram.com/boshisplace|boshi's place]] and gave them a tour of Hex House and Folk!
+    * {{newsletters:453879836_17924697227925430_5273187919321543829_n.jpg?250}}
 ===== What we'll be up to in August =====
-  * **Our next [[TODO|Folk open house is on the night of ??????, August ??]] at our studio in East Williamsburg, Brooklyn.**
+  * **Our next [[https://partiful.com/e/Il1VIfI2qVrOSmx5RntO|Folk open house is on the night of Sunday, August 18]] at our studio in East Williamsburg, Brooklyn.**
+    * Possibly do a workshop to get people familiar with making stuff in the system beforehand
   * 3D calibration: one last accuracy/UI push, get a couple more people to test, then merge :-)
     * also minor compatibility fixes: animation, mask-tags
-    * if we can get it merged, I hope we can start working on some really exciting stuff with precise camera slices: OCRing fields, checkboxes, stuff like that
+    * Omar: if we can get it merged, I hope we can start working on some really exciting stuff with precise camera slices: OCRing fields, checkboxes, stuff like that
   * New evaluator: porting the editor, doing correctness/blinking fixup (maybe we track provenance of statements?)
   * Gadget: need to look into issues with power delivery and with crashing. Maybe upgrade to Pi 5? And build button?
-  * Do a workshop
 ===== Links we've enjoyed =====