Differences

This shows you the differences between two versions of the page.

--- newsletters:2024-06 [2024/06/28 20:59] – [Portable gadget] osnr
+++ newsletters:2024-06 [2024/07/01 08:37] (current) – osnr
@@ Line 1: / Line 1: @@
-====== June 2024 newsletter (WIP) ======
+====== June 2024 newsletter ======
 ===== What we've been up to =====
@@ Line 5: / Line 5: @@
 ==== Demos ====
-  * We printed the 1000th program on folk0 (at Hex House)!
+  * We printed the 1000th program on folk0 at Hex House!
     * {{newsletters:img_5644.mp4?200px}}
   * Our friend [[https://github.com/antimatter15|Kevin Kwok]] came by and reprogrammed his crank gadget to display Folk program AprilTags; we cranked through a lot of the history on folk0, which was fun
     * {{newsletters:img_9029.jpeg?150px}}
-    * {{newsletters:spin_e_apriltags.gif}}
+    * {{newsletters:spin_e_apriltags_smol.gif}}
   * [[https://arcades.agency|Arcade Wise]] made a [[folkarc/uploads|file upload page]] which lets you upload files from a phone/laptop (so you can use them in Folk):
     * {{newsletters:uploader-image0.jpg?200px}}
@@ Line 26: / Line 26: @@
       * [[https://mmm.s-ol.nu|s-ol bekic]] has been working on a branch for supporting WebRTC streaming of video and images in Folk:
         *  {{newsletters:sol-folk-webrtc.png}}
-  * TODO:
+      * It looks like the two branches can eventually be merged, and we might actually pick GStreamer over ''ffmpeg'' because it can also handle audio routing.
 ==== Real-world/3D calibration ====
-Omar: A lot of progress on 3D calibration this month.
+Omar: A lot of progress on 3D calibration this month. As a reminder, the point of this 3D calibration project is
-  * Long list of small steps to improve the accuracy of the intrinsic and extrinsic calibrations:
+  - to track programs in 3D, so you can lift them off the table and they'll projection-map properly, but also
+  - to build much more accurate tracking in general -- millimeter precision -- to enable new applications where you can highlight individual words or facets of objects, and
+  - to track coordinates in real-world meters instead of arbitrary projector or camera pixels, so we can start to integrate multiple projectors, cameras, and other real-world sensors and actuators (phone localization, RFID, CNC machine bed and material, spatial audio...), which will all benefit from having a shared ground coordinate system
+This video (running the osnr/checkerboard-calibration branch on my home system) shows how you can accurately measure the distance between 2 pages in real centimeters, as well as how accurate the 3D projection map can get right now:
+{{newsletters:img_9458.mp4?400px}}
+The distance is being displayed by this sort of hacked-together program using the new concept of ''quad'', which we may change/simplify/wrap:
+<code>
+When 19 has quad /a/ & 13 has quad /b/ {
+  set a [quad change $a "/dev/video4"]
+  set b [quad change $b "/dev/video4"]
+  Wish to draw text with x 800 y 400 text [format {%.2f} [* 100 [norm [sub [quad right $a] [quad left $b]]]]]cm radians 3.14
+}
+</code>
+This past month:
+  * Many small steps to improve the accuracy of the intrinsic and extrinsic calibrations:
     * Instead of end-to-end refinement using the pose estimate, which is slow, we now separately refine the intrinsic calibrations of the camera and projector, which is the more classic [[https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr98-71.pdf|Zhang]]/[[http://www.ok.sc.e.titech.ac.jp/res/PCS/publications/procams2009.pdf|Audet]] technique
       * Then we refine the extrinsics (rotation and translation from camera 3D space -> projector 3D space) as a third distinct step
@@ Line 40: / Line 60: @@
     * Block out skew parameter of cam/proj intrinsics, can assume that it is always 0 (OpenCV does this so we copy them and assume it's OK)
     * Instead of averaging extrinsics (3D cam->proj rotation and translation) derived from each pose, which doesn't make sense (you can't average rotations), we accumulate a list of point pairs from camera->projector space and then use Kabsch algorithm to get a rotation and translation estimate, then do the nonlinear refinement mentioned above
-    * [[https://social.omar.website/@omar/statuses/01J0F5H0GW4HMG3G6B5RR0TQ9T|"Virtual visual servoing"]] pose estimation. Almost completely fixes weird flicker issues! This was surprisingly huge, and it's a really simple technique
+    * [[https://social.omar.website/@omar/statuses/01J0F5H0GW4HMG3G6B5RR0TQ9T|"Virtual visual servoing"]] pose estimation. Almost completely fixes flicker issues with 3D poses! This was surprisingly huge, and it's a really simple technique
+      * Keeps a tiny bit of state (pose from prev frame) and finds the most likely pose for current frame using tag detection + prev state
+      * Can get stuck in weird poses but fixes itself if you cover & uncover tag, this is tolerable for now I think
+      * I really like that this is both pose estimation and 'smoothing' at the same time, and it has some intrinsic model that this is a thing moving around in space that we're trying to estimate -- wonder if same technique could work for RFID
+        * Wonder if we could also teach it about the square tag instead of doing servoing for each point on its own, would that make it more robust
     * Spent a while on performance micro-optimization, went from like 300ms latency to 200ms. Didn't really help on folk0 (and folk-live at my apartment is not that much faster even though its calibration is way better), so ruling out the performance theory of why folk0 doesn't work as well
     * Spent a while on making a separate correct-extrinsics step that you do at the end of calibration, where you leave the board flat on your table and it reprojects tags and measures the offset and tries to correct the extrinsics:
@@ Line 55: / Line 79: @@
-D calibration works extremely well (1-2mm error) on my system in my apartment. Notice how the outline is almost perfectly aligned with the page:
+D calibration works extremely well (1-2mm error) on my personal system in my apartment, as you saw in the video above.
-{{newsletters:img_9350.mp4?300px}}
 It works much less well (1-2cm off) on folk0 at Hex House, and even worse on folk-beads (the cart system at Hex House), where the high distortion and shallow 3D volume of the ultra-short-throw projector make it really hard to calibrate.
@@ Line 63: / Line 85: @@
 I've also made it almost fully backward-compatible with all existing code -- it now derives 2D projector-space region claims from the 3D calibration, so all code that uses regions still works. I deleted all the old 2D calibration and homography and region code (calibrate.tcl), so 3D calibration backs everything on the branch.
-I'd like to get it to at least be on par with the existing 2D calibration in normal cases before we merge it, and we're not there yet...
+I'd like to get it to at least be on par with the existing 2D calibration in most cases before we merge it, and we're not there yet... (working theory is still that the individual intrinsics of camera and projector are derived pretty well, but the extrinsics seem off)
 ----
@@ Line 92: / Line 114: @@
 ----
-In terms of actually being able to use the new evaluator, the main issues are 1. memory crashes and 2. labels/outlines/etc flashing (because statement revocation can happen too early and cause stuff to blink out, it's not strictly ordered to happen after new statements come in like it is on classic Folk). Once the crashes are fixed and we have a good way of dealing with statement revocation (local convergence? marking downstream statements?), we should definitely start moving to the new evaluator.
+{{:newsletters:pasted:20240628-210847.png?500px}}
+----
+After fixing the Hold bug, we got to test the new evaluator on the actual table on folk0. (It's now reliable and fast enough to be worth testing.)
+{{:newsletters:pasted:20240628-210817.png?500px}}
+It performs //really// well, much faster/lower latency/higher throughput than any previous Folk evaluator including the current main one.
+In terms of actually being able to fully switch to the new evaluator, the main open issues are 1. memory crashes and 2. labels/outlines/etc flashing (because statement revocation can happen too early and cause stuff to blink out, it's not strictly ordered to happen after new statements come in like it is on classic Folk). Once the crashes are fixed and we have a good way of dealing with statement revocation (local convergence? marking downstream statements?), we should start moving to the new evaluator.
 ==== Portable gadget ====
-Omar has done a couple [[https://x.com/rsnous/status/1803139935299326152|new iterations]] of the portable gadget from last month (including picking up more CAD software, which is surprisingly inspirational from a UI point of view). Here's what it looks like right now:
+Omar has done a couple [[https://x.com/rsnous/status/1803139935299326152|new iterations]] of the portable gadget from last month (including picking up more CAD software, which as [[https://social.omar.website/@omar/statuses/01J09P0XC14M5TBZTHTRJWPHRA|surprisingly inspirational UI]]). Here's what it looks like right now:
 {{newsletters:img_9297.jpeg?300px}} {{newsletters:img_9291.jpeg?300px}} {{newsletters:img_9299.jpeg?300px}}
-The hand grip and front look good, and the projector and camera and Pi 4 fit well. The hand grip is a little loose, so need rubber or a groove to keep it from swiveling around. The back / wiring isn't fully figured out yet; that also will need a lid/cap that you screw in, I think. I still need to design and wire and do the trigger button, which I think will make a big difference to how you can interact with it.
+The hand grip and front look good, and the projector and camera and Pi 4 fit well. The hand grip is a little loose, so need rubber or a groove to keep it from swiveling around. (The Ultimems projector inside also swivels around and probably also needs rubber or a groove.) The back / wiring isn't fully figured out yet; that also will need a lid/cap that you screw in, I think. I still need to design and wire and do the trigger button, which I think will make a big difference to how you can interact with it.
 I also have issues keeping it powered right now (and I want to figure out a USB power pack situation anyway, where you can choose to use wall power or a power pack). I wonder if the projector is too much for the Pi 4's USB port to power, or if it's overheating, or something.
@@ Line 107: / Line 139: @@
-==== Small system improvements ====
+==== System improvements ====
   * s-ol added FOLK_ERROR macros that can be called from arbitrary C code to immediately return to Tcl and throw a Tcl error
     * Omar: we've wanted this for a while -- we should go through C code and remove a lot of weird calls to exit() and null returns and replace them with proper Tcl errors now. we've had it at the Tcl-C arg/rtype translation layer (it uses setjmp and longjmp to set a checkpoint each time you call from Tcl into C), but s-ol's generalized it so it's callable from user C code also
-  * TODO: s-ol folk.js
-  * TODO: Merged s-ol's work on international keymaps
   * [[https://x.com/naveenma|Naveen Michaud-Agrawal]] [[https://git.folk.computer/folk/commit/?id=d4c238dc9a5cfa97c443d7c2b4ba3628a042a1f8|added support]] for making connections & dynamic connections from points other than the region centroid (so you can connect to left edge, top edge, etc)
     * {{newsletters:pxl_20240612_1151000962.mp4?300px}}
   * Naveen added [[https://git.folk.computer/folk/commit/?id=c14f657f3052680404be9f7571b94f4591ede323|support for the alpha channel in RGBA colors]]
+  * Merged s-ol's work on international keymaps, so keyboards other than US can work (adds console-data dep)
   * Andrés fixed Alt-Esc when a keyboard hasn't been visible to the system yet (it now loads default keymap out of the box, so Alt-Esc works to restart the system)
+=== folk.js ===
+s-ol has added [[https://git.folk.computer/folk/tree/lib/folk.js|folk.js]], a Web client library so you can emit and subscribe to Folk statements from inside a Web page. It also lets you easily convert between Tcl and JS values.
+It feels like it can handle disconnect and status reporting and other such conditions a lot better and cleaner than one-off code. It significantly simplifies the implementation of stuff [[https://git.folk.computer/folk/diff/virtual-programs/web/new-program-web-editor.folk?id=c7cbbe91fae19d63b5fb9c40c2bf43d0f4fbae59|like the Web editor.]]
+> WS message are executed in the context of a match that is tied to the websocket lifetime. An emit prefix msg proc is provided, and the JS code dispatches messages based on the prefix value.
+>
+> Utilities for encoding and decoding Tcl strings are provided, including a tagged template literal:
+>
+> <code>
+const complicatedString = 'This string {contains} some {} odd "stuff"';
+tcl`We ${'embed'} ${complicatedString} here`
+// 'We embed {This string {contains} some {} odd "stuff"} here'
+const json = [ { name: "Sol", age: 27 }, { name: "folk", age: "-1" } ];
+ws.send(tcl`Assert the entities are ${json}`);
+// 'Assert the entities are {{name Sol age 27} {name folk age -1}}'
+const tclVal = '{name Sol age 27} {name folk age -1}';
+loadList(tclVal).map(loadDict);
+// [ { "name": "Sol", "age": "27" }, { "name": "folk", "age": "-1" } ]
+</code>
+It should be very useful and lower the barrier to many phone, laptop, other traditional computing integrations which we've had to do from scratch every time until now (like the work that Arcade has done with tablet and files, that Andrés has done with canvas, & that Omar has done with calibration and CNC preview endpoints).
 ==== Friends and outreach ====
@@ Line 128: / Line 184: @@
 ===== What we'll be up to in July =====
-  * **Our next [[https://partiful.com/e/|Folk open house is in the afternoon of Wednesday, July ??]] at our studio in East Williamsburg, Brooklyn.**
+  * **Our next [[https://partiful.com/e/U0A3JFfs0w9mgzynBcrt|Folk open house is on the night of Saturday, July 27]] at our studio in East Williamsburg, Brooklyn.**
   * Getting 3D calibration to work well on all systems, then merging it.......
-  * More work on new parallel evaluator
+  * Figuring out retraction and transaction behavior for parallel evaluator; testing it more
-  * Gadget iteration
+  * Gadget iteration (need to fix swiveling, fix power issues, improve software)
+  * Video support
+  * Documentation (language reference, new kits and printable worksheets for setting up and learning the Folk language), open up GitHub repo
@@ Line 153: / Line 211: @@
   * [[https://spectrum.ieee.org/how-e-ink-developed-full-color-epaper|How E-Ink Developed Full Color E-paper]] - a great breakdown from IEEE
   * [[https://www.e-ink-info.com/tags/transparency|printable mini e-ink displays with their own batteries]]
+  * [[https://www.instagram.com/reel/C8a3g27ppHM/|The first few art objects in this video]] remind me of large, polished, but simplistic versions of Kevin Kwok's e-ink crank
+  * [[https://www.instagram.com/reel/C7Vu5sDOzHj/?igsh=MTgwOWo2bjVrNzNjcA==|Fascinating multi-colored stamp]] — watching this over and over, it’s inspiring a bunch of ideas around hand stamped fiducial tools :)