Differences

This shows you the differences between two versions of the page.

--- newsletters:2023-11 [2023/11/30 20:52] – Add Andrés' links admin
+++ newsletters:2023-11 [2023/11/30 22:48] (current) – Remove Govind TODO, constrain keyboard GIF admin
@@ Line 20: / Line 20: @@
     * {{newsletters:pxl_20231103_191517745.ts.mp4?150}} {{newsletters:pxl_20231107_213647133.ts.mp4?450}}
     * {{:newsletters:pasted:20231130-161747.png?350}}
-  * TODO: Govind?
 ==== Tabletop / in-system editor ====
@@ Line 30: / Line 28: @@
   * Andrés working on multi-keyboard, moving keyboard into virtual program
     * (we'll finally be able to dump tcl-thread and pi.tcl and have everything in virtual programs and subprocesses and statement communication!)
+    * Here's a preview of it monitoring key presses from multiple keyboards plugged into %%folk0%%:
+    * {{:newsletters:pasted:20231130-223928.gif?350}}
 ==== Calibration and CNC ====
@@ Line 37: / Line 37: @@
 (the material might be an inch or two thick, so you need to be able to projection-map 3D surfaces above the base plane of the Folk system, and you need high accuracy -- current Folk is only calibrated to a single plane homography and is often 1cm off or worse)
-Basically, you need to hold up some known pattern in a few different poses and project onto it, and that lets you characterize the behavior of the camera and projector in 3D space. Most people say that a pretty simple pinhole-camera (linear) model with some adjustment for radial distortion is sufficient for better-than-a-pixel accuracy.
+Basically, in order to do 3D calibration, you need to hold up some known pattern in a few different poses and project onto it, and that lets you characterize the behavior of the camera and projector in 3D space. Most people say that a pretty simple pinhole-camera (linear) model with some adjustment for radial distortion is sufficient for better-than-a-pixel accuracy.
-Using AprilTags for the projected pattern is nice, because we already link the AprilTag library (so no need for more custom computer vsiion), and because they're self-verifying and error-correcting and give you a lot of information -- if you see an AprilTag, you know it was the exact AprilTag you tried to project, and you know its identity and corners and angle, as opposed to projecting stripes or something where you need your own thresholds and classification to figure out what stripes you're seeing (if any).
+Using AprilTags for the projected calibration pattern is nice, because we already link the AprilTag library (so no need for more custom computer vision), and because they're self-verifying and error-correcting and give you a lot of information -- if you see an AprilTag, you know it was the exact AprilTag you tried to project, and you know its identity and corners and angle, as opposed to projecting stripes or something where you need your own thresholds and classification to figure out what stripes you're seeing (if any).
-  * Early November: the first 3D calibration test/demo, using a calibration pattern based on [[http://www.ok.sc.e.titech.ac.jp/res/PCS/research/procamcalib/|Audet (2009)]] but with AprilTags and printing on the outer perimeter instead of alternating
+  * Early November: the first 3D calibration test/demo, using a calibration pattern based on [[http://www.ok.sc.e.titech.ac.jp/res/PCS/research/procamcalib/|Audet (2009)]] but using AprilTag fiducials and printing on the outer perimeter instead of alternating even/odd
     * {{:newsletters:pasted:20231130-183930.png?0x300}}
+    * It's useful to leave gaps in the printed and projected patterns so it's easy to move the board & get the tags 'aligned enough' that the system can see at least 1 projected + 1 printed tag on the same plane, which is then enough information to properly rewarp all the projected tags to fit into the calibration board.
+    * Then once you can see most of the projected + printed tags on the board, you record that as a set of example points for calibration (you now have both board-plane coordinates in meters and camera and/or projector coordinates in pixels for all corners of all those tags)
     * Notice how tags in the center of the calibration board are projected dynamically, while tags on the outer perimeter are printed:
-    * {{:newsletters:pasted:20231106-180904.jpeg?250}} {{newsletters:img_1095.mp4?250}}
+      * {{:newsletters:pasted:20231106-180904.jpeg?250}} {{newsletters:img_1095.mp4?250}}
+    * {{:newsletters:pasted:20231122-164229.png?350}}
-  * {{:newsletters:pasted:20231122-164229.png?350}}
+  * Mid-November: We were running into a problem: how do we know when the rewarped projected tags have actually rewarped? there's some latency (tens of milliseconds? hundreds of milliseconds?) from when Folk rewarps the tags to fit the board and when that rewarp actually appears in the webcam feed. If you read the tags too early and do another rewarp based on that, the tags become unreadable and you have to abort calibration. And you obviously don't want to just have a fixed-length sleep (which is what we did at first), you want to dynamically adapt to whatever latency happens to be present.
+    * So we introduced a concept of //versions//, where we rotate half the projected tags by 90 degrees every rewarp, so we can tell if the rewarp has gone through by the rotations of the detected tags
-  * Mid-November: we started running into a problem: how do we know when the rewarped projected tags have actually rewarped? there's some latency (tens of milliseconds? hundreds of milliseconds?) from when Folk rewarps the tags to fit the board and when that rewarp actually appears in the webcam feed. So we introduced a concept of //versions//, where we rotate half the projected tags by 90 degrees every rewarp, so we can tell if the rewarp has gone through by the rotations of the detected tags
     * {{newsletters:img_1517.mp4?350}}
-  * Late November: Why was it projecting upside-down? Surprisingly intuitive answer
+  * Late November: We started testing the 3D calibration but found that tag outlines were weirdly offset -- on folk-convivial, they were a few inches off, and on my home system, they were actually flipped, where the outline would appear on the other side of the table and move in the opposite direction from the physical tag.
+    * Why was it projecting upside-down or offset? The answer is surprisingly cool and physically intuitive: I was using //camera frame coordinates// (x, y, z from the center of the camera lens) where I should have been using //projector frame coordinates// (x, y, z from the center of the projector lens), so I was projecting the outline as though the projector was where the camera is.
+    * And in folk-convivial, the projector was a few inches from the camera but oriented the same way, while on my home system, the projector was upside-down relative to the camera!
+    * Had to figure out extrinsic information during camera calibration and projector calibration (what is the rotation and translation of each pose relative to the projector & relative to the camera), then use those extrinsics to derive the rotation and translation of the projector frame relative to the camera frame ("stereo calibration")
-  * Late November: Finally got 3D calibration to the point of testing it!
+  * Late November: Finally got 3D calibration to the point of testing it and seeing the outlines align with the tags:
     * {{newsletters:img_1639.mp4}}
+    * The outline is still ~1cm off, but notice how it's the right size no matter what distance from the camera or skew I hold the tag up at. That's a 3D calibration :-) (and internally, coordinates here are all in meters from the center of the camera lens, too)
-  * Next: try to improve smoothness and accuracy by using nonlinear refinement & adding a radial distortion term to the claibration
+  * Next: try to improve smoothness and accuracy by using nonlinear refinement & adding a radial distortion term to the calibration
 I'm excited about real-world calibration because it feels like the first system change in a long time (6 months? a year?) that will actually bring in new modes of interaction, that will //look// different to the untrained eye from putting pieces of paper on a table and watching them light up. (This is also why the [[guides/button|button]] is great.)
@@ Line 74: / Line 80: @@
   * Naveen added [[https://github.com/FolkComputer/folk/pull/112|non-uniform (different in X and Y) image scaling]]
-  * Naveen added [[https://github.com/FolkComputer/folk/pull/119|mailbox clearing when a subprocess goes down due to unmatch]]
+  * Naveen added [[https://github.com/FolkComputer/folk/pull/119|mailbox clearing when a subprocess gets killed by unmatch]]
     * but **not** (yet) when subprocess goes down due to an internal crash (like a segfault) or due to arbitrary external Unix kill (we have this whole supervisor thing already that's meant to be able to waitpid on each subprocess and detect those conditions, we're just not using it)
     * see [[https://discord.com/channels/956758212152025098/956758650700046366/1175908063019008021|Discord discussion of mailbox clearing]]
@@ Line 163: / Line 169: @@
     * Fun interim on-a-wall stage during setup, before we pointed it down at the table:
       * {{:newsletters:pasted:20231130-161524.png?250}}
-      * (whenever I see a projection mapping demo, I always feel like we should be able to subsume it and make it really easy to set up and reprogram, like it was trivial to make the meow text here wiggle around over time)
+      * I think this image is a fun provocation because it's sort of //about the wall//, the drawing on the whiteboard, instead of being about the system. Imagine wheeling a system over in 10 seconds and pointing it at part of your environment because you want a thin layer of computation or sharing or dynamism on top of what's there (and not because you want a generic "Folk system" set up that could be set up anywhere)
+      * (btw, whenever I see a projection mapping demo, I always feel like we should be able to subsume it and make it really easy to set up and reprogram, like it was trivial to make the meow text here wiggle around over time)
   * Simplified install instructions to just use Vulkan binaries from apt, instead of compiling mesa from scratch
@@ Line 172: / Line 179: @@
       * Got a Logitech C922 camera for folk-beads: at least it can do 60fps at 720p, much smoother-feeling than the C920 or Azure Kinect which can only do 30fps at any resolution
+===== Hmm =====
+{{newsletters:pasted:20231130-213756.png?550|one observation re: how people pattern-match and overfit to the examples they see is we get a lot of questions and comments about the AprilTag black-and-white markers we use, even though I would say we view those as much less essential than the programming system and database
+and there are a lot of applications like CNC preview or various projection-mapping demos where I think you should be able to get lots of value out of using Folk w/o having any tags visible in the end interface at all}}
+(I think part of our job has to be to just broaden the examples so that fitting to them still gives you a pretty productive working set)
 ===== What we'll be up to in December =====
+  * **[[https://partiful.com/e/MJ6V5SfEKRPPjOskxeti|The next Folk open house]] will be on the evening of Wednesday, December 13, at our studio in East Williamsburg, Brooklyn.** :-)
   * Still need to complete RFID refactor. I need to make the ringbuffer faster and I need to wait on less data to start parsing ..