Differences

This shows you the differences between two versions of the page.

--- newsletters:2024-06 [2024/06/30 05:05] – [Real-world/3D calibration] osnr
+++ newsletters:2024-06 [2024/07/01 08:37] (current) – osnr
@@ Line 1: / Line 1: @@
-====== June 2024 newsletter (WIP) ======
+====== June 2024 newsletter ======
 ===== What we've been up to =====
@@ Line 36: / Line 36: @@
   - to build much more accurate tracking in general -- millimeter precision -- to enable new applications where you can highlight individual words or facets of objects, and
   - to track coordinates in real-world meters instead of arbitrary projector or camera pixels, so we can start to integrate multiple projectors, cameras, and other real-world sensors and actuators (phone localization, RFID, CNC machine bed and material, spatial audio...), which will all benefit from having a shared ground coordinate system
+This video (running the osnr/checkerboard-calibration branch on my home system) shows how you can accurately measure the distance between 2 pages in real centimeters, as well as how accurate the 3D projection map can get right now:
+{{newsletters:img_9458.mp4?400px}}
+The distance is being displayed by this sort of hacked-together program using the new concept of ''quad'', which we may change/simplify/wrap:
+<code>
+When 19 has quad /a/ & 13 has quad /b/ {
+  set a [quad change $a "/dev/video4"]
+  set b [quad change $b "/dev/video4"]
+  Wish to draw text with x 800 y 400 text [format {%.2f} [* 100 [norm [sub [quad right $a] [quad left $b]]]]]cm radians 3.14
+}
+</code>
 This past month:
@@ Line 46: / Line 60: @@
     * Block out skew parameter of cam/proj intrinsics, can assume that it is always 0 (OpenCV does this so we copy them and assume it's OK)
     * Instead of averaging extrinsics (3D cam->proj rotation and translation) derived from each pose, which doesn't make sense (you can't average rotations), we accumulate a list of point pairs from camera->projector space and then use Kabsch algorithm to get a rotation and translation estimate, then do the nonlinear refinement mentioned above
-    * [[https://social.omar.website/@omar/statuses/01J0F5H0GW4HMG3G6B5RR0TQ9T|"Virtual visual servoing"]] pose estimation. Almost completely fixes weird flicker issues! This was surprisingly huge, and it's a really simple technique
+    * [[https://social.omar.website/@omar/statuses/01J0F5H0GW4HMG3G6B5RR0TQ9T|"Virtual visual servoing"]] pose estimation. Almost completely fixes flicker issues with 3D poses! This was surprisingly huge, and it's a really simple technique
+      * Keeps a tiny bit of state (pose from prev frame) and finds the most likely pose for current frame using tag detection + prev state
+      * Can get stuck in weird poses but fixes itself if you cover & uncover tag, this is tolerable for now I think
+      * I really like that this is both pose estimation and 'smoothing' at the same time, and it has some intrinsic model that this is a thing moving around in space that we're trying to estimate -- wonder if same technique could work for RFID
+        * Wonder if we could also teach it about the square tag instead of doing servoing for each point on its own, would that make it more robust
     * Spent a while on performance micro-optimization, went from like 300ms latency to 200ms. Didn't really help on folk0 (and folk-live at my apartment is not that much faster even though its calibration is way better), so ruling out the performance theory of why folk0 doesn't work as well
     * Spent a while on making a separate correct-extrinsics step that you do at the end of calibration, where you leave the board flat on your table and it reprojects tags and measures the offset and tries to correct the extrinsics:
@@ Line 61: / Line 79: @@
-D calibration works extremely well (1-2mm error) on my system in my apartment. Notice how the outline is almost perfectly aligned with the page:
+D calibration works extremely well (1-2mm error) on my personal system in my apartment, as you saw in the video above.
-{{newsletters:img_9350.mp4?300px}}
 It works much less well (1-2cm off) on folk0 at Hex House, and even worse on folk-beads (the cart system at Hex House), where the high distortion and shallow 3D volume of the ultra-short-throw projector make it really hard to calibrate.
@@ Line 69: / Line 85: @@
 I've also made it almost fully backward-compatible with all existing code -- it now derives 2D projector-space region claims from the 3D calibration, so all code that uses regions still works. I deleted all the old 2D calibration and homography and region code (calibrate.tcl), so 3D calibration backs everything on the branch.
-I'd like to get it to at least be on par with the existing 2D calibration in most cases before we merge it, and we're not there yet...
+I'd like to get it to at least be on par with the existing 2D calibration in most cases before we merge it, and we're not there yet... (working theory is still that the individual intrinsics of camera and projector are derived pretty well, but the extrinsics seem off)
 ----
@@ Line 112: / Line 128: @@
 ==== Portable gadget ====
-Omar has done a couple [[https://x.com/rsnous/status/1803139935299326152|new iterations]] of the portable gadget from last month (including picking up more CAD software, which is surprisingly inspirational from a UI point of view). Here's what it looks like right now:
+Omar has done a couple [[https://x.com/rsnous/status/1803139935299326152|new iterations]] of the portable gadget from last month (including picking up more CAD software, which as [[https://social.omar.website/@omar/statuses/01J09P0XC14M5TBZTHTRJWPHRA|surprisingly inspirational UI]]). Here's what it looks like right now:
 {{newsletters:img_9297.jpeg?300px}} {{newsletters:img_9291.jpeg?300px}} {{newsletters:img_9299.jpeg?300px}}
-The hand grip and front look good, and the projector and camera and Pi 4 fit well. The hand grip is a little loose, so need rubber or a groove to keep it from swiveling around. The back / wiring isn't fully figured out yet; that also will need a lid/cap that you screw in, I think. I still need to design and wire and do the trigger button, which I think will make a big difference to how you can interact with it.
+The hand grip and front look good, and the projector and camera and Pi 4 fit well. The hand grip is a little loose, so need rubber or a groove to keep it from swiveling around. (The Ultimems projector inside also swivels around and probably also needs rubber or a groove.) The back / wiring isn't fully figured out yet; that also will need a lid/cap that you screw in, I think. I still need to design and wire and do the trigger button, which I think will make a big difference to how you can interact with it.
 I also have issues keeping it powered right now (and I want to figure out a USB power pack situation anyway, where you can choose to use wall power or a power pack). I wonder if the projector is too much for the Pi 4's USB port to power, or if it's overheating, or something.
@@ Line 168: / Line 184: @@
 ===== What we'll be up to in July =====
-  * **Our next [[https://partiful.com/e/U0A3JFfs0w9mgzynBcrt|Folk open house is in the afternoon of Tuesday, July 30th]] at our studio in East Williamsburg, Brooklyn.**
+  * **Our next [[https://partiful.com/e/U0A3JFfs0w9mgzynBcrt|Folk open house is on the night of Saturday, July 27]] at our studio in East Williamsburg, Brooklyn.**
   * Getting 3D calibration to work well on all systems, then merging it.......
   * Figuring out retraction and transaction behavior for parallel evaluator; testing it more
   * Gadget iteration (need to fix swiveling, fix power issues, improve software)
   * Video support
-  * Documentation (language reference, new kits and printable worksheets for setting up and learning the Folk language)
+  * Documentation (language reference, new kits and printable worksheets for setting up and learning the Folk language), open up GitHub repo
@@ Line 196: / Line 212: @@
   * [[https://www.e-ink-info.com/tags/transparency|printable mini e-ink displays with their own batteries]]
   * [[https://www.instagram.com/reel/C8a3g27ppHM/|The first few art objects in this video]] remind me of large, polished, but simplistic versions of Kevin Kwok's e-ink crank
+  * [[https://www.instagram.com/reel/C7Vu5sDOzHj/?igsh=MTgwOWo2bjVrNzNjcA==|Fascinating multi-colored stamp]] — watching this over and over, it’s inspiring a bunch of ideas around hand stamped fiducial tools :)