Differences

This shows you the differences between two versions of the page.

--- newsletters:2026-07 [2026/08/16 05:09] – osnr
+++ newsletters:2026-07 [2026/08/18 17:21] (current) – [Correction experiments] osnr
@@ Line 1: / Line 1: @@
 ====== July 2026 newsletter ======
-TODO:
-  * [ ] Schedule Substack to send
-  * [ ] Address TODOs in the body of the July Dokuwiki document.
 {{htmlmetatags>metatag-og:title=(Folk Computer July 2026 newsletter)
 metatag-media-og:image=(newsletters:hue_spinner_to_outline_color.gif)
-metatag-og:description=(SAM2 multiple object tracking; blending and more color spinner; RFID localization; autoexposure; Oana Olteanu's piece)}}
+metatag-og:description=(SAM2 multiple object tracking; blending and more color spinner; RFID localization; autoexposure; Oana Olteanu's piece)
+}}
 Our next Folk open house will be in the evening on **[[https://luma.com/91fmi1b2|Wednesday, August 26]]**, at our studio in Williamsburg, Brooklyn.
@@ Line 66: / Line 62: @@
     * This program prints with a red circle in its center, for example:
       * {{newsletters:img_7254.jpeg?300px}}
-  * Andrés: TODO
 === Shapes performance fix ===
@@ Line 123: / Line 118: @@
 Contour tracing is a useful primitive (it's how you go from image to queryable/collidable geometry), but it's very dependent on lighting conditions, it doesn't give you contour/object recognition and identity across frames, and the contours are distorted based on camera position.
+=== New viewport ===
+Last month, we worked on a dynamic viewport where you move the two corners (gold) to resize a larger-than-page-sized viewport (green):
+{{newsletters:img_6439.jpeg?200px}}
+This dynamic viewport has issues with actually getting it on the right plane so you can accurately projection-map it. (Do you take the plane from one of the corner programs? They'll have subtly different planes, like a couple millimeters variation in pose between the two of them. Do you average the corner programs? Do you try to find a plane underneath both? Do you preregister the table plane at calibration time and downproject onto it?)
+{{newsletters:screenshot-2026-07-10-at-12.56.17-pm.png?300px}}
+This month, we've been prototyping with a big acrylic viewport with tag in corner and manually enlarged geometry (on left).
+{{.:pasted:20260818-170321.jpeg?300px}}
+This works pretty well and is about the right size that Karl wants. We should be able to go back to the old dynamic viewport later, but punting on it for now.
 === SAM2 + contour tracing ===
@@ Line 219: / Line 230: @@
 This is a pretty high-level API, although you still need to think about camera images/frames/slices and pixel locations (mostly because a camera frame also reflects a specific time that your prompt is from). But it's nice that it hides most of the track loop from you and lets you think in terms of objects semantically.
+(We made APIs on top that you can invoke that give you contour or foot contour or center, which have more geometric information than the mask. You could imagine ultimately getting a rough quad for the object, so you can projection map it or do points-at / other queries, or maybe we generalize those so they work with contours.)
 === Physical interface for multiple object registration ===
@@ Line 244: / Line 257: @@
 {{.:pasted:20260813-162034.png?400px}}
+All these pages have a 1-1 correspondence between registration page and object. You need to print multiple registration pages to track multiple objects in parallel. We might change this soon. (It is nice because you can just make the object ID from the registration page ID and point to the registration page to clear/hold state.)
 === Web UIs ===
 We have a few interesting web debugging UIs for SAM2 tracking.
-SAM2 per-object mask views:
+SAM2 per-object mask views, so you can see what SAM2 has locked onto if anything:
 {{newsletters:screenshot-2026-07-10-at-3.34.23-pm.png?300px}}
-Camera slice /camera-slice/VIEWPORT view -- this is what the camera sees, rewarped to make the viewport rectangular:
+Camera slice /camera-slice/ID view -- this is what the camera sees, rewarped to make the object ID you're slicing rectangular and axis-aligned (two slices side-by-side here: the viewport and a blank page inside the viewport):
-{{newsletters:screenshot-2026-07-10-at-3.56.20-pm.png?300px}}
 {{newsletters:screenshot-2026-07-24-at-2.50.55-pm.png?600px}}
@@ Line 283: / Line 294: @@
 == Camera-facing arc experiment ==
-Omar had the idea that we just look at where the object is relative to the camera and exclude the side that's likely to be distorted.
+Omar had the idea that we just look at where the object is relative to the camera and exclude the side that's likely to be distorted. Here, the camera is above and to the right of the viewport, so we cast rays from the camera to the convex hull of the contour of the object and take that green arc between the two tangent points:
 {{newsletters:screenshot-2026-07-24-at-5.11.38-am.png?400px}}
 {{newsletters:img_6973.mp4?350px}}
+That should get us the 'side' of the contour that is touching the viewport plane (should be unaffected by moving the cup further away or to different sides of the camera, unlike just finding centroid of contour), and then we fit a circle of the right diameter to it? More on this next month.
 === Issues ===
@@ Line 299: / Line 312: @@
 The fix was to [[https://github.com/FolkComputer/folk/commit/b8673265396fda41fdab937f72b243fc4f299176|do a more sophisticated rewarp when making the camera slice.]]
+(I honestly think this makes a lot of methods we tried and discarded earlier more viable.)
-== Feedback issue ==
+== Projector feedback issue ==
 One issue we keep running into with SAM2 is that if we project onto the tracked object, the tracker locks onto the projection as well as the actual object. You get both fun and annoying feedback effects:
@@ Line 326: / Line 341: @@
   * [[https://github.com/FolkComputer/folk/commit/59d22b1bfcd84cf0d6f15e1cc7e4c078ebc66987|Send mask returned from SAM2 as bytestring]] (saves 30ms on the 90ms SAM2 inference) instead of string list. [[https://github.com/FolkComputer/folk/commit/a8b7eead675cd40c4d4bbad4847a7c4bbf6ee785|Fixed mask encoding/decoding bugs]] when shipping data between Folk and Python
   * Python bug fixes:
-    * old tracking sticking around:
+    * Old tracker state sticking around when you try to track a new object:
       * {{youtube>lDSfOJQz6-8?}}
-    * threads sticking around. twisted. memory leaks
+      * [[https://github.com/FolkComputer/folk/commit/84bf8a64eb7a03ec657c9b00e8a517892e6f5c9e|Fixed by fixing encoding of object IDs when sending back to Folk!]]
+    * Python threads sticking around; huge memory leaks
       * {{.:pasted:20260803-012250.png?500px}}
-      * These are all actually from the same Python process, just multiple threads. It's taking like 30GB of RAM. This was because of a leak in the predictor class where we were just storing every frame forever.
+      * These are all actually from the same Python process, just multiple threads. It's taking like 30GB of RAM. This was because of a [[https://github.com/FolkComputer/folk/commit/601be98085134acdb6c5393a5fa84a20f92b5c60|leak in the predictor class where we were just storing every frame forever.]]
         * (Our predictor is sort of hacked together -- real-time support isn't native in SAM2, so I ripped off another repo)
@@ Line 390: / Line 406: @@
 [[https://www.motivenotes.ai/|Oana Olteanu]] (friend of [[https://janpaulposma.nl/|JP Posma]]) visited last month and very kindly wrote up her thoughts from the visit: https://www.motivenotes.ai/p/against-usefulness
-HN
+It got to the front page of Hacker News as well: [[https://news.ycombinator.com/item?id=48882956|https://news.ycombinator.com/item?id=48882956]]
 === Visitors ===
@@ Line 414: / Line 431: @@
   * Omar: DrawTalking / object tracking demos using SAM2 (need to build applications around the tracking capability that we have); go from distorted contours to centers or other simpler useful geometric info; optical flow for better response time
   * Omar: More stability and performance fixes (SAM2 Python pileup, editor slowness)
-  * Andrés: TODO
+  * Andrés: Building a kit for people to try out making Folk desk lamps! Feel free to ([[https://bsky.app/profile/folkcomputer.bsky.social|DM on Bluesky or]] [[mailto:[email protected]|email us]] if you'd like to talk about purchasing a kit)
+  * Andrés: Browser! Or some sort of HTML renderer (I'm currently imagining: ''%%Wish $this loads the webpage https://example.com with scroll position 10% in a viewport 8.5" wide by 11"%%'')
+  * Andrés: Pong, with [[https://folk.computer/newsletters/2026-07?s[]=probe#:~:text=Omar%20and%20Josh%20implemented%20%27%27imageLib%20get%27%27%20(originally%20called%20probe%20and%20printed%20separately)|the imageLib-get/visual probe tools]], pong you play with paper or your hands!
+  * Andrés: Revisiting audio pipelining (see: Pipewire) and adding video via ''%%Wish $this plays video ...%%''
 ===== Links we've enjoyed =====
@@ Line 423: / Line 443: @@
   * [[https://dynamicland.org/archive/2015/Laser_pen_cap|Laser pen cap]]:
     * {{newsletters:original_9ae1c33379892f9635fc21b62407356a.gif?120px}}
+  * [[https://dynamicland.org/archive/2015/Laser_pen_cap|Laser pen cap - Dynamicland archive]]
   * [[https://youtube.com/watch?v=JM0PU0z6nyM&is=YXpMu-MVI-C2WzNU|Making my dream city builder: Part 1 - Parcels]] (Video, 27:20)
   * [[https://youtu.be/4heM4MqK82w|How I released a game that has no assets]] (Video, 33:11)
   * [[https://youtu.be/Qr3VsZYQy4s|The Nintendo DS Could See You Close It - and These Games Got Weird About It (Phantom Hourglass)]] (Video, 5:58)
   * [[https://youtube.com/watch?v=kYLJLJkVfLk&is=Ap5x-ViuGgyZutJj|Making an SNES Game the Way Nintendo Intended]] (Video, 25:30)
-  * [[6_attachment-dl2015-08-7d2e11.jpg|attachment-DL2015-08-7d2e11]]
+  * [[https://tashian.com/articles/dynamicland/|At Dynamicland, The Building Is The Computer — Carl Tashian]]
-  * [[https://tashian.com/articles/dynamicland/|At Dynamicland, The Building Is The Computer — Carl Tashian]] (Web, ~6500 words||26:00)
-  * [[https://dynamicland.org/archive/2015/Laser_pen_cap|Laser pen cap - Dynamicland archive]] (Web, 55 words||0:13)
 ==== Omar ====