Differences

This shows you the differences between two versions of the page.

--- newsletters:2026-04 [2026/05/04 23:37] – [April 2026 newsletter (WIP)] osnr
+++ newsletters:2026-04 [2026/05/05 20:08] (current) – [NaN handling] osnr
@@ Line 1: / Line 1: @@
-====== April 2026 newsletter (WIP) ======
+====== April 2026 newsletter ======
-{{htmlmetatags>metatag-og:title=(Folk Computer March 2026 newsletter)
+{{htmlmetatags>metatag-og:title=(Folk Computer April 2026 newsletter)
-metatag-media-og:image=(newsletters:pasted:20260311-191224.jpeg)
+metatag-media-og:image=(newsletters:img_3690.jpeg)
 metatag-og:description=(More new studio setup (new 4K projector, new L area); multi-display and multi-camera; calibration improvements)
 }}
@@ Line 11: / Line 11: @@
-We'll be giving a talk at SVA this Friday, May 8:
+We'll also be giving a talk at School of Visual Arts in Manhattan on this coming Friday, May 8:
 [[{}https://luma.com/a93vdm38?tk=4M4Vnb|Talk at SVA on Friday, May 8]]
 ===== What we've been up to =====
+==== General system improvements ====
+  * [[https://github.com/sorig|Esben Sørig]] added [[https://github.com/FolkComputer/folk/pull/253|support for tangential distortion parameters in camera calibration]], which makes the calibration much more accurate on his system (has an ELP camera + lens)
+  * Andrés [[https://github.com/FolkComputer/folk/pull/257|brought back tag masking]] so projections don't interfere with tag detection
+    * {{.:pasted:20260428-173111.png?250px}}
+  * Andrés [[https://github.com/FolkComputer/folk/pull/256|fixed support for left-margined, right-margined, footnoted]] ways to draw text on margins of a program
+  * Andrés [[https://github.com/FolkComputer/folk/pull/254|fixed transparency in compositing]]
+  * Omar improved the new logging system:
+    * Put all logs for a given folk PID in /var/tmp/folk-PID subfolder
+    * Report web system error at console since without working web page, hard to debug failures
+    * Fix various cases where we weren't capturing output into stdout or stderr (library output, subprocesses) by turning off FORTIFY_SOURCE; made OTHER.stdout and OTHER.stderr as catch-alls when we can't do proper redirection (as opposed to going to /dev/null)
+  * Omar improved the in-system build of libapriltag and libwslay (moved them out of Makefile last month) so they rebuild on any change in the builtin programs, don't require patchelf hack, and hold all dependent programs back until actually compiled
+=== Bluetooth keyboard support ===
+Omar and Andrés started working on new Bluetooth keyboard support, where we don't need a custom udev device path in ''by-path'' -- ideally, we could pair and reconnect a keyboard just from the table (and ideally completely automatically if already paired).
+{{newsletters:c0cd1dcc-f626-46b0-9f07-f4dee96c0298-7942-00000d891e1303c7.png?300px}}
+=== Multi-display / multi-camera support ===
+Omar: The new Folk setup at the studio has this L layout where we want at least 2 projectors and 2 cameras connected to the same PC, so I've been working on refactoring Folk so that one Folk system can calibrate and drive all that hardware simultaneously. (until now, every Folk system has had exactly 1 active projector and 1 active camera)
+While at it, I also wanted to improve support for these types of ELP stereo cameras, which we have been using in the gadget (but are not really using the stereo yet). Trying to use it in the studio Folk setup as well.
+{{newsletters:img_4134.jpeg?200px}}
+Given this stereo camera, how can we add crop support to the UI and ultimately calibrate and use both 'eyes' to do better pose estimation?
+=== Easier camera config ===
+Added a setting on /camera that lets you set the exposure (and auto-exposure mode) live right there. Also now iframe /camera on the calibration page, so we have just that one uniform UI for camera config (including any future settings we might want to add), rather than /calibrate having its own one-off exposure slider.
+=== Camera crop ===
+I added a camera crop UI to the Folk /setup interface. You create a "virtual cropped camera", which makes a green rectangle that lets you select an area to crop a camera to. The virtual camera then exists with that cropped area, and you can calibrate, track tags, etc on that camera:
+{{.:pasted:20260505-143048.png?400px}}
+(We use this to pick just the left eye for now out of a stereo side-by-side ELP image, but eventually want to use this to create two virtual cameras, one from each eye, to co-calibrate.)
+=== GPU refactor for multiple displays ===
+Omar: [[https://github.com/FolkComputer/folk/pull/255|Did a big refactor]] of the gpu/draw.folk and gpu/gpu.folk and other code -- split out most of the pipeline compilation logic and made the draw library not dependent on the specific display and display mode, so we can reuse pipelines to draw on any display & have multiple displays active at once.
+Now when you boot Folk on a PC with two displays, we can draw to both, but it's still the same stuff drawn to both. In order to have a properly multi-display Folk system, we need to change a lot of the draw statements to be scoped to a specific display, which may be a breaking API change.
+==== Ideas for fixing pose estimation errors ====
+Some systems have [[https://github.com/AprilRobotics/apriltag/issues/71|pose estimation problems]] while we rely on monocular vision. One solution is to use a stereo camera to remove ambiguity; another is to account for the table plane in some way to narrow down candidates.
+Here is a simple proposal:
+{{newsletters:screenshot-2026-04-24-at-5.10.07-pm.png?400px}}
 ==== New studio setup ====
+Last month, we started [[newsletters/2026-03#moving-into-the-new-studio|moving]] out of Hex House and into a new studio space in Williamsburg. We continued that process this month and finalized a lot of the layout (although we don't necessarily have final mounting situation and tables/storage yet).
 === L-shaped dedicated area for Folk ===
@@ Line 30: / Line 90: @@
 Was trying the Logitech Brio 4K on one side and NexiGo on the other side at first. Horrible issues with USB3; ultimately gave up on the Brio 4K for now.
+{{newsletters:img_4149.mp4?250px}}
 {{.:pasted:20260504-222528.png?350px}}
-(I think it's the SAM2 demo that's running/rerunning all the time and really pushing the Nvidia drivers.)
+(I think it might be the SAM2 demo I have on the table that's running/rerunning all the time and really pushing the Nvidia drivers and causing deadlock.)
 === Communal table (folk-convivial) / JMGO N1 Ultra setup ===
@@ Line 50: / Line 113: @@
 It's not as light as the AAXA 4K1 (4ish pounds), but it's similar price and much, much brighter.
-C-stand setup
+Set it up on a tall C-stand (8-10 feet) for now, and wrapped a lampshade around it (see the head on table in second-to-last photo). C-stands have been useful for temporary mounting of heavy-ish stuff, but ultimately, should probably suspend from ceiling to get it totally out of the way.
-{{newsletters:img_3640.jpeg?0x400px}}
+{{newsletters:img_3640.jpeg?0x300px}} {{newsletters:img_3613.jpeg?0x300px}} {{newsletters:img_4150.jpeg?0x300px}} {{newsletters:img_4162.jpeg?0x300px}} {{newsletters:img_4152.jpeg?0x300px}}
-{{newsletters:img_4150.jpeg?0x300px}} {{newsletters:img_4152.jpeg?0x300px}}
 === Moving the CNC machine ===
@@ Line 71: / Line 132: @@
 ==== SVA setup ====
-TODO
+  * Andrés set up a small Folk system in the library of the SVA MFA Interaction Design department for students in their UI/UX Design History class to try out, here's Andrés walking a student through creating her first Folk program:
-==== General system improvements ====
+    * {{:newsletters:sva_demo_shortened.mp4?400px|SVA demo — making a simple "Wish $this is outlined green" program via the web editor}}
+    * {{newsletters:sva_small_demo_pic.jpeg?400px|A blue outlined program on the Folk Computer shelf at SVA}}
-  * TODO
-  * Andrés [[https://github.com/FolkComputer/folk/pull/257|brought back tag masking]] so projections don't interfere with tag detection
-    * {{.:pasted:20260428-173111.png?250px}}
-  * Andrés [[https://github.com/FolkComputer/folk/pull/256|fixed support for left-margined, right-margined, footnoted]] ways to draw text on margins of a program
-  * Andrés [[https://github.com/FolkComputer/folk/pull/254|fixed transparency in compositing]]
-  * Omar improved the new logging system:
-    * Put all logs for a given folk PID in /var/tmp/folk-PID subfolder
-    * Report web system error at console since without working web page, hard to debug failures
-    * Fix various cases where we weren't capturing output into stdout or stderr (library output, subprocesses) by turning off FORTIFY_SOURCE; made OTHER.stdout and OTHER.stderr as catch-alls when we can't do proper redirection (as opposed to going to /dev/null)
+==== Interactively refine calibration ====
-=== Bluetooth keyboard support ===
-Omar started working on new Bluetooth keyboard support where we don't need a custom udev device path in ''by-path''
-=== Multi-display / multi-camera support ===
-Omar: The new Folk setup at the studio has this L layout where we want at least 2 projectors and 2 cameras connected to the same PC, so I've been working on refactoring Folk so that one Folk system can calibrate and drive all that hardware simultaneously. (until now, every Folk system has had exactly 1 active projector and 1 active camera)
-While at it, I also wanted to improve support for these types of ELP stereo cameras, which we have been using in the gadget (but are not really using the stereo yet). Trying to use it in the studio Folk setup as well.
-{{newsletters:img_4134.jpeg?200px}}
-Given this stereo camera, how can we add crop support to the UI and ultimately calibrate and use both 'eyes' to do better pose estimation?
-=== Camera crop ===
-I added a camera crop UI to the Folk /setup interface. You create a "virtual cropped camera", which makes a green rectangle that lets you select an area to crop a camera to. The virtual camera then exists with that cropped area, and you can calibrate, track tags, etc on that camera:
-{{newsletters:01827c1c-c07b-4c75-9f7d-f440d13fdd06-7942-00000c69fe151596.png?0x250px}} {{newsletters:b90d72df-2be1-410a-a7ca-9f6127a8a3d1-7942-00000c6a066d002c.png?0x250px}}
-(We use this to pick just the left eye for now out of a stereo side-by-side ELP image, but eventually want to use this to create virtual cameras out of both eyes to co-calibrate.)
-=== Easier camera config ===
-Added a setting on /camera that lets you set the exposure (and auto-exposure mode) live right there. Also now iframe /camera on the calibration page, so we have just that one uniform UI for camera config (including any future settings we might want to add), rather than /calibrate having its own one-off exposure slider.
-=== GPU refactor for multiple displays ===
-=== Interactively refine calibration ===
 Omar: This is still on a branch, but hasn't been that successful an experiment. Will try a few more things. The goal here is to get calibrate to 1. work better on various cameras we have (ELP stereo camera has a lot of distortion) and 2. be easier to fixup later if you bump the camera.
@@ Line 121: / Line 142: @@
 (Usually, in my experience, calibration quality rides on the specific camera you're using. Projectors are usually easy-ish to calibrate [except ultra-short-throw], while cameras have more distortion and variability. My home system pretty much always calibrates perfectly -- the Brio 4K was mediocre, and the NexiGo has been good. The ELP is also mediocre.)
-== NaN handling ==
+=== NaN handling ===
 People regularly have this silent failure when calibrating, where there's not enough variance in their poses (they kept the board flat on the table the whole time), and the optimization and/or refinement fails to converge:
@@ Line 127: / Line 148: @@
 {{newsletters:screenshot-2026-04-29-at-6.04.12-pm.png?500px}}
+Now capping the number of optimization iterations and throwing an error when NaN appears in any parameter. Still need to report to frontend UI.
-== Live preview on /calibrate ==
+=== Inline live preview on /calibrate ===
 Made preview on /calibrate a side panel: reduces the length of the page / amount of stuff on the critical path that you read, makes it easier to see what's going on as you calibrate and refine and adjust exposure in real-time.
@@ Line 150: / Line 171: @@
 ===== What we'll be up to in April =====
-  * Our next [[https://luma.com/gmcw2fhy|RSVP for the Folk open house on Thursday, May 28]], at our **new studio in Williamsburg**
+  * [[https://luma.com/a93vdm38?tk=4M4Vnb|We'll be giving a talk at SVA on Friday, May 8]].
-  * [[https://luma.com/a93vdm38?tk=4M4Vnb|We'll be giving a talk at SVA on Friday, May 8]]
+  * Our next [[https://luma.com/gmcw2fhy|RSVP for the Folk open house on Thursday, May 28]], at our new studio in Williamsburg.
+  * Omar: I'll be trying to improve calibration of our ELP camera & working on the Folk gadget.
 ===== Links we've enjoyed =====