Differences

This shows you the differences between two versions of the page.

--- newsletters:2024-08 [2024/09/03 23:56] – [System improvements] osnr
+++ newsletters:2024-08 [2024/09/04 02:36] (current) – [3D calibration] osnr
@@ Line 1: / Line 1: @@
-====== August 2024 newsletter (WIP) ======
+====== August 2024 newsletter ======
-We’re looking to build up more small sponsors now that Folk is fully open-source (+ cover recent gaps in contracts/grants).
+(If you enjoy following our progress and want us to continue, definitely consider [[https://github.com/sponsors/FolkComputer|sponsoring Folk on GitHub Sponsors]].)
-If you enjoy following our progress and want us to continue, definitely consider [[https://github.com/sponsors/FolkComputer|sponsoring Folk on GitHub Sponsors]].
-(our first goal is to cover our studio rent with sponsorships!)
-----
 Some links, especially if you're new to Folk:
@@ Line 14: / Line 8: @@
   * [[https://folk.computer/pilot|Pilot page]]
   * [[https://forms.gle/eMGCkQjYRvxf8S2S6|Meet with Andrés (virtually, 10 minutes) about setting up your own Folk instance]]
-----
 ===== What we've been up to =====
@@ Line 27: / Line 19: @@
       * {{newsletters:img_0083.mp4?250px}}
     * (isn't close to finished yet, may even require new evaluator because it feels like latency and frame drops degrade the experience a lot, you expect it to keep up with iPhone scrolling. good push for performance though)
+  * [[https://danielpipk.in|Daniel Pipkin]] has been looking at using depth camera (Kinect + libfreenect2) to implement [[https://www.robertxiao.ca/research/direct/|touch detection]]
 ==== Friends and outreach ====
@@ Line 65: / Line 58: @@
 ==== System improvements ====
+  * Merged [[https://github.com/FolkComputer/folk/pull/161|Andrés's change from ''Commit'' to ''Hold'' in the basic Folk syntax]], which feels more appropriate to what it does
+    * Shows deprecated warning on programs that still use ''Commit''
+    * Also changed 3D calibration code accordingly
+  * Daniel and Andrés made some hotfixes because Ctrl-S and Ctrl-P were acting up on table keyboards
+  * Omar: [[https://github.com/FolkComputer/folk/commit/2ac150d912f44dbafb1faf64658289dcf314736e|Use 4 instead of 6 vertices in vkCmdDraw]], which fixes glitching on Pi gadget (and probably other random GPUs)
   * [[https://danielpipk.in|Daniel Pipkin]] wrote [[https://github.com/FolkComputer/folk/pull/169|an ESC/POS printing backend]] to support his [[https://epson.com/For-Work/POS-System-Devices/POS-Printers/TM-L90-Plus-Label-and-Barcode-Printer/p/C31C412602|Epson TM-L90 Plus receipt printer]]
     * (Omar: I've seen other ESC/POS printers around, so this should come in handy in general.)
     * Daniel's pull request is built on 3D calibration and provides geometry for the printed receipts and tags (since they're different sizes than our normal half-letter programs), so they should projection-map correctly.
     * {{:newsletters:pasted:20240902-034529.png?300px}}
+    * Should be able to merge soon (it was dependent on 3D calibration getting merged), just figuring out some edge cases with editing/overwriting existing programs
 === Documentation ===
@@ Line 97: / Line 96: @@
 ==== Handheld gadget ====
+Omar: An inventory of some of the gadget ideas I've been throwing around for the last few weeks as I've demoed it to people (some are general Folk ideas also, but they feel more vital and mobile with the handheld gadget):
+  * Trigger button to 'scan' QR code (or whatever other computer-readable thing) in environment, project content (linked PDF) next to it
+    * notice how other people can see it, it's more social, it's not just on your phone
+  * How to connect gadget to local Wi-Fi network? You carry a laminated card with fields for SSID and password + a fiducial marker, you fill out the fields with a pen, then scan the card with the Folk gadget to connect
+  * Fuller handwriting interaction where you write in a notebook and the gadget projects responses, you get like a 'REPL' or conversational interaction (you can program this way, or ask questions, or run commands)
+  * Receipt printer + gadget let you carry a complete system in a lunchbox and pull it out and cover a whole table with computational objects
+  * Segment parts of the world and projection-map them with colors or patterns automatically
+  * Highlight text in a book or printout that the gadget is pointed to
+  * Point gadget at your laptop screen, hold trigger, and drag objects (files, tabs, windows, photos, videos) out of your laptop and onto a Folk table as virtual objects (or into a receipt printer to make them physical objects)
+  * Gadget as flashlight that shows a 'spotlight' information augmenting the world at the world's scale, not at all like a screen (this requires spatial tracking to know how far the gadget is from the surface, so fiducials and/or world tracking, which is exactly what 3D calibration is meant to enable)
+    * want more brightness or higher pixel density? just move the gadget closer!
+We need to make a lot of progress on Folk and on the gadget design to be able to build these, but it's very exciting, and it does feel like the ideas are generated by having the gadget in hand. I think it's been well worth the effort of building it.
+=== Playbook ===
+Over the course of my week in San Francisco, I kind of developed a playbook of quick virtual demos that show some programmability and dynamism on the Folk gadget (although not as good as a full tabletop system with physical programs yet, obviously).
+It goes something like:
+  - Set up Wi-Fi on laptop, plug gadget in with huge extension cord: red calibration error appears on projection, so you know it's on
+  - Go to http://gadget-blue.local:4273/new; program ''Wish $this is outlined green'', and green rectangle shows on wall/table, bright enough to be visible
+  - Program ''Wish $this is labelled "Hello!"'' (or the person's name), text shows in the green rectangle
+  - Program scaled label (we should add sugar so this is less of a jump from previous): shows live programmability <code>
+When $this has region /r/ & the clock time is /t/ {
+  lassign [region centroid $r] x y
+  Wish to draw text with text "Hello" x $x y $y scale 3
+}
+</code>
+  - Program animating scaled label: shows movement, not a static image anymore <code>
+When $this has region /r/ & the clock time is /t/ {
+  lassign [region centroid $r] x y
+  Wish to draw text with text "Hello" x $x y $y scale [expr {abs(sin($t))*5.0}]
+}
+</code>
+  - Program camera feedback: shows that camera is active, gadget interacts with the outside world and isn't just a screen <code>
+When camera /any/ has frame /f/ at timestamp /ts/ {
+  Wish $this displays image $f
+}
+</code>
+  - Calibrate on http://gadget-blue.local:4273/calibrate (I carried a mini foldable cardboard calibration board with me with 14.5mm tags): shows the role of fiducial markers, shows how the gadget can be more like a 'flashlight' overlaying on the environment where it always projects at the same physical size
+    - {{newsletters:img_1588.mp4?250px}}
 === Hardware ===
@@ Line 104: / Line 148: @@
 And I got shorter cables with right angles and/or ribbons, which save space and remove strain (from cables being too long) that was causing a lot of random instability/crashes (HDMI and power would jostle out of the projector and/or Pi before, I think).
-{{:newsletters:pasted:20240902-011823.jpeg?250px}} {{:newsletters:pasted:20240902-011804.jpeg?250px}} {{:newsletters:pasted:20240902-012126.jpeg?250px}} {{:newsletters:pasted:20240902-012244.jpeg?250px}}
+{{:newsletters:pasted:20240902-011823.jpeg?200px}} {{:newsletters:pasted:20240902-011804.jpeg?200px}} {{:newsletters:pasted:20240902-012126.jpeg?200px}} {{:newsletters:pasted:20240902-012244.jpeg?200px}}
 Along with the current revision of the 3D-printed chassis, the gadget feels pretty stable in terms of hardware now, and it doesn't randomly crash (I think this is also thanks to the RAM watchdog), and the cabling is much simpler from the outside (just one power cable out), and I've been able to show it off many times in New York and SF with few problems. (It also helps that it always boots into the red calibration warning now that 3D calibration is merged, so it's easy to tell if it's up.)
@@ Line 134: / Line 178: @@
 A few different Pi-5-specific (and/or portable-gadget-specific) software issues:
+== Pass 4 vertices to VK_PRIMITIVE_TOPOLOGY_TRIANGLE_STRIP ==
+This was also mentioned under misc. system improvements -- initially flagged by [[https://mmm.s-ol.nu|s-ol bekic]] in Discord a few weeks ago but appeared really strongly in checkerboard calibration on the Pi 5 gadget. Weird glitching when rendering calibration tags onto the checkerboard.
+[[https://github.com/FolkComputer/folk/commit/2ac150d912f44dbafb1faf64658289dcf314736e|Realized we need to pass 4 vertices, not 6, to vkCmdDraw]] to make it work on a bunch of GPUs (why were we passing 6 before? was it for the Chromebook GPUs or something? need to test)
 == Descriptor indexing ==
@@ Line 161: / Line 211: @@
 I've been working on the camera (I think it's a Pi camera module 2, but all the Pi camera modules work the same way more or less) to try to get calibration to work.
-I want to switch from using libcamerify+v4l2 (which gives me very little control, to adjust exposure in particular, which is often an issue during calibration) to using libcamera directly. So I [[https://github.com/FolkComputer/folk/commit/b4022818b198234ace61f006c0684c7a2883511c|added C++ support to Folk]], which took like an hour. And it's not too bad to use:
+I want to switch from using libcamerify+v4l2 (which gives me very little control, in particular to adjust exposure, which is often an issue during calibration) to using libcamera directly. So I [[https://github.com/FolkComputer/folk/commit/b4022818b198234ace61f006c0684c7a2883511c|added C++ support to Folk]], which took like an hour. And it's not too bad to use:
 {{:newsletters:pasted:20240903-193025.png?400px}}
@@ Line 181: / Line 231: @@
     * Current situation is putting a few spacers behind half the camera to tilt it up toward the projector area, but even this drifts because the projector drifts, and it feels too dependent on how the screws are tightened, not replicable.
     * {{:newsletters:pasted:20240902-012829.jpeg?150px}}
+    * Maybe use a wide-angle camera instead? or shift it upward physically?
+    * Maybe lock the projector in place better?
+    * Side note: maybe multiple cameras for more stable depth perception? cameras are cheap
   * Figure out the Wi-Fi situation, maybe run a hotspot from the gadget automatically for now instead of connecting to Wi-Fi? we don't really use internet from it anyway right now
+  * Maybe try to run some AI vision models? to segment environments, identify arbitrary objects by name, etc. opens up new interactions. GGML seems to have a Vulkan backend now, and the Pi 5 does have a Vulkan driver, even if it's not great. Need to spend a few days to actually try
 ==== 3D calibration ====
 Omar: We merged 3D calibration! This is a huge advance that's been in the works for much of the last year -- we're being aggressive by merging, there are still things to fix and it's only roughly on par with the old 2D calibration (better in some cases, worse in others) -- but this is why we are pre-alpha and don't promise anything to users yet :-)
+(it also means that regions are now deprecated compared to quads)
+{{:newsletters:pasted:20240904-023549.png?400px}}
 (3D calibration is also quite important for the portable gadget system, since it has no fixed plane in front of it that we could 2D-calibrate.)
-Mostly refinement of individual systems this month. Why aren't they as accurate as my home system? (Often the answer is exposure settings, or too-coplanar poses, or some other bit of tacit knowledge about calibration). They aren't too bad now, though:
+Several small changes before merging:
+  * Fixed [[https://haiperspace.com|Jacob Haip's]] mask-tags which keeps us from projecting on top of tags and potentially breaking them; now works in/based on 3D space
+  * [[https://github.com/FolkComputer/folk/commit/fd98eec92de3174e0a304ada0c592cde0dfd2c45|Fixed reprojection of camera slices]]
+  * RAM watchdog
+  * Allow negative distances in ''quad buffer'', implement ''quad move'', fix ''quad scale''
+    * with the camera slice fixes,  mostly fixes animation program as well once it's been tweaked to use quads
+Other than these & table refine below, mostly refinement/recalibration of individual systems this month. Why aren't they as accurate as my home system? (Often the answer is exposure settings, or too-coplanar poses, or some other bit of tacit knowledge about calibration). They aren't too bad now, though:
   * On folk-convivial:
@@ Line 241: / Line 307: @@
 The hack I came up with to maintain all these properties is to have an outside monitor process that just checks every few milliseconds and rebalances stuff (makes new processes, kills extra processes).
-I looked at Erlang's scheduler and Go's scheduler for inspiration after struggling with this a bit. (a lot of my attempts were unpredictable or would get into terminal too-few-processes state after killing stuff)
+I looked at Erlang's ([[https://medium.com/@jlouis666/erlang-dirty-scheduler-overhead-6e1219dcc7|"dirty"]]) scheduler and [[https://github.com/golang/go/blob/88dee90d6a296d7ed479505b6dc862a54f026c1c/src/runtime/proc.go|Go's scheduler]] for inspiration after struggling with this a bit. (a lot of my attempts were unpredictable or would get into terminal too-few-processes state after killing stuff)
+Erlang is weird because of all the [[https://www.erlang.org/docs/20/man/erl_nif#lengthy_work|hinting]] it requires. Erlang programmers aren't normally directly writing and calling C functions, so they treat it as an exception that you mark up if you expect unusual blocking.
-Erlang is weird because of all the hinting it requires. Erlang programmers aren't normally directly writing and calling C functions.
+{{:newsletters:pasted:20240904-023519.png?400px}}
-Go actually seems very close to what we want. It's a work-stealing scheduler that can deal with arbitrary CPU blocks and I/O blocks in a goroutine. It has a little bit of hinting (it can detect syscalls, since they all go through the Go runtime), but it also has something called 'sysmon' which is basically my hacky outside monitor process proposal!
+Go actually seems very close to what we want. It's a work-stealing scheduler that can deal with arbitrary CPU blocks and I/O blocks in a goroutine. It has a little bit of hinting to know about long-running work in advance (it can detect syscalls, since they all go through the Go runtime), but it also has something called [[https://medium.com/@blanchon.vincent/go-sysmon-runtime-monitoring-cff9395060b5|'sysmon']] which is basically my hacky outside monitor process proposal, so it can adapt to arbitrary blocking like we want.
 ===== What we'll be up to in September =====
@@ Line 262: / Line 330: @@
     * Trigger button
     * Cat printer support
+    * Put design online?
+  * Merge Daniel's receipt printer support
+  * Daniel hacking on touch detection a bit
   * Omar: I hope we can start working on some of the stuff with precise camera slices from the new calibration: OCRing fields, checkboxes, stuff like that
   * New evaluator: new scheduler, work on evaluation semantics, eventually resume porting the editor