Differences

This shows you the differences between two versions of the page.

--- newsletters:2024-10 [2024/10/31 06:22] – [RFID localization] osnr
+++ newsletters:2024-10 [2024/10/31 10:04] (current) – [System improvements] osnr
@@ Line 1: / Line 1: @@
-====== October 2024 newsletter (WIP) ======
+====== October 2024 newsletter ======
 (Our next Folk open house is on **[[https://partiful.com/e/rKpJ32TqUZkNSLWrygyu|the evening of Wednesday, November 20]]**, at our studio in East Williamsburg, Brooklyn.)
@@ Line 17: / Line 17: @@
     * {{:newsletters:pasted:20241029-111557.jpeg?0x200px}} {{:newsletters:pasted:20241029-111712.jpeg?0x200px}} {{:newsletters:pasted:20241029-111740.jpeg?0x200px}}
     * Some issues with the recording, but here's the video:
-      * [[https://x.com/GopalKRaman/status/1844135235128262934|{{newsletters:fa648de6-d872-4299-af31-d4ae1f759f2f-3841-000001a4bc2b8e5f.png?300px}}]]
+      * [[https://x.com/GopalKRaman/status/1844135235128262934|{{newsletters:fa648de6-d872-4299-af31-d4ae1f759f2f-3841-000001a4bc2b8e5f.png?350px}}]]
   * Omar [[https://x.com/rsnous/status/1847321061261484107|talked about]] his background & the motivation and ideas behind Folk on [[https://podcasts.apple.com/us/podcast/the-orthogonal-bet-from-online-communities-to-in/id1615629580?i=1000670683552|the Lux Riskgaming podcast with Sam Arbesman]].
+    * [[https://x.com/rsnous/status/1847321061261484107|{{:newsletters:pasted:20241031-075205.png?350px}}]]
 ==== Applications and demos ====
-  * [[https://danielpipk.in|Daniel Pipkin]] Folktober
+  * [[https://danielpipk.in|Daniel Pipkin]] was [[https://x.com/_ppkn/status/1851674991609159803|trying "Folktober"]], where he made Folk programs based on the daily Inktober prompts.
+    * [[https://x.com/_ppkn/status/1851674991609159803|{{:newsletters:pasted:20241031-071510.png?400px}}]]
+    * He ended up making 8 programs: "Backpack", "Discover", "Boots", "Binoculars", "Trek", "Passport", "Hike", and "Sun"
 ==== Desksaver ====
@@ Line 32: / Line 35: @@
 {{:newsletters:pasted:20241030-221705.jpeg?0x300px}}
-==== System improvements ====
+==== Gadget/system improvements ====
-Andrés and Omar have been showing the gadget a lot lately, both at outside presentations and at open houses, so we've made a lot of small gadget-related fixes.
+Andrés and Omar have been showing the gadget a lot lately, both at outside presentations and at open houses, so Omar has made a lot of small gadget-related fixes.
-(these all apply to the system more generally -- so the gadget is paying off for the normal ceiling systems as well -- but have been particularly acute or frequent with the gadget, because it has more tags at the fringe, it moves more, it has less RAM, etc)
+These all come together to make the gadget feel much more like the computational flashlight we want it to be:
+{{youtube>SHHPDiTw8jo?}}
+(these improvements all apply to the system more generally -- so the gadget is paying off for the normal ceiling systems as well -- but have been particularly acute or frequent with the gadget, because it has more tags at the fringe, it moves more, it has less RAM, etc)
   * [[https://github.com/FolkComputer/folk/commit/e19372bccb796d7bd3827fe71ca660dec4a0665a|Fixed crash where number of Gauss-Newton iters gets really high]] when tag is near edge of camera space, which was causing segfault in those cases
-    * Omar: I guess that we were just blowing out the stack with tens of thousands of iterations (and associated stack-allocated matrices) when tags are too off-camera since it just may never converge in that case
+    * I guess that we were just blowing out the stack with tens of thousands of iterations (and associated stack-allocated matrices) when tags are too off-camera since it just may never converge in that case
     * (normally you can get pretty good in 10-20 iterations, in my experience, so we could maybe reduce this cap further)
   * evaluator: Did some heap profiling and [[https://github.com/FolkComputer/folk/commit/1d82d222e2f8e593b4e1ff18f6199f32134ede8d|fixed several memory leaks]]
-    * (at first heap profiled with [[https://gperftools.github.io/gperftools/heapprofile.html|gperftools]], and then with [[https://www.tcl.tk/man/tcl8.6/TclLib/TCL_MEM_DEBUG.htm|Tcl memory profiling]], because Tcl uses its own allocator for ckalloc anyway and we mostly use that outside apriltag lib, and then finally just looked at the memory map in /proc and verified that the heap is the problem, dumped the heap with gdb and looked at it in a hex editor and saw all these strings repeated)
+    * (at first heap profiled with [[https://gperftools.github.io/gperftools/heapprofile.html|gperftools]], and then with [[https://www.tcl.tk/man/tcl8.6/TclLib/TCL_MEM_DEBUG.htm|Tcl memory profiling]], because Tcl uses its own allocator for ckalloc anyway and we mostly use that outside apriltag lib)
+    * (and then finally just looked at the memory map in /proc and verified that the heap is the problem)
+      * {{newsletters:6faf21bb-484b-4d14-8bd8-b6e63615c69f-36499-000005b0b42804e3.png?400px}}
+      * (then dumped the heap with gdb and looked at it in a hex editor and saw a lot of strings repeated, leading to the fix below)
     * Mostly just fixes in evaluator.tcl -- Omar had been lazy and was allocating new strings in various places that should just be allocated once and reused (for stuff like ''::matchId'' and for other dict keys that are fixed)
     * It still leaks, but way slower (the 2GB Pi 5 used to crash in 12 minutes or so, now //at least// 20-30 minutes uptime, could be much more). More to do here!
-=== camera-rpi improvements ===
+=== Gadget-specific improvements ===
 These apply only to the gadget camera (or to any other system that uses Pi cameras instead of USB webcams) -- they're improvements to the camera-rpi.folk driver:
   * Massively sped up gadget camera (5fps->60fps) by [[https://github.com/FolkComputer/folk/commit/572f5e5546daa3314dc392b3bd645d96b78ec03d|telling libcamera to use YUV420 format directly]] (which already has a gray plane that we can just copy) instead of using ARGB8888 and converting to grayscale
-    * We were literally spending 200ms each frame converting the RGB camera image to grayscale :-)
+    * Saw that tracking was slow on the gadget, looked at metrics, saw that the camera thread was only running at 5fps so that was obviously the bottleneck
-  * Hack to fix dequeue timeout crash
+    * We were literally spending 200ms each frame converting the RGB camera image to grayscale :-) it's like 99% of the flamegraph of the camera thread:
+      * {{:newsletters:pasted:20241031-070311.svg}}
+  * Hack to fix dequeue timeout crash (which sometimes makes camera stop working after a while): [[https://github.com/raspberrypi/picamera2/issues/952#issuecomment-2082145259|increase timeout in libcamera environment config]]
+    * {{newsletters:9cffd576-df47-4134-acd3-f51e36e95ac9-36499-000005ac93ff4c15.png}}
   * Noticed that the Pi wide-angle camera has focus control, unlike the normal camera module, and we hadn't set the focus, so it was at [[https://libcamera.org/api-html/namespacelibcamera_1_1controls.html#acfb623408035e51db904a0f3928a863e|the default of 2m away]], which is way too far; locked focus closer, to ~30cm away from the camera which seems like a good start and seems to help with recognition stability a bit
+== EDID fix ==
+The combination of the Pi 5, ribbon-type micro-HDMI to HDMI cable, and pico projector (Ultimems HD305D1-C1) seems to sometimes (maybe only at the Linux fbcon level? outside when Folk controls Vulkan) produce garbled output. I figured out this is because the EDID data doesn't get through properly to provide a reasonable default resolution?
+[[https://forums.raspberrypi.com/viewtopic.php?t=362124|Solution]]: [[https://github.com/FolkComputer/gadget/commit/20d0208a8f6cd8225db63bb6cfb9b176d5a50bc3#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R115|hard-code the EDID to 720p in /boot/firmware/cmdline.txt.]]
+== Vulkan fix ==
+Sometimes you only see llvmpipe (the not-useful software renderer) and not the actual Pi 5 GPU on ''vulkaninfo --summary''. I've found that this gets fixed by doing ''sudo rpi-update'' (brings back a good Pi kernel with good GPU support?)
 ==== Cat printer printing ====
-Omar: I wanted printing that we could take on the go (to the SPC demo night and abroad), so I revived [[https://ijc8.me|Ian Clester's]] indexcard layout for printing on cat printer receipts (and updated it to align & got cat printer printing over BLE working on our gadget.
+Omar: I wanted printing that we could take on the go (to the SPC demo night and abroad), so I revived [[https://ijc8.me|Ian Clester's]] indexcard layout for printing on cat printer receipts & got cat printer printing over BLE to work.
-(only the layout changes are really part of the Folk codebase, but it's still nice to see all this come together and understand the Linux and CUPS steps needed)
+The receipts are about playing-card width, so I've been carrying around a deck of playing cards and attach each program to a card (to make it flat and also feel more object-y).
+{{:newsletters:pasted:20241030-203540.png?0x200px}}  {{:newsletters:pasted:20241031-070741.jpeg?0x200px}}
-Folk sees the cat printer as just another CUPS printer (that you manually configure to use indexcard layout in setup.folk); I've been installing and running [[https://github.com/NaitLee/Cat-Printer|NaitLee Cat-Printer]] which [[http://hamin.se/post/catPrinterWithAppleTalk/|provides CUPS interface to the printer]]; Ian got this to work on his system last year but we hadn't tried it ourselves & we hadn't updated the layout as the printing system has changed.
+(only the print layout changes are really part of the Folk codebase, but it's still nice to see all this come together and understand the Linux and CUPS steps needed)
-{{:newsletters:pasted:20241029-181744.png?0x300px}} {{:newsletters:pasted:20241029-181759.png?0x300px}}
+Folk sees the cat printer as just another CUPS printer (that you manually configure to use indexcard layout in setup.folk); I've been installing and running [[https://github.com/NaitLee/Cat-Printer|NaitLee Cat-Printer]] which [[http://hamin.se/post/catPrinterWithAppleTalk/|provides CUPS interface to the printer]]; Ian got this to work on his system last year, but we hadn't tried it ourselves & we hadn't updated the layout as the printing system has changed.
-{{:newsletters:pasted:20241029-181822.png?0x300px}} {{:newsletters:pasted:20241030-203540.png?0x300px}}
+{{:newsletters:pasted:20241029-181744.png?0x200px}} {{:newsletters:pasted:20241029-181759.png?0x200px}}
+{{:newsletters:pasted:20241029-181822.png?0x200px}}
+=== Cat printer cards ===
+Finally, I made cards to scan for and connect the printer, so you don't need a laptop to do it (in theory):
+{{:newsletters:pasted:20241031-070811.jpeg?250px}}
+Here's the "Connect printer" card:
+{{:newsletters:pasted:20241031-073701.jpeg?250px}}
+Its source code:
+<code>
+Wish $this is labelled [exec curl -s \
+  -X POST http://localhost:8095/connect -d {{"device":"MX06,DF:07:26:13:53:BB"}}]
+</code>
@@ Line 74: / Line 116: @@
 Omar: I've been picking this up again in October because I want to get it to work.
-And I have good news: I fixed the bug that had me stuck in [[https://folk.computer/newsletters/2024-05#small-system-improvements|May]] and June, where the in-band reader would crash after the first round completed. It turned out that the problem was that I was using the ''%'' (mod) operator in the hot transmit loop (to decide the index of the sample to send out), millions of time per second, and that operator is actually pretty slow. I added another counter and if statement instead, and that works great:
+And I have good news: I fixed the bug that had me stuck in [[https://folk.computer/newsletters/2024-05#small-system-improvements|May]] and June, where the in-band reader would crash after the first round completed. It turned out that the problem was that I was using the ''%'' (mod) operator in the hot transmit loop (to decide the index of the sample to send out), millions of times per second, and that operator is actually pretty slow. I added another counter and if statement instead, and that works great:
 {{:newsletters:pasted:20241031-061559.png?500px}}
@@ Line 86: / Line 128: @@
 We need to build the OOB logic again (what samples to transmit and receive), which isn't too bad since it's much less intelligent than IB and not on a deadline, it's mostly just bookkeeping and hopping, and we also need all the server and synchronization stuff to boot OOB, timesync OOB with IB, feed the samples to the PC.
-Anyway, with the IB RFID reader working, I feel much more optimistic that we can bring this all together and release it in some useful form.
+Anyway, with the IB RFID reader working, I feel optimistic that we can bring this all together and release it in some useful form.
 === Unreliability of IB reader ===
-it's a little finicky because the decoder is so much less robust, it's on a deadline
+One issue has been that reading the tags is finicky (sometimes you have to move it, or it doesn't work as well on my home system, etc) because the new fast decoder is so much less robust. (maybe continuous reading instead of one-round will help with this once we get it?)
+(the reader is live & on a deadline now, so it's just using a binary threshold on amplitude, and it only uses the first few samples to compute that threshold; the old decoder looked at the whole stream and tried to decode it in the optimal way).
+How to threshold something like this? So much stuff that's in the middle even on visual inspection.
-How to threshold this?
+{{:newsletters:pasted:20241029-182100.png?500px}}
-{{:newsletters:pasted:20241029-182100.png?300px}}
+You often get read failures when the tag responses are sketchy like that right now.
 ==== Friends and outreach ====
@@ Line 117: / Line 163: @@
 ==== Some demo lessons ====
-Hard-won experience from what things we needed (& what things broke down) at mobile demos over the last couple of months:
+Hard-won experience from what things we needed (& what things broke down) (and what questions we got) at mobile demos over the last couple of months:
 (at Hex House or at home, we have a little more scope to just restart things, we have laptops and other tools around, the Wi-Fi is known-good, etc, so these problems are not as acute)
-  * Grayed-out printing of receipts -- not reliable?
+  * Sometimes printing of tags on receipts is a bit faded -- makes detection less reliable
-  * Calibration gets knocked around, is often off
+  * Calibration gets knocked around if the front panel moves (which it will, it's only attached at 2 screw points) or the camera wiggles off its PCB (the glue seems weak? is it the heat?)
-    * Solution (not implemented yet): Need to make camera attachment to gadget more rigid
+    * Solution 0 (done): superglue the camera back to the PCB
-    * Another solution: Need to make calibration fast and reliable on the go
+    * Solution 1: make camera attachment to gadget more rigid; put the threaded holes on the gadget chassis instead of on the separate panel
+    * Solution A: make calibration fast and reliable on the go
       * or have a fast, quick calibration refinement we can do, rather than the full calibration
-  * Printer status not clear
+  * Printer status not clear, pressing Ctrl-P can be uncertain
-    * Solution 1: Printer control cards
+    * Solution 1 (done): Printer control cards
-  * Wi-Fi is questionable
+    * Solution 2: Ambiently show printer status, print job status once you hit Ctrl-P?
-  * TODO: EDID fix
+  * Wi-Fi is questionable as a way to talk to Folk -- you need to manually get it on the network SSID and password & sometimes [[https://x.com/rsnous/status/1849252167053721945|local connections are blocked]]
+    * Solution 1: use Bluetooth Low Energy
+    * Solution 2 (achieved): carry portable wireless USB keyboard that can always go to a Linux terminal, enter Wi-Fi options, do other stuff
+    * Solution 3: some kind of wired serial terminal that you can drive from laptop?
+    * Solution 4: run an ad-hoc Wi-Fi always? haven't figured this out yet
   * Reliance on power outlets is very annoying -- what if you have a meeting at a cafe? what if you just want to carry the thing around the warehouse and point it at walls?
     * Solution: Waveshare makes a [[https://www.waveshare.com/ups-module-3s.htm|standalone battery unit for the Pi 5]] that is perfect -- it's not a hat, so it works with our existing design -- Velcro it to the top of the gadget for now
       * {{:newsletters:pasted:20241029-090746.jpeg?150px}} {{newsletters:img_8468.jpg?150px}}
       * (maybe do a respin with a built-in battery hat? unit would get significantly bigger)
+      * does this break the projector?
   * Calibration not reliable?
-  * Would be good to pre-print:
+    * -> "camera occlusion seems to be a big problem (i don't really feel this but it could be the position from which I was presenting)"
+    * -> "My arms kept getting in the way. I think more than occlusion, the tag masking wasn't working great for some of the programs I had out and that was getting misinterpreted as being an occlusion problem."
+    * -> "yeah, if you're leaning hard on tag masking I think you need 1mm accuracy -- often I would say that tweaking exposure is the key thing (and there's a lot we could do in the UI to support making better calibration, too)"
+  * Daniel: Would be good to pre-print:
     * Recalibrate card
-    * Display camera frame on table card (to help adjust exposure)
+    * Display camera frame on table card (to help adjust exposure while calibrating, debug tag detection failures)
-Escape hatches: bring a laptop, bring a mobile hotspot, ssh, web ui
+Daniel's escape hatches: bring a laptop, bring a mobile hotspot, ssh, use the web ui
 ===== What we'll be up to in November =====
@@ Line 146: / Line 201: @@
   * New gadget design with built-in battery? with support for the off-the-shelf AnyBeam? maybe trigger button finally? second camera?
     * hope: we can actually make it turnkey to manufacture or sell or buy one, not be limited by imports
-  * Get RFID  oob working, start localizing
+  * Get RFID out-of-band radio working, start localizing again
-  * New evaluator scheduling
+  * New evaluator scheduler needs to be finished
-  * 3d calibration exposure ui, refinement
+  * Improvements to 3D calibration to make it easier to run on the fly & to get good results: exposure control, feedback on how many tags were detected, post-calibration refinement procedure
-  * Write in fields using 3d calibration
+  * Write in fields using the high-accuracy 3D calibration + some OCR system
+    * First example: geometry setter?
   * Upgrade documentation
   * Create printable booklets and exercises to interactively teach people Folk programming