Folk Computer

User Tools

Site Tools

Trace: • 2024-08 • 2024-10 • 2023-12 • 2025-01 • 2024-12 • 2024-05 • 2024-07 • 2023-11 • 2024-09 • 2024-01
newsletters:2023-11

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
newsletters:2023-11 [2023/11/30 20:15] – osnrnewsletters:2023-11 [2023/11/30 22:48] (current) – Remove Govind TODO, constrain keyboard GIF admin
Line 21: Line 21:
     * {{:newsletters:pasted:20231130-161747.png?350}}     * {{:newsletters:pasted:20231130-161747.png?350}}
  
-  * TODO: Govind? +==== Tabletop / in-system editor ====
- +
-==== Editor ====+
  
   * Naveen working on [[https://gist.github.com/nmichaud/f584bc39aa6f84cf7a275b920239b9da|a local editor]] on his system   * Naveen working on [[https://gist.github.com/nmichaud/f584bc39aa6f84cf7a275b920239b9da|a local editor]] on his system
Line 30: Line 28:
   * Andrés working on multi-keyboard, moving keyboard into virtual program   * Andrés working on multi-keyboard, moving keyboard into virtual program
     * (we'll finally be able to dump tcl-thread and pi.tcl and have everything in virtual programs and subprocesses and statement communication!)     * (we'll finally be able to dump tcl-thread and pi.tcl and have everything in virtual programs and subprocesses and statement communication!)
 +    * Here's a preview of it monitoring key presses from multiple keyboards plugged into %%folk0%%:
 +    * {{:newsletters:pasted:20231130-223928.gif?350}}
  
 ==== Calibration and CNC ==== ==== Calibration and CNC ====
Line 37: Line 37:
 (the material might be an inch or two thick, so you need to be able to projection-map 3D surfaces above the base plane of the Folk system, and you need high accuracy -- current Folk is only calibrated to a single plane homography and is often 1cm off or worse) (the material might be an inch or two thick, so you need to be able to projection-map 3D surfaces above the base plane of the Folk system, and you need high accuracy -- current Folk is only calibrated to a single plane homography and is often 1cm off or worse)
  
-Basically, you need to hold up some known pattern in a few different poses and project onto it, and that lets you characterize the behavior of the camera and projector in 3D space. Most people say that a pretty simple pinhole-camera (linear) model with some adjustment for radial distortion is sufficient for better-than-a-pixel accuracy.+Basically, in order to do 3D calibration, you need to hold up some known pattern in a few different poses and project onto it, and that lets you characterize the behavior of the camera and projector in 3D space. Most people say that a pretty simple pinhole-camera (linear) model with some adjustment for radial distortion is sufficient for better-than-a-pixel accuracy.
  
-Using AprilTags for the projected pattern is nice, because we already link the AprilTag library (so no need for more custom computer vsiion), and because they're self-verifying and error-correcting and give you a lot of information -- if you see an AprilTag, you know it was the exact AprilTag you tried to project, and you know its identity and corners and angle, as opposed to projecting stripes or something where you need your own thresholds and classification to figure out what stripes you're seeing (if any).+Using AprilTags for the projected calibration pattern is nice, because we already link the AprilTag library (so no need for more custom computer vision), and because they're self-verifying and error-correcting and give you a lot of information -- if you see an AprilTag, you know it was the exact AprilTag you tried to project, and you know its identity and corners and angle, as opposed to projecting stripes or something where you need your own thresholds and classification to figure out what stripes you're seeing (if any).
  
-  * Early November: the first 3D calibration test/demo, using a calibration pattern based on [[http://www.ok.sc.e.titech.ac.jp/res/PCS/research/procamcalib/|Audet (2009)]] but with AprilTags and printing on the outer perimeter instead of alternating+  * Early November: the first 3D calibration test/demo, using a calibration pattern based on [[http://www.ok.sc.e.titech.ac.jp/res/PCS/research/procamcalib/|Audet (2009)]] but using AprilTag fiducials and printing on the outer perimeter instead of alternating even/odd
     * {{:newsletters:pasted:20231130-183930.png?0x300}}     * {{:newsletters:pasted:20231130-183930.png?0x300}}
 +    * It's useful to leave gaps in the printed and projected patterns so it's easy to move the board & get the tags 'aligned enough' that the system can see at least 1 projected + 1 printed tag on the same plane, which is then enough information to properly rewarp all the projected tags to fit into the calibration board.
 +    * Then once you can see most of the projected + printed tags on the board, you record that as a set of example points for calibration (you now have both board-plane coordinates in meters and camera and/or projector coordinates in pixels for all corners of all those tags)
     * Notice how tags in the center of the calibration board are projected dynamically, while tags on the outer perimeter are printed:     * Notice how tags in the center of the calibration board are projected dynamically, while tags on the outer perimeter are printed:
-    * {{:newsletters:pasted:20231106-180904.jpeg?0x300}} {{newsletters:img_1095.mp4?0x300}}+      * {{:newsletters:pasted:20231106-180904.jpeg?250}} {{newsletters:img_1095.mp4?250}} 
 +    * {{:newsletters:pasted:20231122-164229.png?350}}
  
-  * {{:newsletters:pasted:20231122-164229.png?350}} +  * Mid-November: We were running into a problem: how do we know when the rewarped projected tags have actually rewarped? there's some latency (tens of milliseconds? hundreds of milliseconds?) from when Folk rewarps the tags to fit the board and when that rewarp actually appears in the webcam feed. If you read the tags too early and do another rewarp based on that, the tags become unreadable and you have to abort calibration. And you obviously don't want to just have a fixed-length sleep (which is what we did at first), you want to dynamically adapt to whatever latency happens to be present. 
- +    * So we introduced a concept of //versions//, where we rotate half the projected tags by 90 degrees every rewarp, so we can tell if the rewarp has gone through by the rotations of the detected tags
-  * Mid-November: we started running into a problem: how do we know when the rewarped projected tags have actually rewarped? there's some latency (tens of milliseconds? hundreds of milliseconds?) from when Folk rewarps the tags to fit the board and when that rewarp actually appears in the webcam feed. So we introduced a concept of //versions//, where we rotate half the projected tags by 90 degrees every rewarp, so we can tell if the rewarp has gone through by the rotations of the detected tags+
     * {{newsletters:img_1517.mp4?350}}     * {{newsletters:img_1517.mp4?350}}
  
-  * Late November: Why was it projecting upside-down? Surprisingly intuitive answer+  * Late November: We started testing the 3D calibration but found that tag outlines were weirdly offset -- on folk-convivial, they were a few inches off, and on my home system, they were actually flipped, where the outline would appear on the other side of the table and move in the opposite direction from the physical tag. 
 +    * Why was it projecting upside-down or offset? The answer is surprisingly cool and physically intuitive: I was using //camera frame coordinates// (x, y, z from the center of the camera lens) where I should have been using //projector frame coordinates// (x, y, z from the center of the projector lens), so I was projecting the outline as though the projector was where the camera is. 
 +    * And in folk-convivial, the projector was a few inches from the camera but oriented the same way, while on my home system, the projector was upside-down relative to the camera! 
 +    * Had to figure out extrinsic information during camera calibration and projector calibration (what is the rotation and translation of each pose relative to the projector & relative to the camera), then use those extrinsics to derive the rotation and translation of the projector frame relative to the camera frame ("stereo calibration") 
 + 
 +  * Late November: Finally got 3D calibration to the point of testing it and seeing the outlines align with the tags: 
 +    * {{newsletters:img_1639.mp4}} 
 +    * The outline is still ~1cm off, but notice how it's the right size no matter what distance from the camera or skew I hold the tag up at. That's a 3D calibration :-) (and internally, coordinates here are all in meters from the center of the camera lens, too) 
 + 
 +  * Next: try to improve smoothness and accuracy by using nonlinear refinement & adding a radial distortion term to the calibration 
 + 
 +I'm excited about real-world calibration because it feels like the first system change in a long time (6 months? a year?) that will actually bring in new modes of interaction, that will //look// different to the untrained eye from putting pieces of paper on a table and watching them light up. (This is also why the [[guides/button|button]] is great.)
  
-I'm excited about real-world calibration because it feels like the first system change in a long time (6 months? a year?) that will actually bring in new modes of interaction, that will //look// different to the untrained eye from putting pieces of paper on a table and watching them light up. (This is also why the [[guides/button|button]] is great.)  I've spent a lot of the last 6-12 months on performance and on making cleaner and more powerful APIs, but I haven't pushed the interaction paradigm or the application set enough yet. Seeing preview lines on chunks of wood on a CNC machine bed, having a projection-mapped dollhouse or town, highlighting and sharing sections of a book, writing in blank fields on printed programs -- these are the directions where we should be pushing.+I've spent a lot of the last 6-12 months on performance and on making cleaner and more powerful APIs, but I haven't pushed the interaction paradigm or the application set enough yet. Seeing preview lines on chunks of wood on a CNC machine bed, having a projection-mapped dollhouse or town, highlighting and sharing sections of a book, writing in blank fields on printed programs -- these are the directions where we should be pushing.
 ==== Other system improvements ==== ==== Other system improvements ====
  
Line 67: Line 80:
   * Naveen added [[https://github.com/FolkComputer/folk/pull/112|non-uniform (different in X and Y) image scaling]]   * Naveen added [[https://github.com/FolkComputer/folk/pull/112|non-uniform (different in X and Y) image scaling]]
  
-  * Naveen added [[https://github.com/FolkComputer/folk/pull/119|mailbox clearing when a subprocess goes down due to unmatch]]+  * Naveen added [[https://github.com/FolkComputer/folk/pull/119|mailbox clearing when a subprocess gets killed by unmatch]]
     * but **not** (yet) when subprocess goes down due to an internal crash (like a segfault) or due to arbitrary external Unix kill (we have this whole supervisor thing already that's meant to be able to waitpid on each subprocess and detect those conditions, we're just not using it)     * but **not** (yet) when subprocess goes down due to an internal crash (like a segfault) or due to arbitrary external Unix kill (we have this whole supervisor thing already that's meant to be able to waitpid on each subprocess and detect those conditions, we're just not using it)
     * see [[https://discord.com/channels/956758212152025098/956758650700046366/1175908063019008021|Discord discussion of mailbox clearing]]     * see [[https://discord.com/channels/956758212152025098/956758650700046366/1175908063019008021|Discord discussion of mailbox clearing]]
Line 75: Line 88:
 Omar has been throwing around ideas for a parallel Folk evaluator. It's an interesting design problem -- was talking it over in Discord with Naveen and Arcade and others. Omar has been throwing around ideas for a parallel Folk evaluator. It's an interesting design problem -- was talking it over in Discord with Naveen and Arcade and others.
  
-The motivation is actually sort of a user interface motivation: I want people to feel comfortable writing programs that hang or segfault, and I want people to feel comfortable putting 30 different programs on the table, without being scared that they'll crash the whole Folk system or break their (in-system) editor. That requires Unix process isolation, I think, and it requires preemptive multitasking.+The main motivation is actually sort of a user interface motivation: I want people to feel comfortable writing programs that hang or segfault, and I want people to feel comfortable putting 30 different programs on the table, without being scared that they'll crash the whole Folk system or break their (in-system) editor. That requires Unix process isolation, I think, and it requires preemptive multitasking.
  
 And having the system schedule and parallelize everything itself could reduce latency: right now, even though the system is multiprocess (camera intake, tag detection, and display run on separate Folk subprocesses), the main process is a latency bottleneck for the entire system.  And having the system schedule and parallelize everything itself could reduce latency: right now, even though the system is multiprocess (camera intake, tag detection, and display run on separate Folk subprocesses), the main process is a latency bottleneck for the entire system. 
 +
 +<details>
  
 {{:newsletters:pasted:20231130-193154.png?500|you can kind of think of latency as being about how long it takes, with respect to some user action like moving a page, to do the specific critical execution path of input (camera intake) -> compute (apriltag recognition, program execution, shader wish construction) -> output (vulkan draw) {{:newsletters:pasted:20231130-193154.png?500|you can kind of think of latency as being about how long it takes, with respect to some user action like moving a page, to do the specific critical execution path of input (camera intake) -> compute (apriltag recognition, program execution, shader wish construction) -> output (vulkan draw)
Line 85: Line 100:
 (And I think it'd be more elegant -- it's a return to some of the original tuplespace pitch, as a concurrency model -- it could free the user from some implementation details of processes and memory allocation and what calls block and what calls are fast -- it could let debugging tools have more access to the running database and not interfere with the runtime -- it would enforce a very attractive purity that you can't use global variables or functions, everything really must be shared through the database, forcing us to improve the database and language ergonomics accordingly) (And I think it'd be more elegant -- it's a return to some of the original tuplespace pitch, as a concurrency model -- it could free the user from some implementation details of processes and memory allocation and what calls block and what calls are fast -- it could let debugging tools have more access to the running database and not interfere with the runtime -- it would enforce a very attractive purity that you can't use global variables or functions, everything really must be shared through the database, forcing us to improve the database and language ergonomics accordingly)
  
-<details> 
 > [[https://discord.com/channels/956758212152025098/956765077883744266/1176921576483078345|Continued thinking about parallel Folk:]] I feel like there are two sort of platonic designs with different advantages & disadvantages > [[https://discord.com/channels/956758212152025098/956765077883744266/1176921576483078345|Continued thinking about parallel Folk:]] I feel like there are two sort of platonic designs with different advantages & disadvantages
 >  > 
Line 108: Line 122:
 > >
  
-== jim ==+----
  
 One thought is that a parallel Folk evaluator would basically be a mix of [[https://jim.tcl.tk/index.html/doc/www/www/index.html|jimtcl]] + a C kernel/database/evaluator which knows all the statements and can pull a When body string out and jimtcl-evaluate it on any CPU One thought is that a parallel Folk evaluator would basically be a mix of [[https://jim.tcl.tk/index.html/doc/www/www/index.html|jimtcl]] + a C kernel/database/evaluator which knows all the statements and can pull a When body string out and jimtcl-evaluate it on any CPU
Line 125: Line 139:
 Yeah Folk has the feeling that things eventually converge Yeah Folk has the feeling that things eventually converge
 }} }}
 +
 </details> </details>
  
Line 154: Line 169:
     * Fun interim on-a-wall stage during setup, before we pointed it down at the table:     * Fun interim on-a-wall stage during setup, before we pointed it down at the table:
       * {{:newsletters:pasted:20231130-161524.png?250}}       * {{:newsletters:pasted:20231130-161524.png?250}}
-      * (whenever I see a projection mapping demo, I always feel like we should be able to subsume it and make it really easy to set up and reprogram, like it was trivial to make the meow text here wiggle around over time)+      * I think this image is a fun provocation because it's sort of //about the wall//, the drawing on the whiteboard, instead of being about the system. Imagine wheeling a system over in 10 seconds and pointing it at part of your environment because you want a thin layer of computation or sharing or dynamism on top of what's there (and not because you want a generic "Folk system" set up that could be set up anywhere) 
 +      * (btw, whenever I see a projection mapping demo, I always feel like we should be able to subsume it and make it really easy to set up and reprogram, like it was trivial to make the meow text here wiggle around over time)
  
   * Simplified install instructions to just use Vulkan binaries from apt, instead of compiling mesa from scratch   * Simplified install instructions to just use Vulkan binaries from apt, instead of compiling mesa from scratch
Line 163: Line 179:
       * Got a Logitech C922 camera for folk-beads: at least it can do 60fps at 720p, much smoother-feeling than the C920 or Azure Kinect which can only do 30fps at any resolution       * Got a Logitech C922 camera for folk-beads: at least it can do 60fps at 720p, much smoother-feeling than the C920 or Azure Kinect which can only do 30fps at any resolution
  
 +===== Hmm =====
 +
 +{{newsletters:pasted:20231130-213756.png?550|one observation re: how people pattern-match and overfit to the examples they see is we get a lot of questions and comments about the AprilTag black-and-white markers we use, even though I would say we view those as much less essential than the programming system and database
 +
 +and there are a lot of applications like CNC preview or various projection-mapping demos where I think you should be able to get lots of value out of using Folk w/o having any tags visible in the end interface at all}}
 +
 +(I think part of our job has to be to just broaden the examples so that fitting to them still gives you a pretty productive working set)
 ===== What we'll be up to in December ===== ===== What we'll be up to in December =====
 +
 +  * **[[https://partiful.com/e/MJ6V5SfEKRPPjOskxeti|The next Folk open house]] will be on the evening of Wednesday, December 13, at our studio in East Williamsburg, Brooklyn.** :-)
  
   * Still need to complete RFID refactor. I need to make the ringbuffer faster and I need to wait on less data to start parsing ..   * Still need to complete RFID refactor. I need to make the ringbuffer faster and I need to wait on less data to start parsing ..
Line 172: Line 197:
  
   * Hacking on parallel Folk?   * Hacking on parallel Folk?
 +
 +  * Finishing up tabletop editor
  
  
Line 179: Line 206:
 ==== Andrés ==== ==== Andrés ====
   * [[https://github.com/tophtucker/printcode|Toph made a tool to print out a repo of code nicely]]   * [[https://github.com/tophtucker/printcode|Toph made a tool to print out a repo of code nicely]]
 +  * [[https://twitter.com/disconcision/status/1710716024323006878?s=12&t=AUhNsAKs8G7beopf-_iD3Q|Fence diagrams]]
 +  * [[https://www.youtube.com/watch?v=Y8Qm48_GhJ8&list=PLYCO5FdtsVwtOT66VfhUkkRjIyLpKkTLB&index=7&loop=0|Andrew Sutherland and the Paper Computer, interactivity, computability and no screens!]]
 +  * [[https://www.midst.press/|Midst - What if you could watch your favorite poet write?]]
 +  * [[https://dailypapert.com/hard-fun/|Hard Fun — Seymour Papert]]
 +  * [[https://pixelpoppers.com/2018/03/who-frustration-is-good-for/|Who Frustration is Good For
 +]]
  
 ==== Omar ==== ==== Omar ====
Line 186: Line 219:
   * [[https://sosheskaz.github.io/tutorial/2016/09/26/Avahi-HTTP-Service.html|How to use Avahi to advertise an HTTP service]]   * [[https://sosheskaz.github.io/tutorial/2016/09/26/Avahi-HTTP-Service.html|How to use Avahi to advertise an HTTP service]]
   * [[https://sci-hub.ru/10.1109/2.44903|Linda meets Unix]]   * [[https://sci-hub.ru/10.1109/2.44903|Linda meets Unix]]
 +  * [[https://types.pl/@disconcision/111467347191732238|nool sounds]]
 +  * [[https://blog.geocities.institute/archives/7047|Shawna, a Set Maker]]
newsletters/2023-11.1701375308.txt.gz · Last modified: 2023/11/30 20:15 by osnr
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki