newsletters:2024-06
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| newsletters:2024-06 [2024/06/28 21:05] – [New parallel evaluator] osnr | newsletters:2024-06 [2024/07/01 08:37] (current) – osnr | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ====== June 2024 newsletter | + | ====== June 2024 newsletter ====== |
| ===== What we've been up to ===== | ===== What we've been up to ===== | ||
| Line 5: | Line 5: | ||
| ==== Demos ==== | ==== Demos ==== | ||
| - | * We printed the 1000th program on folk0 (at Hex House)! | + | * We printed the 1000th program on folk0 at Hex House! |
| * {{newsletters: | * {{newsletters: | ||
| * Our friend [[https:// | * Our friend [[https:// | ||
| * {{newsletters: | * {{newsletters: | ||
| - | * {{newsletters: | + | * {{newsletters: |
| * [[https:// | * [[https:// | ||
| * {{newsletters: | * {{newsletters: | ||
| Line 26: | Line 26: | ||
| * [[https:// | * [[https:// | ||
| * {{newsletters: | * {{newsletters: | ||
| - | | + | |
| ==== Real-world/ | ==== Real-world/ | ||
| - | Omar: A lot of progress on 3D calibration this month. | + | Omar: A lot of progress on 3D calibration this month. |
| - | | + | |
| + | - to build much more accurate tracking in general -- millimeter precision -- to enable new applications where you can highlight individual words or facets | ||
| + | - to track coordinates in real-world meters instead of arbitrary projector or camera pixels, so we can start to integrate multiple projectors, cameras, and other real-world sensors and actuators (phone localization, | ||
| + | |||
| + | This video (running the osnr/ | ||
| + | |||
| + | {{newsletters: | ||
| + | |||
| + | The distance is being displayed by this sort of hacked-together program using the new concept of '' | ||
| + | |||
| + | < | ||
| + | When 19 has quad /a/ & 13 has quad /b/ { | ||
| + | set a [quad change $a "/ | ||
| + | set b [quad change $b "/ | ||
| + | Wish to draw text with x 800 y 400 text [format {%.2f} [* 100 [norm [sub [quad right $a] [quad left $b]]]]]cm radians 3.14 | ||
| + | } | ||
| + | </ | ||
| + | |||
| + | This past month: | ||
| + | |||
| + | * Many small steps to improve the accuracy of the intrinsic and extrinsic calibrations: | ||
| * Instead of end-to-end refinement using the pose estimate, which is slow, we now separately refine the intrinsic calibrations of the camera and projector, which is the more classic [[https:// | * Instead of end-to-end refinement using the pose estimate, which is slow, we now separately refine the intrinsic calibrations of the camera and projector, which is the more classic [[https:// | ||
| * Then we refine the extrinsics (rotation and translation from camera 3D space -> projector 3D space) as a third distinct step | * Then we refine the extrinsics (rotation and translation from camera 3D space -> projector 3D space) as a third distinct step | ||
| Line 40: | Line 60: | ||
| * Block out skew parameter of cam/proj intrinsics, can assume that it is always 0 (OpenCV does this so we copy them and assume it's OK) | * Block out skew parameter of cam/proj intrinsics, can assume that it is always 0 (OpenCV does this so we copy them and assume it's OK) | ||
| * Instead of averaging extrinsics (3D cam-> | * Instead of averaging extrinsics (3D cam-> | ||
| - | * [[https:// | + | * [[https:// |
| + | * Keeps a tiny bit of state (pose from prev frame) and finds the most likely pose for current frame using tag detection + prev state | ||
| + | * Can get stuck in weird poses but fixes itself if you cover & uncover tag, this is tolerable for now I think | ||
| + | * I really like that this is both pose estimation and ' | ||
| + | * Wonder if we could also teach it about the square tag instead of doing servoing for each point on its own, would that make it more robust | ||
| * Spent a while on performance micro-optimization, | * Spent a while on performance micro-optimization, | ||
| * Spent a while on making a separate correct-extrinsics step that you do at the end of calibration, | * Spent a while on making a separate correct-extrinsics step that you do at the end of calibration, | ||
| Line 55: | Line 79: | ||
| - | 3D calibration works extremely well (1-2mm error) on my system in my apartment. Notice how the outline is almost perfectly aligned with the page: | + | 3D calibration works extremely well (1-2mm error) on my personal |
| - | + | ||
| - | {{newsletters: | + | |
| It works much less well (1-2cm off) on folk0 at Hex House, and even worse on folk-beads (the cart system at Hex House), where the high distortion and shallow 3D volume of the ultra-short-throw projector make it really hard to calibrate. | It works much less well (1-2cm off) on folk0 at Hex House, and even worse on folk-beads (the cart system at Hex House), where the high distortion and shallow 3D volume of the ultra-short-throw projector make it really hard to calibrate. | ||
| Line 63: | Line 85: | ||
| I've also made it almost fully backward-compatible with all existing code -- it now derives 2D projector-space region claims from the 3D calibration, | I've also made it almost fully backward-compatible with all existing code -- it now derives 2D projector-space region claims from the 3D calibration, | ||
| - | I'd like to get it to at least be on par with the existing 2D calibration in normal | + | I'd like to get it to at least be on par with the existing 2D calibration in most cases before we merge it, and we're not there yet... |
| ---- | ---- | ||
| Line 92: | Line 114: | ||
| ---- | ---- | ||
| - | After fixing the Hold bug, we got to test the new evaluator on the actual table on folk0. (It's now mostly reliable and fast enough to be worth testing.) | + | {{: |
| - | It performs //really// well, much faster/ | + | ---- |
| - | In terms of actually being able to fully switch to the new evaluator, the main issues are 1. memory crashes and 2. labels/ | + | After fixing the Hold bug, we got to test the new evaluator on the actual table on folk0. (It's now reliable and fast enough to be worth testing.) |
| + | |||
| + | {{: | ||
| + | |||
| + | It performs //really// well, much faster/ | ||
| + | |||
| + | In terms of actually being able to fully switch to the new evaluator, the main open issues are 1. memory crashes and 2. labels/ | ||
| ==== Portable gadget ==== | ==== Portable gadget ==== | ||
| - | Omar has done a couple [[https:// | + | Omar has done a couple [[https:// |
| {{newsletters: | {{newsletters: | ||
| - | The hand grip and front look good, and the projector and camera and Pi 4 fit well. The hand grip is a little loose, so need rubber or a groove to keep it from swiveling around. The back / wiring isn't fully figured out yet; that also will need a lid/cap that you screw in, I think. I still need to design and wire and do the trigger button, which I think will make a big difference to how you can interact with it. | + | The hand grip and front look good, and the projector and camera and Pi 4 fit well. The hand grip is a little loose, so need rubber or a groove to keep it from swiveling around. |
| I also have issues keeping it powered right now (and I want to figure out a USB power pack situation anyway, where you can choose to use wall power or a power pack). I wonder if the projector is too much for the Pi 4's USB port to power, or if it's overheating, | I also have issues keeping it powered right now (and I want to figure out a USB power pack situation anyway, where you can choose to use wall power or a power pack). I wonder if the projector is too much for the Pi 4's USB port to power, or if it's overheating, | ||
| Line 111: | Line 139: | ||
| - | ==== Small system | + | ==== System |
| * s-ol added FOLK_ERROR macros that can be called from arbitrary C code to immediately return to Tcl and throw a Tcl error | * s-ol added FOLK_ERROR macros that can be called from arbitrary C code to immediately return to Tcl and throw a Tcl error | ||
| * Omar: we've wanted this for a while -- we should go through C code and remove a lot of weird calls to exit() and null returns and replace them with proper Tcl errors now. we've had it at the Tcl-C arg/rtype translation layer (it uses setjmp and longjmp to set a checkpoint each time you call from Tcl into C), but s-ol's generalized it so it's callable from user C code also | * Omar: we've wanted this for a while -- we should go through C code and remove a lot of weird calls to exit() and null returns and replace them with proper Tcl errors now. we've had it at the Tcl-C arg/rtype translation layer (it uses setjmp and longjmp to set a checkpoint each time you call from Tcl into C), but s-ol's generalized it so it's callable from user C code also | ||
| - | * TODO: s-ol folk.js | ||
| - | * TODO: Merged s-ol's work on international keymaps | ||
| * [[https:// | * [[https:// | ||
| * {{newsletters: | * {{newsletters: | ||
| * Naveen added [[https:// | * Naveen added [[https:// | ||
| + | * Merged s-ol's work on international keymaps, so keyboards other than US can work (adds console-data dep) | ||
| * Andrés fixed Alt-Esc when a keyboard hasn't been visible to the system yet (it now loads default keymap out of the box, so Alt-Esc works to restart the system) | * Andrés fixed Alt-Esc when a keyboard hasn't been visible to the system yet (it now loads default keymap out of the box, so Alt-Esc works to restart the system) | ||
| + | === folk.js === | ||
| + | |||
| + | s-ol has added [[https:// | ||
| + | |||
| + | It feels like it can handle disconnect and status reporting and other such conditions a lot better and cleaner than one-off code. It significantly simplifies the implementation of stuff [[https:// | ||
| + | |||
| + | > WS message are executed in the context of a match that is tied to the websocket lifetime. An emit prefix msg proc is provided, and the JS code dispatches messages based on the prefix value. | ||
| + | > | ||
| + | > Utilities for encoding and decoding Tcl strings are provided, including a tagged template literal: | ||
| + | > | ||
| + | > < | ||
| + | const complicatedString = 'This string {contains} some {} odd " | ||
| + | tcl`We ${' | ||
| + | // 'We embed {This string {contains} some {} odd " | ||
| + | |||
| + | const json = [ { name: " | ||
| + | ws.send(tcl`Assert the entities are ${json}`); | ||
| + | // ' | ||
| + | |||
| + | const tclVal = '{name Sol age 27} {name folk age -1}'; | ||
| + | loadList(tclVal).map(loadDict); | ||
| + | // [ { " | ||
| + | </ | ||
| + | |||
| + | It should be very useful and lower the barrier to many phone, laptop, other traditional computing integrations which we've had to do from scratch every time until now (like the work that Arcade has done with tablet and files, that Andrés has done with canvas, & that Omar has done with calibration and CNC preview endpoints). | ||
| ==== Friends and outreach ==== | ==== Friends and outreach ==== | ||
| Line 132: | Line 184: | ||
| ===== What we'll be up to in July ===== | ===== What we'll be up to in July ===== | ||
| - | * **Our next [[https:// | + | * **Our next [[https:// |
| * Getting 3D calibration to work well on all systems, then merging it....... | * Getting 3D calibration to work well on all systems, then merging it....... | ||
| - | * More work on new parallel evaluator | + | * Figuring out retraction and transaction behavior for parallel evaluator; testing it more |
| - | * Gadget iteration | + | * Gadget iteration |
| + | * Video support | ||
| + | * Documentation (language reference, new kits and printable worksheets for setting up and learning the Folk language), open up GitHub repo | ||
| Line 157: | Line 211: | ||
| * [[https:// | * [[https:// | ||
| * [[https:// | * [[https:// | ||
| + | * [[https:// | ||
| + | * [[https:// | ||
newsletters/2024-06.1719608755.txt.gz · Last modified: by osnr