Our next Folk open house is in the evening on Thursday, February 26th, in East Williamsburg, Brooklyn.

• Daniel Pipkin fixed our ESC/POS receipt printer support for folk2: now it uses fn and Subscribe:/Notify: instead of globals
• Esben Sørig contributed some camera/setup fixes:
  • Extend camera load timeout to 2 seconds and add retry mechanism, which seems to fix the Logitech C270 camera
    • Also don't die on non-fatal JPEG decompress errors
  • Make aspect ratio of camera preview on /calibrate adapt to actual camera aspect ratio
• Andrés and Brian worked on the shape library motivated by ideas for drawing shapes to highlight objects on the table
  • 
• Discussions from the Folk Discord:
  • Preview of the simplified shape code working in folk2 to draw a square on a quad
  • Hamish Todd delivered a version his presentation on his setup and perspective experiments in Folk in the Discord
  • TJoh, Paul, and Mason discussed implementing physical “knobs” using tag angles; confirmed that regions are deprecated in favor of quads in folk2, and discussed the math for extracting angles from quads
  • Richard asked about persistent memory for programs not on the table; Mason pointed to the Hold! -save functionality in folk2
  • Esben Sørig kicked off a thread on 4K camera recommendations, with Audrey providing notes on ELP camera compatibility
  • doodr shared progress on a custom “dotframe” encoding/decoding system built with OpenCV
  • Omar suggested implementing automatic ephemeral temp folders for programs to more cleanly handle downloads and Unix dependencies

Omar: I've been wanting to do non-AprilTag inputs for a while, especially handwriting input (see proposal from last year, for example).

I started looking at OCR/handwriting recognition more seriously a couple of months ago, and I concluded that TrOCR was probably the right building block (AI model) for this. It runs pretty fast (0.1s to 1s) on most systems on GPU and CPU, I was able to run it locally, it's trained for handwriting and not print. I did some prototyping in Python for that.

Now that we have folk2, it's a lot easier to run synchronous tasks like handwriting recognition without blocking the rest of Folk, so I want to finally get stuff like this deeply integrated (and have the integration be pretty simple and idiomatic) and prototype how to interact with it.

So at the end of January, I finally started hacking on a TrOCR integration that runs inside Folk, so we can actually write on the table and get live text output.

Here's a draft trocr.folk that exports a callable function (more verbose / AI-y than it probably should be, but it's fully self-contained, which is great).

Easy invocation on a static on-disk image (to use live, put it in an infinite loop and query for camera slice):

When the image library is / imageLib/ &\
     the TrOCR function is /TrOCR/ {
  fn TrOCR
  set im [$imageLib loadJpeg "a01-122-02.jpg"]
  puts [TrOCR $im]
}

It runs really fast on folk-hex when CUDA is on (maybe 10x faster, multiple frames per second). That liveness is fun to play with.

Excited to show more demos around this next month.

A lot of this is built on uvx, which I just learned about this month. uvx gives us the ability to run Python with arbitrary dependencies in 'immediate mode', which aligns a lot better with Folk (wanting self-contained declarative Folk programs, wanting to be able to change them to do anything at any time, etc) than a traditional project/env-oriented Python workflow would. You just run uvx with all dependencies as arguments and then can eval any Python you want, and it automatically caches the dependencies so it doesn't have to reinstall from scratch every time.

Some issues to resolve:

• TrOCR really can't cope with any padding around the handwriting; it needs a tight box around the text, so we need a separate text detection model to maximize usability (otherwise you have to write your word to exactly fill the camera slice, which is sort of annoying, especially if the slice is reasonably large)
• I think we'll need to push harder on finding and keeping a really good calibration if we want to be able to project onto/alongside handwriting
• I also want to figure out a camera slice change detection system so we don't have to run the OCR model continuously

I started experimenting with the CRAFT text detector, which seems to be what people generally pair with TrOCR.

I tuned it to use the GPU and to be more aggressive in linking regions, so we capture the whole “wow” in one box here:

Here's the draft code that does both CRAFT and TrOCR, which I haven't tested on a table (I'm now working on a more general Python FFI so that the OCR pipeline isn't one big monolith, and so we can try other models easily).

Omar:

TODO: table one

• We had a small open house on Thursday, January 29:
  • Open house visitors going through the new Folk introductory booklet:
    • 

• Our next Folk open house is in the evening on Thursday, February 26th, in East Williamsburg, Brooklyn.
• Andrés: Finishing shape library
• Andrés: Working on restoring and making demos with the dot detector capability
• Omar: Make and post some CRAFT+TrOCR demos; publish Python FFI; maybe start playing with segmentation
• Omar: Fix some bugs with folk2 (texture blinking, leaks)
• Omar: Maybe try and start on stereo calibration and/or calibration refinement

• https://old.reddit.com/r/electronics/comments/4r7euc/those_who_say_surface_mount_is_harder_than/
• https://forums.raspberrypi.com/viewtopic.php?t=376578
• https://www.delioa.com/products/es228w-1/
• https://electronics.stackexchange.com/questions/466185/how-to-calculate-the-size-of-ground-plane-we-need-to-use-in-a-pcb-printed-circu
• https://old.reddit.com/r/ryobi/comments/oxn53g/psa_ryobi_is_using_chinese_18650_cells_in_most_of/
• http://harmonyzone.org/blog/posts/dog%20mural/
• https://www.gnu.org/software/emacs/manual/html_mono/transient.html
• https://xkcd.com/color/rgb/
• https://www.youtube.com/watch?v=9ATpyhBWoPk

• A large collection of PDFs from Xerox PARC
• Diagrams Matter by Stan Allen
• Moondream a tiny vision + language model for running on things like Raspberry Pis