Jacob Haip, April 2024

I am interested in integrating more physical hardware and sensors into Folk Computer to expand computing to the size of the room and to take advantage of the affordances of hardware that the projected AR system alone can't do. Things like sensing the noise level of the environment, motors that spin a fan and create wind, and the physical feel of a knob that clicks when turning off.

WiFi-connected microcontrollers are the primary tool I have been using to integrate sensors and actuators with Folk. Boards I have used include the Adafruit Feather ESP32, RPi Pico W, and the Particle Photon. They are either programmed using Arduino or MicroPython. I have also integrated microcontrollers using BLE or a wired serial connection but I tend to like WiFi for long-lived stable devices embedded in my space.

Folk has the ability to create arbitrary HTTP endpoints and a Websocket connection that proxies through Folk Statements, but neither are currently bidirectional and both are heavy protocols for microcontrollers to send or get data from Folk.

Lately I have stated using MQTT as the protocol for microcontrollers to talk to the Folk System. It is a lighter weight Pub/Sub messaging protocol that is primarily done on top of a TCP connection. Microcontrollers talk to a MQTT Broker running alongside Folk and then a Folk program talks to the broker as well.

I am using the Mosquitto MQTT Broker which runs on port 1883.

sudo apt install mosquitto mosquitto-clients -y
sudo systemctl status mosquitto
sudo systemctl start mosquitto
sudo systemctl enable mosquitto
mosquitto_sub -h address -t topic
mosquitto_pub -h address -t topic -m “message”

The bridge from the MQTT broker to Folk is the mqtt.folk program which adds support for wishes like Wish MQTT publish $message on topic $feed at timestamp $timestamp and When MQTT claims message /message/ topic /topic/ timestamp /t/ { ... }.

mqtt.folk

# This program depends on the https://chiselapp.com/user/schelte/repository/mqtt/home MQT library being installed at the /home/folk/mqtt directory
Start process mqtt {
    Wish $::thisProcess receives statements like \
        [list /someone/ wishes MQTT publish /...anything/]
    Wish $::thisProcess receives statements like \
        [list $::thisNode has step count /...anything/]
    Wish $::thisProcess shares statements like \
        [list MQTT claims message /message/ topic /topic/ timestamp /timestamp/]
 
    ::tcl::tm::path add /home/folk/mqtt
    package require mqtt
 
    set client [mqtt new]
    set ::mqttclient $client
 
    proc cb {topic content status} {
        set message [encoding convertfrom utf-8 $content]
        puts "MQTT CALLBACK: $topic: $message"
        set now [clock milliseconds]
        Assert MQTT claims message $message topic $topic timestamp $now

        # Retract all events that are more than 5 seconds old.
        set events [Statements::findMatches [list MQTT claims message /message/ topic /topic/ timestamp /timestamp/]]
        foreach event $events {
            dict with event {
                if {$now - $timestamp > 5000} {
                    Retract MQTT claims message $message topic $topic timestamp $timestamp
                }
            }
        }
    }
    $client connect test-client localhost 1883
    $client subscribe "#" cb
 
    When /someone/ wishes MQTT publish /m/ on topic /t/ at timestamp /ts/ {
        # puts "publishing $m on $t at $ts"
        $::mqttclient publish "$t" "$m"
    }
 
    while true {
        # Step for folk, update for TCL and MQTT
        Step
        update
    }
}

rpi-pico-w-micropython-button-press.py

from machine import Pin, I2C
import network
import utime
import time
from umqtt.simple import MQTTClient
 
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.config(pm = 0xa11140) # Diable powersave mode
wlan.connect("XXX", "XXX")
 
max_wait = 10
while max_wait > 0:
    if wlan.status() < 0 or wlan.status() >= 3:
        break
    max_wait -= 1
    print('waiting for connection...')
    utime.sleep(1)
 
#Handle connection error
if wlan.status() != 3:
    raise RuntimeError('wifi connection failed')
else:
    print('connected')
    status = wlan.ifconfig()
    print('ip = ' + status[0])
 
def connectMQTT():
    client = MQTTClient(client_id=b"kudzai_raspberrypi_picow",
        server=b"192.168.1.34",
        port=1883,
    )
    client.connect()
    return client
 
def reconnect():
    print('Failed to connect to the MQTT Broker. Reconnecting...')
    time.sleep(5)
    machine.reset()
 
client = connectMQTT()
MQTT_BUTTON1_TOPIC = "pico/button1"
button1 = Pin(16, Pin.IN, Pin.PULL_UP)
button1_last_time = time.ticks_ms()
 
def mqtt_send_with_reset(topic, msg):
    try:
        client.publish(topic, msg)
    except:
        import sys
        print(sys.print_exception(e))
        machine.reset()
 
def publish_mqtt_button1_msg(t):
    global button1_last_time
    if time.ticks_diff(time.ticks_ms(), button1_last_time) > 200:
        mqtt_send_with_reset(MQTT_BUTTON1_TOPIC, "pressed")
        button1_last_time = time.ticks_ms()    
 
button1.irq(publish_mqtt_button1_msg, Pin.IRQ_RISING )
 
while True:
    mqtt_send_with_reset("pico/test", "hello")
    print("publish")
    utime.sleep(30)

show-button-press.folk

When MQTT claims message /message/ topic "pico/button1" timestamp /t/ {
   Wish $this is labelled "BUTTON 1 PRESSED"
}

My favorite demo of this is a set of 4 buttons that my toddler can press to play a song on Spotify.

spotify-button-press.folk

Claim button 1 uri is "spotify:track:5Ea0sJ11pTMxnVEBWbyiYq"
Claim button 2 uri is "spotify:track:2yzshFeBIwH8tWIqHEFLeD"
Claim button 3 uri is "spotify:track:5ygDXis42ncn6kYG14lEVG"
Claim button 4 uri is "spotify:track:5kIPbUWHrRL98Sd4OckvAn"
 
When MQTT claims message /message/ topic "pico/button1" timestamp /t/ & button 1 uri is /uri/ {
  Wish spotify plays uri $uri at time $t
}
When MQTT claims message /message/ topic "pico/button2" timestamp /t/ & button 2 uri is /uri/ {
  Wish spotify plays uri $uri at time $t
}
When MQTT claims message /message/ topic "pico/button3" timestamp /t/ & button 3 uri is /uri/ {
  Wish spotify plays uri $uri at time $t
}
When MQTT claims message /message/ topic "pico/button4" timestamp /t/ & button 4 uri is /uri/ {
  Wish spotify plays uri $uri at time $t
}

I have this cool Adafruit MatrixPortal M4 Pixel Screen that can be programmed with Arduino. I set it up to listen on the /home/matrixportal MQTT topic where a message is a new screen to display.

matrix-portal.c

#include <SPI.h>
#include <WiFiNINA.h>
#include <Adafruit_Protomatter.h>
#include <PubSubClient.h>
 
char ssid[] = "XXXX";   // your network SSID (name)
char pass[] = "XXXX"; // your network password (use for WPA, or use as key for WEP)
 
const char* mqttServer = "192.168.1.34";
const int mqttPort = 1883;
const char* mqttTopic = "/home/matrixportal";
 
WiFiClient espClient;
PubSubClient client(espClient);
 
uint8_t rgbPins[] = {7, 8, 9, 10, 11, 12};
uint8_t addrPins[] = {17, 18, 19, 20};
uint8_t clockPin = 14;
uint8_t latchPin = 15;
uint8_t oePin = 16;
 
Adafruit_Protomatter matrix(
    64, 4, 1, rgbPins, 4, addrPins, clockPin, latchPin, oePin, false);
 
uint64_t pixelPalette[] = {
    matrix.color565(0, 0, 0),
    matrix.color565(157, 157, 157),
    matrix.color565(255, 255, 255),
    matrix.color565(190, 38, 51),
    matrix.color565(224, 111, 139),
    matrix.color565(73, 60, 43),
    matrix.color565(164, 100, 34),
    matrix.color565(235, 137, 49),
    matrix.color565(247, 226, 107),
    matrix.color565(47, 72, 78),
    matrix.color565(68, 137, 26),
    matrix.color565(163, 206, 39),
    matrix.color565(27, 38, 50),
    matrix.color565(0, 87, 132),
    matrix.color565(49, 162, 242),
    matrix.color565(178, 220, 239),
    matrix.color565(255, 0, 255)};
 
 
//--------- WIFI -------------------------------------------
 
void wifi_connect() {
  Serial.print("Starting connecting WiFi.");
  delay(10);
  WiFi.begin(ssid, pass);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}
 
//------------------ MQTT ----------------------------------
void mqtt_setup() {
  client.setServer(mqttServer, mqttPort);
  client.setBufferSize(64*32 + 64);
  client.setCallback(callback);
  Serial.println("Connecting to MQTT…");
  while (!client.connected()) {        
      String clientId = "ESP32Client-";
      clientId += String(random(0xffff), HEX);
      if (client.connect(clientId.c_str())) {
          Serial.println("connected");
      } else {
          Serial.print("failed with state  ");
          Serial.println(client.state());
          delay(2000);
      }
  }
 
  client.subscribe(mqttTopic);
}
 
void callback(char* topic, byte* payload, unsigned int length) {
 
    Serial.print("Message arrived in topic: ");
    Serial.println(topic);
 
    String byteRead = "";
    Serial.print("Message: ");
    for (int i = 0; i < length; i++) {
        byteRead += (char)payload[i];
    }    
    Serial.println(byteRead);
 
    matrix.fillScreen(matrix.color565(0, 0, 0));
 
    for (int i = 0; i < byteRead.length(); i++)
    {
        uint64_t pixelColor = matrix.color565(0, 0, 0);
        String pixelChar = String(byteRead[i]);
        int colorIndex = (int)strtol(&pixelChar[0], NULL, 16);
        if (colorIndex >= 0 && colorIndex < 16)
        {
            pixelColor = pixelPalette[colorIndex];
        }
        matrix.drawPixel(i % 64, i / 64, pixelColor);
    }
    matrix.show(); // Copy data to matrix buffers
}
 
 
void setup()
{
    Serial.begin(9600);
 
    // Initialize matrix...
    ProtomatterStatus pmstatus = matrix.begin();
    Serial.print("Protomatter begin() status: ");
    Serial.println((int)pmstatus);
    if (pmstatus != PROTOMATTER_OK)
    {
        for (;;)
            ;
    }
 
    matrix.println("wifi..."); // Default text color is white
    matrix.show();                   // Copy data to matrix buffers
 
    wifi_connect();
 
    matrix.println("mqtt..."); // Default text color is white
    matrix.show();                   // Copy data to matrix buffers
 
    mqtt_setup();  
}
 
void loop()
{
    client.loop();
}

And with this you can make a crude clock that fills up the screen horizontally as time passes.

crude-matrixportal-clock.folk

When /node/ has step count /c/ {
  if {[expr {int($c) % 1000}] == 0} {
    set currentTimeMillis [clock milliseconds]
    set millisSinceMinuteStart [expr {$currentTimeMillis % 60000}]
    set fillValue [expr {$millisSinceMinuteStart * 64 / 60000}]
 
    set matrixportaloutput ""
    for {set y 0} {$y < 32} {incr y} {
      for {set x 0} {$x < 64} {incr x} {
        if {$x < $fillValue} {
          set matrixportaloutput "9$matrixportaloutput"
        } else {
          set matrixportaloutput "0$matrixportaloutput"
        }
      }
    }
    set c [clock milliseconds]
    Wish MQTT publish "$matrixportaloutput" on topic /home/matrixportal at timestamp $currentTimeMillis
  }
}

Additionally, you can make a web app that also speaks to the MQTT Broker using MQTT.js. In this case, the Folk system actually isn't used at all because all the logic is within the Javascript of the web page to edit the pixels and send MQTT messages.

Making simple web pages using the MQTT.js library also bring bidirectional communication to Folk. In the current state of the Folk code, you had to long poll for updates but using MQTT like this is nice because you don't have to poll. I like these basic web apps as a way to bring phones and laptops into the Folk system as new inputs and outputs.

First we make a web app that plays a sound when it receives an MQTT messages:

play-sound.html

<html>
  <head><title>Websocket client</title></head>
  <body>
    <h1>Plays a sound when /web/playsound receives a MQTT message</h1>
    <audio id="myAudio" controls preload="preload">
        <source src="https://www.w3schools.com/html/horse.ogg" type="audio/ogg">
        <source src="https://www.w3schools.com/html/horse.mp3" type="audio/mpeg">
        Your browser does not support the audio element.
    </audio>
    <script src="https://unpkg.com/mqtt/dist/mqtt.min.js"></script>
    <script>
const clientId = 'mqttjs_' + Math.random().toString(16).substr(2, 8)
const host = 'ws://192.168.1.34.local:8080'
const options = {
  keepalive: 60,
  clientId: clientId,
  protocolId: 'MQTT',
  protocolVersion: 4,
  clean: true,
  reconnectPeriod: 1000,
  connectTimeout: 30 * 1000,
  will: {
    topic: 'WillMsg',
    payload: 'Connection Closed abnormally..!',
    qos: 0,
    retain: false
  },
}
console.log('Connecting mqtt client')
const client = mqtt.connect(host, options)
 
client.on('error', (err) => {
  console.log('Connection error: ', err)
  client.end()
})
 
client.on('reconnect', () => {
  console.log('Reconnecting...')
})
 
client.on('connect', () => {
  console.log('Client connected:' + clientId)
  client.subscribe('/web/playsound', { qos: 0 })
});
 
client.on('message', (topic, message, packet) => {
  console.log('Received Message: ' + message.toString() + '\nOn topic: ' + topic)
  document.getElementById("myAudio").currentTime = 0;
  document.getElementById("myAudio").play();
});
</script>
  </body>
</html>

And then in folk you can make every phone and laptop with that web page open play a sound when a button is pressed:

trigger-sound.folk

When MQTT claims message /message/ topic "/button1" timestamp /t/ {
   Wish $this is labelled "BUTTON 1 PRESSED"
   set ts [clock milliseconds]
   Wish MQTT publish "make some noise" on topic "/web/playsound" at timestamp $ts
}