Differences

This shows you the differences between two versions of the page.

--- notes:mqtt [2024/04/18 01:46] – add architecture image discord
+++ notes:mqtt [2024/04/20 20:46] (current) – discord
@@ Line 1: / Line 1: @@
-Written by Jacob Haip on April 17, 2024
+====== MQTT and Microcontroller Folk Integrations ======
+Jacob Haip, April 2024
-I am interested in integrating more physical hardware and sensors into Folk Computer to help expand the scope to the full room and to take advantage of the affordances of hardware that the projected AR system alone can't do. Things like sensing the noise level of the environment, motors that spin a fan and create wind, and the physical feel of a knob that clicks when turning off.
+I am interested in integrating more physical hardware and sensors into Folk Computer to expand computing to the size of the room and to take advantage of the affordances of hardware that the projected AR system alone can't do. Things like sensing the noise level of the environment, motors that spin a fan and create wind, and the physical feel of a knob that clicks when turning off.
 WiFi-connected microcontrollers are the primary tool I have been using to integrate sensors and actuators with Folk. Boards I have used include the [[https://www.adafruit.com/product/5400|Adafruit Feather ESP32]], [[https://www.adafruit.com/product/5526|RPi Pico W]], and the [[https://store.particle.io/products/photon-2|Particle Photon]]. They are either programmed using Arduino or MicroPython. I have also integrated microcontrollers using BLE or a wired serial connection but I tend to like WiFi for long-lived stable devices embedded in my space.
@@ Line 72: / Line 73: @@
 </code>
-=== Example code using the Adafruit MagTag E-Ink Microcontroller ===
+===== Getting a button input into Folk =====
-[[https://www.adafruit.com/product/4800|Adafruit MagTag]] MicroPython Code that displays texts messages received on the MQTT /magtag/message topic, and publishes a heartbeat message on the MQTT /feeds/test topic.
+{{micro-button-press.gif}}
-<code python magtag-micropython.py>
-import os
+<code c rpi-pico-w-micropython-button-press.py>
+from machine import Pin, I2C
+import network
+import utime
 import time
-import ssl
+from umqtt.simple import MQTTClient
-import socketpool
-import wifi
-import adafruit_minimqtt.adafruit_minimqtt as MQTT
-from adafruit_magtag.magtag import MagTag
-magtag = MagTag()
+wlan = network.WLAN(network.STA_IF)
-magtag.add_text(
+wlan.active(True)
-    text_position=(
+wlan.config(pm = 0xa11140) # Diable powersave mode
-,
+wlan.connect("XXX", "XXX")
-        (magtag.graphics.display.height // 2) - 1,
-    ),
-    text_scale=3,
-)
-magtag.set_text("Hello World")
-print(f"Connecting to {os.getenv('CIRCUITPY_WIFI_SSID')}")
+max_wait = 10
-print(f"Connecting to {os.getenv('CIRCUITPY_WIFI_PASSWORD')}")
+while max_wait > 0:
-wifi.radio.connect(os.getenv("CIRCUITPY_WIFI_SSID"), os.getenv("CIRCUITPY_WIFI_PASSWORD"))
+    if wlan.status() < 0 or wlan.status() >= 3:
-print(f"Connected to {os.getenv('CIRCUITPY_WIFI_SSID')}!")
+        break
+    max_wait -= 1
+    print('waiting for connection...')
+    utime.sleep(1)
-sending_feed = "/feeds/test"
+#Handle connection error
-message_feed = "/magtag/message"
+if wlan.status() != 3:
+    raise RuntimeError('wifi connection failed')
+else:
+    print('connected')
+    status = wlan.ifconfig()
+    print('ip = ' + status[0])
-def connected(client, userdata, flags, rc):
+def connectMQTT():
-    client.subscribe(message_feed)
+    client = MQTTClient(client_id=b"kudzai_raspberrypi_picow",
+        server=b"192.168.1.34",
+        port=1883,
+    )
+    client.connect()
+    return client
-def disconnected(client, userdata, rc):
+def reconnect():
-    print("Disconnected")
+    print('Failed to connect to the MQTT Broker. Reconnecting...')
+    time.sleep(5)
+    machine.reset()
-def message(client, topic, message):
+client = connectMQTT()
-    print(f"New message on topic {topic}: {message}")
+MQTT_BUTTON1_TOPIC = "pico/button1"
-    magtag.set_text(message)
+button1 = Pin(16, Pin.IN, Pin.PULL_UP)
+button1_last_time = time.ticks_ms()
-pool = socketpool.SocketPool(wifi.radio)
+def mqtt_send_with_reset(topic, msg):
-ssl_context = ssl.create_default_context()
+    try:
-mqtt_client = MQTT.MQTT(
+        client.publish(topic, msg)
-    broker="folk-haip.local",
+    except:
-    port=1883,
+        import sys
-    socket_pool=pool,
+        print(sys.print_exception(e))
-    ssl_context=ssl_context,
+        machine.reset()
-)
-# Setup the callback methods above
+def publish_mqtt_button1_msg(t):
-mqtt_client.on_connect = connected
+    global button1_last_time
-mqtt_client.on_disconnect = disconnected
+    if time.ticks_diff(time.ticks_ms(), button1_last_time) > 200:
-mqtt_client.on_message = message
+        mqtt_send_with_reset(MQTT_BUTTON1_TOPIC, "pressed")
+        button1_last_time = time.ticks_ms()
-# Connect the client to the MQTT broker.
+button1.irq(publish_mqtt_button1_msg, Pin.IRQ_RISING )
-print("Connecting...")
-mqtt_client.connect()
-test_val = 0
 while True:
-    # Poll the message queue
+    mqtt_send_with_reset("pico/test", "hello")
-    mqtt_client.loop(timeout=1)
+    print("publish")
+    utime.sleep(30)
+</code>
-    # Send a new message
+<code tcl show-button-press.folk>
-    print(f"Sending value: {test_val}...")
+When MQTT claims message /message/ topic "pico/button1" timestamp /t/ {
-    mqtt_client.publish(sending_feed, test_val)
+   Wish $this is labelled "BUTTON 1 PRESSED"
-    print("Sent!")
+}
-    test_val += 1
 </code>
-A folk program to make a crude clock on the MagTag display:
+My favorite demo of this is a set of 4 buttons that my toddler can press to play a song on Spotify.
-<code tcl crude-clock.folk>
+{{button-kid-press.jpeg}}
+<code tcl spotify-button-press.folk>
+Claim button 1 uri is "spotify:track:5Ea0sJ11pTMxnVEBWbyiYq"
+Claim button 2 uri is "spotify:track:2yzshFeBIwH8tWIqHEFLeD"
+Claim button 3 uri is "spotify:track:5ygDXis42ncn6kYG14lEVG"
+Claim button 4 uri is "spotify:track:5kIPbUWHrRL98Sd4OckvAn"
+When MQTT claims message /message/ topic "pico/button1" timestamp /t/ & button 1 uri is /uri/ {
+  Wish spotify plays uri $uri at time $t
+}
+When MQTT claims message /message/ topic "pico/button2" timestamp /t/ & button 2 uri is /uri/ {
+  Wish spotify plays uri $uri at time $t
+}
+When MQTT claims message /message/ topic "pico/button3" timestamp /t/ & button 3 uri is /uri/ {
+  Wish spotify plays uri $uri at time $t
+}
+When MQTT claims message /message/ topic "pico/button4" timestamp /t/ & button 4 uri is /uri/ {
+  Wish spotify plays uri $uri at time $t
+}
+</code>
+===== Output from Folk to Microcontroller =====
+I have this cool [[https://www.adafruit.com/product/4745|Adafruit MatrixPortal M4]] Pixel Screen that can be programmed with Arduino. I set it up to listen on the /home/matrixportal MQTT topic where a message is a new screen to display.
+<code c matrix-portal.c>
+#include <SPI.h>
+#include <WiFiNINA.h>
+#include <Adafruit_Protomatter.h>
+#include <PubSubClient.h>
+char ssid[] = "XXXX";   // your network SSID (name)
+char pass[] = "XXXX"; // your network password (use for WPA, or use as key for WEP)
+const char* mqttServer = "192.168.1.34";
+const int mqttPort = 1883;
+const char* mqttTopic = "/home/matrixportal";
+WiFiClient espClient;
+PubSubClient client(espClient);
+uint8_t rgbPins[] = {7, 8, 9, 10, 11, 12};
+uint8_t addrPins[] = {17, 18, 19, 20};
+uint8_t clockPin = 14;
+uint8_t latchPin = 15;
+uint8_t oePin = 16;
+Adafruit_Protomatter matrix(
+, 4, 1, rgbPins, 4, addrPins, clockPin, latchPin, oePin, false);
+uint64_t pixelPalette[] = {
+    matrix.color565(0, 0, 0),
+    matrix.color565(157, 157, 157),
+    matrix.color565(255, 255, 255),
+    matrix.color565(190, 38, 51),
+    matrix.color565(224, 111, 139),
+    matrix.color565(73, 60, 43),
+    matrix.color565(164, 100, 34),
+    matrix.color565(235, 137, 49),
+    matrix.color565(247, 226, 107),
+    matrix.color565(47, 72, 78),
+    matrix.color565(68, 137, 26),
+    matrix.color565(163, 206, 39),
+    matrix.color565(27, 38, 50),
+    matrix.color565(0, 87, 132),
+    matrix.color565(49, 162, 242),
+    matrix.color565(178, 220, 239),
+    matrix.color565(255, 0, 255)};
+//--------- WIFI -------------------------------------------
+void wifi_connect() {
+  Serial.print("Starting connecting WiFi.");
+  delay(10);
+  WiFi.begin(ssid, pass);
+  while (WiFi.status() != WL_CONNECTED) {
+    delay(500);
+    Serial.print(".");
+  }
+  Serial.println("WiFi connected");
+  Serial.println("IP address: ");
+  Serial.println(WiFi.localIP());
+}
+//------------------ MQTT ----------------------------------
+void mqtt_setup() {
+  client.setServer(mqttServer, mqttPort);
+  client.setBufferSize(64*32 + 64);
+  client.setCallback(callback);
+  Serial.println("Connecting to MQTT…");
+  while (!client.connected()) {
+      String clientId = "ESP32Client-";
+      clientId += String(random(0xffff), HEX);
+      if (client.connect(clientId.c_str())) {
+          Serial.println("connected");
+      } else {
+          Serial.print("failed with state  ");
+          Serial.println(client.state());
+          delay(2000);
+      }
+  }
+  client.subscribe(mqttTopic);
+}
+void callback(char* topic, byte* payload, unsigned int length) {
+    Serial.print("Message arrived in topic: ");
+    Serial.println(topic);
+    String byteRead = "";
+    Serial.print("Message: ");
+    for (int i = 0; i < length; i++) {
+        byteRead += (char)payload[i];
+    }
+    Serial.println(byteRead);
+    matrix.fillScreen(matrix.color565(0, 0, 0));
+    for (int i = 0; i < byteRead.length(); i++)
+    {
+        uint64_t pixelColor = matrix.color565(0, 0, 0);
+        String pixelChar = String(byteRead[i]);
+        int colorIndex = (int)strtol(&pixelChar[0], NULL, 16);
+        if (colorIndex >= 0 && colorIndex < 16)
+        {
+            pixelColor = pixelPalette[colorIndex];
+        }
+        matrix.drawPixel(i % 64, i / 64, pixelColor);
+    }
+    matrix.show(); // Copy data to matrix buffers
+}
+void setup()
+{
+    Serial.begin(9600);
+    // Initialize matrix...
+    ProtomatterStatus pmstatus = matrix.begin();
+    Serial.print("Protomatter begin() status: ");
+    Serial.println((int)pmstatus);
+    if (pmstatus != PROTOMATTER_OK)
+    {
+        for (;;)
+            ;
+    }
+    matrix.println("wifi..."); // Default text color is white
+    matrix.show();                   // Copy data to matrix buffers
+    wifi_connect();
+    matrix.println("mqtt..."); // Default text color is white
+    matrix.show();                   // Copy data to matrix buffers
+    mqtt_setup();
+}
+void loop()
+{
+    client.loop();
+}
+</code>
+And with this you can make a crude clock that fills up the screen horizontally as time passes.
+{{crude-clock.gif}}
+<code tcl crude-matrixportal-clock.folk>
 When /node/ has step count /c/ {
-  if {[expr {int($c) % 5000}] == 0} {
+  if {[expr {int($c) % 1000}] == 0} {
-    set ts [clock milliseconds]
+    set currentTimeMillis [clock milliseconds]
-    Wish MQTT publish $ts on topic jhaip/feeds/onoff at timestamp $ts
+    set millisSinceMinuteStart [expr {$currentTimeMillis % 60000}]
+    set fillValue [expr {$millisSinceMinuteStart * 64 / 60000}]
+    set matrixportaloutput ""
+    for {set y 0} {$y < 32} {incr y} {
+      for {set x 0} {$x < 64} {incr x} {
+        if {$x < $fillValue} {
+          set matrixportaloutput "9$matrixportaloutput"
+        } else {
+          set matrixportaloutput "0$matrixportaloutput"
+        }
+      }
+    }
+    set c [clock milliseconds]
+    Wish MQTT publish "$matrixportaloutput" on topic /home/matrixportal at timestamp $currentTimeMillis
   }
 }
 </code>
-A folk program that shows the latest published message from the MagTag device:
+Additionally, you can make a web app that also speaks to the MQTT Broker using [[https://github.com/mqttjs/MQTT.js|MQTT.js]]. In this case, the Folk system actually isn't used at all because all the logic is within the Javascript of the web page to edit the pixels and send MQTT messages.
+{{matrixportal-live-edit.gif}}
+===== Using MQTT on web pages for bidirectional communication with Folk =====
+Making simple web pages using the [[https://github.com/mqttjs/MQTT.js|MQTT.js]] library also bring bidirectional communication to Folk. In the current state of the Folk code, you had to long poll for updates but using MQTT like this is nice because you don't have to poll. I like these basic web apps as a way to bring phones and laptops into the Folk system as new inputs and outputs.
+First we make a web app that plays a sound when it receives an MQTT messages:
+<code html play-sound.html>
+<html>
+  <head><title>Websocket client</title></head>
+  <body>
+    <h1>Plays a sound when /web/playsound receives a MQTT message</h1>
+    <audio id="myAudio" controls preload="preload">
+        <source src="https://www.w3schools.com/html/horse.ogg" type="audio/ogg">
+        <source src="https://www.w3schools.com/html/horse.mp3" type="audio/mpeg">
+        Your browser does not support the audio element.
+    </audio>
+    <script src="https://unpkg.com/mqtt/dist/mqtt.min.js"></script>
+    <script>
+const clientId = 'mqttjs_' + Math.random().toString(16).substr(2, 8)
+const host = 'ws://192.168.1.34.local:8080'
+const options = {
+  keepalive: 60,
+  clientId: clientId,
+  protocolId: 'MQTT',
+  protocolVersion: 4,
+  clean: true,
+  reconnectPeriod: 1000,
+  connectTimeout: 30 * 1000,
+  will: {
+    topic: 'WillMsg',
+    payload: 'Connection Closed abnormally..!',
+    qos: 0,
+    retain: false
+  },
+}
+console.log('Connecting mqtt client')
+const client = mqtt.connect(host, options)
+client.on('error', (err) => {
+  console.log('Connection error: ', err)
+  client.end()
+})
+client.on('reconnect', () => {
+  console.log('Reconnecting...')
+})
+client.on('connect', () => {
+  console.log('Client connected:' + clientId)
+  client.subscribe('/web/playsound', { qos: 0 })
+});
+client.on('message', (topic, message, packet) => {
+  console.log('Received Message: ' + message.toString() + '\nOn topic: ' + topic)
+  document.getElementById("myAudio").currentTime = 0;
+  document.getElementById("myAudio").play();
+});
+</script>
+  </body>
+</html>
+</code>
+And then in folk you can make every phone and laptop with that web page open play a sound when a button is pressed:
+{{button-horse.mp4}}
-<code tcl show-message.folk>
+<code tcl trigger-sound.folk>
-When MQTT claims message /message/ topic "/feeds/test" timestamp /t/ {
+When MQTT claims message /message/ topic "/button1" timestamp /t/ {
-   Wish $this is labelled "MagTag value: $message"
+   Wish $this is labelled "BUTTON 1 PRESSED"
+   set ts [clock milliseconds]
+   Wish MQTT publish "make some noise" on topic "/web/playsound" at timestamp $ts
 }
 </code>