Chapter 11: Magic Wand

Application Structure

Arduino Nano 33 BLE Sense board are equipped with three-axis (x, y, and z-axes) accelerometers. This chapter uses the accelerometer to detect the digits 0-9.

The model's output will undergo some further logic (based on threshold and the number of consecutive positive predictions) to determine the final output. The final output will be printed to the serial monitor.

Model Architecture

Our model is a convolutional neural network that takes in 128 sets of x, y, and z values sampled at 25 Hz (i.e., about 5 seconds of data) and outputs probability scores for four classes: one representing each digit.

Arduino Code

This sketch integrates various components like IMU (Inertial Measurement Unit), BLE (Bluetooth Low Energy), and TensorFlow Lite Micro for gesture recognition.

Initialization (setup() function)

  1. Serial Communication: Initializes serial communication for debugging.
  2. IMU: Starts the Inertial Measurement Unit to read acceleration and gyroscope data.
  3. BLE: Sets up Bluetooth Low Energy with specific characteristics and services.
  4. TensorFlow Lite Micro: Sets up TensorFlow Lite model interpreter, allocates tensors, and ensures the model is compatible.

Main Loop (loop() function)

  1. BLE Connection: Checks if a BLE central device is connected.
  2. IMU Data: Reads and processes acceleration and gyroscope data.
  3. Gesture Recognition:
    • Waits for a gesture to be complete (done_just_triggered flag).
    • Rasterizes the gesture to create an image.
    • Passes the image to the TensorFlow Lite model for inference.
    • Parses the output to identify the gesture.

Important Variables

  • tensor_arena: Memory pool for TensorFlow model tensors.
  • raster_buffer: Buffer to store the rasterized image.
  • labels: Possible gestures the model can recognize.

Libraries and Namespaces

  • Various TensorFlow Lite Micro libraries are included for model interpretation.
  • A namespace wraps constant and variable declarations to prevent global scope pollution.

BLE Specifics

  • A BLE service and characteristic are defined for transferring stroke data (strokeCharacteristic).

Error Handling

  • Checks and error messages are included, which will output through the serial interface in case of failures.

IMU Provider

imu_provider.h is a collection of helper functions that read and process data from the IMU sensor. But before we get into those functions, let's take a look at the IMU sensor itself. The Arduino Nano 33 BLE Sense board is equipped with a LSM9DS1 sensor, which is a 9-DOF (Degrees of Freedom) sensor that combines a 3D accelerometer, 3D gyroscope, and a 3D magnetometer.

Arduino_LSM9DS1.h

The Arduino_LSM9DS1.h library is a standard library for interfacing with the LSM9DS1 sensor on some Arduino boards like the Arduino Nano 33 BLE Sense. This sensor provides 9-DOF (Degrees of Freedom) by combining a 3D accelerometer, 3D gyroscope, and a 3D magnetometer.

Initialization

Firstly, include the library at the top of your sketch:

#include <Arduino_LSM9DS1.h>

Initialize the sensor within the setup() function:

void setup() {
    if (!IMU.begin()) {
        Serial.println("Failed to initialize IMU!");
        while (1);
    }
    Serial.begin(9600);

    // Make sure we are pulling measurements into a FIFO.
    // If you see an error on this line, make sure you have at least v1.1.0 of the
    // Arduino_LSM9DS1 library installed.
    IMU.setContinuousMode();

    acceleration_sample_rate = IMU.accelerationSampleRate();
    gyroscope_sample_rate = IMU.gyroscopeSampleRate();
}

Reading Data

You can read data from the sensor within the loop() function. Here are some examples:

  • Accelerometer
float x, y, z;

if (IMU.accelerationAvailable()) {
    IMU.readAcceleration(x, y, z);
    Serial.print("Acceleration - x: ");
    Serial.print(x);
    Serial.print(", y: ");
    Serial.print(y);
    Serial.print(", z: ");
    Serial.println(z);
}
  • Gyroscope
if (IMU.gyroscopeAvailable()) {
    IMU.readGyroscope(x, y, z);
    Serial.print("Gyroscope - x: ");
    Serial.print(x);
    ...
}
  • Magnetometer
if (IMU.magneticFieldAvailable()) {
    IMU.readMagneticField(x, y, z);
    Serial.print("Magnetometer - x: ");
    Serial.print(x);
    ...
}

That's a simple way to start using the LSM9DS1 sensor with the Arduino_LSM9DS1.h library. This can be useful for motion tracking, gesture recognition, and other sensor fusion applications.

UpdateStroke()

void UpdateStroke(int new_samples, bool* done_just_triggered) processes stroke data based on gyroscope readings. The function updates the state of the stroke, computes a series of points, and conditions whether the stroke has been completed. Below is a detailed explanation of its logic:

  • Function Parameters:

  • int new_samples: The number of new gyroscope samples available.

  • bool* done_just_triggered: A pointer to a boolean that indicates whether a stroke was just completed.

  • Constants and Initial Setup:

constexpr int moving_sample_count = 50;
constexpr int minimum_stroke_length = moving_sample_count + 10;
constexpr float minimum_stroke_size = 0.2f;
  • moving_sample_count
  • minimum_stroke_length and minimum_stroke_size define minimum criteria for a valid stroke.
  • *done_just_triggered = false: Initially, sets the stroke's "done" flag to false.

Loop Through New Samples:

for (int i = 0; i < new_samples; ++i) {
  // ...
}

  • Iterates through each new gyroscope sample.

  • State Management: The stroke_state variable holds the current state of the stroke (eWaiting, eDrawing, eDone).

    • If in eWaiting or eDone and movement is detected, it transitions to eDrawing.
    • If in eDrawing and no movement is detected, it either marks the stroke as eDone if it's long enough, or resets to eWaiting.

  • Accumulate Gyroscope Data: Computes mean x, y, and z angular orientations based on gyroscope data.

  • Map Orientation to 2D Plane: Takes orientation data and projects it into a 2D XY plane. This is done using some normalization and some projection math that involves the roll orientation of the Arduino.

  • Check Stroke Size : After completing a stroke (*done_just_triggered == true), checks if the stroke is too small based on x_range and y_range. If it's too small, it cancels the stroke.

IsMoving

The function bool IsMoving(int samples_before) determines if an Arduino device is moving based on the mean squared difference (MSD) of current and previous gyroscope data (same window but shifted by samples_before):

It returns true if the device is moving and false otherwise.

EstimateGyroscopeDrift()

The function void EstimateGyroscopeDrift(float* drift) is designed to estimate the drift of a gyroscope sensor when the device it's part of isn't moving. This is done by averaging the gyroscope data over a period of time.

EstimateGravityDirection()

The function void EstimateGravityDirection(float* gravity) is designed to estimate the direction of gravity based on acceleration data. This is done by averaging the acceleration data over a period of time.

UpdateOrientation()

The function void UpdateOrientation(int new_samplesfloat* drift) aims to update the orientation of a device based on its gyroscope data. It is done by Euler integration, which is a method for numerically approximating the solution of a differential equation: