We at AgriConnect have been using Rust to develop firmware for ESP32 (and even part of our IoT platform backend). From today we start a series to help fellows on the same journey. "Over-the-air firmware update" is the first sharing because it was the topic that one of the authors of esp-idf-hal suggested me to write.

For OTA firmware update, esp-idf-svc already provides API. What it lacks is a documentation how to use the API, how to prepare partitions to use with that API. This post will complement that.

(Bản tiếng Việt ở đây.)

Prepare partition

To use this feature, your ESP32 flash storage must have at least these partitions: otadata (of ota subtype), app0 (of ota_0 subtype), app1 (of ota_1 subtype).

Name	Type	SubType
otadata	data	ota
app0	app	ota_0
app1	app	ota_1

You can copy one of partition table description files in Arduino-ESP32 repository. Remember to choose the one that matches your device flash size.

After downloading, rename the file as partitions.csv and save in your top-level folder of the code base. Later, when flashing the firmware, add --partition-table option to espflash command. For example, in my case, it will be:

$ espflash flash -B 921600 -s 8mb --partition-table=partitions.csv target/xtensa-esp32s3-espidf/release/ebee

The basic

The basic usage of Esp OTA API is as following:

• Create an EspOta instance.
• Create an EspOtaUpdate instance from the previous EspOta instance.
• Call EspOtaUpdate.write() with a buffer which contains the new firmware.
• Call EspOtaUpdate.complete() after finish writing. This will mark the parition to boot up next time.

The code is roughly like this:

use esp_idf_svc::ota::EspOta;
use esp_idf_svc::sys::EspError;

fn oversimplified_update_firmware() -> Result<(), EspError> {
    let mut ota = EspOta::new()?;
    let mut work = ota.initiate_update()?;
    let buff: Vec<u8> = get_firmware_buffer_from_somewhere();
    work.write(&buff)?;
    work.complete()
}

But in practice, you need to do more works. The tricky part is that, ESP32's' RAM is not big enough to download the firmware as a whole: you won't have a buffer that contains the entire firmware binary like the line 7 does.

Note that, in the program, we design functions to return Result, and inside function, we use ? operator, to take advantage of Rust wonderful error handling. I have another post about Rust error handling here.

Simple case

Now we make the function more realistic, that is, it receives an URL, downloading firmware from that URL chunk by chunk and write immediately to Flash before downloading next chunk. By that, we can workaround the RAM issue. But we have to accept the downside: we don't have a chance to verify the file before writing to Flash.

The function will look like this:

use core::mem::size_of;

use http::header::ACCEPT;
use http::Uri;
use embedded_svc::ota::FirmwareInfo;
use embedded_svc::http::{client::Client, Method};
use esp_idf_svc::http::client::{Configuration, EspHttpConnection};
use esp_idf_svc::ota::EspOta;
use esp_idf_svc::sys::{EspError, ESP_ERR_IMAGE_INVALID, ESP_ERR_INVALID_RESPONSE};

const FIRMWARE_DOWNLOAD_CHUNK_SIZE: usize = 1024 * 20;
// Not expect firmware bigger than 2MB
const FIRMWARE_MAX_SIZE: usize = 1024 * 1024 * 2;
const FIRMWARE_MIN_SIZE: usize = size_of::<FirmwareInfo>() + 1024;

pub fn simple_download_and_update_firmware(url: Uri) -> Result<(), EspError> {
    let mut client = Client::wrap(EspHttpConnection::new(&Configuration {
        buffer_size: Some(1024 * 4),
        ..Default::default()
    })?);
    let headers = [(ACCEPT.as_str(), mime::APPLICATION_OCTET_STREAM.as_ref())];
    let surl = url.to_string();
    let request = client
        .request(Method::Get, &surl, &headers)
        .map_err(|e| e.0)?;
    let mut response = request.submit().map_err(|e| e.0)?;
    if response.status() != 200 {
        log::info!("Bad HTTP response: {}", response.status());
        return Err(esp_err!(ESP_ERR_INVALID_RESPONSE));
    }
    let file_size = response.content_len().unwrap_or(0) as usize;
    if file_size <= FIRMWARE_MIN_SIZE {
        log::info!(
            "File size is {file_size}, too small to be a firmware! No need to proceed further."
        );
        return Err(esp_err!(ESP_ERR_IMAGE_INVALID));
    }
    if file_size > FIRMWARE_MAX_SIZE {
        log::info!("File is too big ({file_size} bytes).");
        return Err(esp_err!(ESP_ERR_IMAGE_INVALID));
    }
    let mut ota = EspOta::new()?;
    let mut work = ota.initiate_update()?;
    let mut buff = vec![0; FIRMWARE_DOWNLOAD_CHUNK_SIZE];
    let mut total_read_len: usize = 0;
    let mut got_info = false;
    let dl_result = loop {
        let n = response.read(&mut buff).unwrap_or_default();
        total_read_len += n;
        if !got_info {
            match get_firmware_info(&buff[..n]) {
                Ok(info) => log::info!("Firmware to be downloaded: {info:?}"),
                Err(e) => {
                    log::error!("Failed to get firmware info from downloaded bytes!");
                    break Err(e);
                }
            };
            got_info = true;
        }
        if n > 0 {
            if let Err(e) = work.write(&buff[..n]) {
                log::error!("Failed to write to OTA. {e}");
                break Err(e);
            }
        }
        if total_read_len >= file_size {
            break Ok(());
        }
    };
    if dl_result.is_err() {
        return work.abort();
    }
    if total_read_len < file_size {
        log::error!("Supposed to download {file_size} bytes, but we could only get {total_read_len}. May be network error?");
        return work.abort();
    }
    work.complete()
}

Why is the input an URL? Because HTTP is the most convenient way to get a firmware over-the-air. Also we should break our program into units. This unit is to download and write firmware down. Other unit is to obtain the URL. The step of obtaining URLs are not the same for all projects / companies. In some projects, the ESP32 app finds the URL by looking into some JSON file hosted on S3. In other use cases, the ESP32 app has the URL because some servers push to it via MQTT, WebSocket. In one of my old projects, I created an API server with Django, and ESP32 called that API to check for new firmware, and download from there. In this post, I won't talk about "how to have the firmware URL" part.

Now I explain the simple_download_and_update_firmware function.

This function take an input parameter of Uri type (from http crate).

Why an Uri instead of &str? It is because I want to make sure the URL we received is a legitime URL, not arbitrary string. We don't bother to call this function up if the string we obtain in previous step is not an URL. Outside the function, it can be used after parsing URL string, like this:

fn outer_function() -> Result<(), AppError> {
    let url = get_url_from_somewhere(); // "https://agriconnect.vn/firmware.bin";
    if let Ok(u: Uri) = Uri::try_from(url) {
        simple_download_and_update_firmware(url: u)?
    } else {
        log::warn!("Invalid URL to download firmware");
    }
    Ok(())
}

To download firmware via HTTP, I use the pre-included HTTP client library in esp-idf-svc. You can use some HTTP client libraries from crates.io, but I haven't tried.

In the code above, the buffer_size, FIRMWARE_DOWNLOAD_CHUNK_SIZE, FIRMWARE_MAX_SIZE, FIRMWARE_MIN_SIZE values are just from experiment. I don't have a principle to infer them yet. You can try with bigger values, just be careful of stack overflow!

The esp_err! macro is just my own macro. It is defined as:

/// Macro to quickly create EspError from an ESP_ERR_ constant.
#[macro_export]
macro_rules! esp_err {
    ($x:ident) => {
        EspError::from_infallible::<$x>()
    };
}

Without it, some lines in the example code will be like this:

return Err(EspError::from_infallible::<ESP_ERR_INVALID_RESPONSE>());

Verbose, huh?

Here, we assume that the download URL is pure HTTP, not HTTPS, so the HTTP client is configured as simple as this:

let mut client = Client::wrap(EspHttpConnection::new(&Configuration {
	buffer_size: Some(1024 * 4),
	..Default::default()
})?);

If you are downloading via HTTPS, the config part will be:

use esp_idf_svc::sys::esp_crt_bundle_attach;

...
Configuration {
	buffer_size: Some(1024 * 4),
	use_global_ca_store: true,
	crt_bundle_attach: Some(esp_crt_bundle_attach),
	..Default::default()
}

In the lines involving calling HTTP client methods, you see:

.map_err(|e| e.0)

It is because these methods return Result<T, EspIOError> but our function expects error to be EspError. We have that map_err(|e| e.0) to extract the EspError from EspIOError.

You can see the documentation of EspIoError here:

pub struct EspIOError(pub EspError);

It means that EspIOError is a tuple struct (struct whose fields don't have a name and we refer a field by its number index, like 0, 1, 2). Its first and only field is of EspError type and fortunately it is public, we can take it by .0.

At the line 21, we use the constants from http and mime crates, to prevent typo mistake when setting HTTP headers.

In the lines 27 - 41, we do some simple checks, to reject invalid HTTP response.

To receive the download chunk, we define a buffer at line 44, then we read part of firmware from HTTP response at the line 49, saving to the buffer, and write downloaded chunk to flash storage at the line 52. These actions are placed in a loop to repeat until we finish retrieving file.

let dl_result = loop {
	let n = response.read(&mut buff).unwrap_or_default();
	total_read_len += n;
	if !got_info {
		match get_firmware_info(&buff[..n]) {
			Ok(info) => log::info!("Firmware to be downloaded: {info:?}"),
			Err(e) => {
				log::error!("Failed to get firmware info from downloaded bytes!");
				break Err(e);
			}
		};
		got_info = true;
	}
	if n > 0 {
		if let Err(e) = work.write(&buff[..n]) {
			log::error!("Failed to write to OTA. {e}");
			break Err(e);
		}
	}
	if total_read_len >= file_size {
		break Ok(());
	}
};

We must be aware that, network issue is possible, the chunk we receive can be smaller than the buffer length, so when we write to flash, we must not blindly take the whole buffer. We only take a part, with &buff[..n], that is the n first bytes.

In the lines 51 - 57, we are checking the first chunk, to see if the file we are downloading is an ESP32 firmware. If get_firmware_info fails on this chunk, it means that the server is returning wrong file, we stop doing further.

The get_firmware_info function is defined like this:

fn get_firmware_info(buff: &[u8]) -> Result<FirmwareInfo, EspError> {
    let mut loader = EspFirmwareInfoLoader::new();
    loader.load(buff)?;
    loader.get_info()
}

Here we can see an interesting feature of Rust. The loop can also return a value, via break, as if it is a function. By that, we can present the state of the sequence in loop, finished successfully, or interrupted early with error.

I intentionally define buff as a Vec, not array, because I want to store this data in heap, to prevent stack overflow. In AgriConnect product, this function runs in a thread, so it should not use too much stack memory. Please note that it is not the only choice. You can still use array and adjust thread stack size. I'm just lazy to do experiment to find a fit value.

After finishing writing new firmware, we call EspOtaUpdate.complete() to set the active parition for the next bootup.

Previously, I said that we can not have entire file to verify before writing. But we can use some hash function which digests data as stream, and when at the last chunk, we will have the checksum of the file. We can verify that file was not corrupted with the checksum, and cancel OTA before calling EspOtaUpdate.complete(). To keep the example code short, I didn't add checksum computation. We can leave it for reader as exercise.

Advance case

In advance case, you can call this function in a thread, and you may want to track the progress, to report to a web UI for example. You will need these Rust feature:

• Thread support.
• Communication with thread via multi-producer, single-consumer channel.

Here are a bit revalation of how it is in AgriConnect product:

use std::thread;
use std::sync::mpsc::channel;

// We are about to create two threads:
// - One to download and flash firmware to board.
// - One to get report from the previous thread and save to AppState.
// It may look complex, but we choose channel to not block the firmware-flashing thread,
// which is the case if using Mutex to modify app-wised variable.
// In the future, we will use RwLock if it is available for ESP32, to not create too many threads.
let (tx, rx) = channel::<FwUpdateProgress>();

// Thread to download and flash firmware.
thread::Builder::new()
    .stack_size(4 * 1024)
    .spawn(move || download_and_update_firmware(&download_url, tx))
    .map_err(|_e| esp_err!(ESP_ERR_NO_MEM))?;
CommonResponse::make_ok_response(request)?;

// Thread to get report and save
thread::spawn(move || save_fw_update_progress_in_thread(rx, fw_update_status, restart_flag));

The file generated by cargo build is not ready for OTA yet. We still need to trim it with espflash save-image command. For my case, it will be:

$ espflash save-image --chip esp32s3 -s 8mb target/xtensa-esp32s3-espidf/release/ebee /tmp/ebee.bin

Conclusion

We have finished the guide for adding OTA firmware feature. Thanks to the folks from esp-rs project who have made the low-level stuffs to let us enjoy Rust with embedded programming.