Improving UDP throughput

samvrit
Posts: 1
Joined: Sun Jul 19, 2020 9:08 am

Improving UDP throughput

Postby samvrit » Mon Jul 20, 2020 1:03 am

Hi,

My problem statement is that I have two separate ESP32 devices sensing some physical quantity and sending a float value packed into bytes using UDP to another ESP32, which is running in softAP mode. My goal is to get the time interval between two sensor readings from each sensor within 100 microseconds at the AP device. Considering the throughput specifications, this should be achievable. Currently I am getting an interval of ~1000 microseconds between sensor readings, which is not acceptable since this is for a hard-real-time controls application. The two ESP32 sensors are programmed using Arduino, running at 80MHz. The ESP32 running in AP mode is programmed using esp-idf, and is running at 240MHz with the following optimizations:
  • Optimization level: Release
  • WiFi IRAM optimization enabled
  • UDP IRAM optimization enbled
  • UDP mailbox size: 64
  • FreeRTOS tick rate: 1000
  • Layer 2 to layer 3 copy enabled
I am open to exploring both high-level design changes, as well as low-level code optimizations. One of the high-level deign changes I was considering was to create a mesh topology, in which one sensor is running only STA mode, connected to another sensor running AP+STA mode, which is then connected to the base ESP32 device, running only AP mode. Would this be a good idea?

Arduino Code

Code: Select all

//===================Includes===================//
#include <WiFi.h>
#include <WiFiUdp.h>
#include <SPI.h>

//===================Defines===================//
#define PI 3.1415F
#define ANGLE_SCALING_FACTOR (-PI/32768U) // reverse sign to comply with model

//===================Global Variables===================//
typedef union {
    float value;
    char buffer[sizeof(float)];
} udpPacket_t;
udpPacket_t udpPacket;
byte mac[6];
bool device1, device2;
//==================================================//

//===================Wi-Fi Configs===================//
const char * networkName = "ublox";
const char * networkPswd = "furuta123";
IPAddress *ip;  // instantiate an IP address, but initialize it depending on MAC address
IPAddress gateway(10, 0, 0, 1);
IPAddress subnet(255, 255, 255, 0);
IPAddress dns1(10, 0, 0, 1);
IPAddress dns2(10, 0, 0, 1);

const char * udpAddress = "10.0.0.1"; // IP address to send the data to
const int udpPort = 5050;
const int udpLocalPort = 8888;

WiFiUDP udp;
void connectToWiFi(const char * ssid, const char * pwd);
void WiFiEvent(WiFiEvent_t event);
boolean wifi_connected = false;
//==================================================//

//===================SPI Configs===================//
SPIClass * hspi = NULL;
static const int spiClk = 25000000; // 25 MHz
//================================================//

void setup()
{
  hspi = new SPIClass(HSPI);
  hspi->begin();
  pinMode(15, OUTPUT); //HSPI SS
  Serial.begin(115200);

  WiFi.macAddress(mac); // get MAC address of the PHY
  device1 = ((mac[3] == 0x7A) && (mac[4] == 0x10) && (mac[5] == 0xEC)) ? 1 : 0; // use MAC address to differentiate between the two devices
  device2 = !device1;

  if (device1)
    ip = new IPAddress(10, 0, 0, 5);
  else if (device2)
    ip = new IPAddress(10, 0, 0, 10);

  connectToWiFi(networkName, networkPswd);  //Connect to the WiFi network
}

void loop()
{
  udpPacket.value = getAngle();
  Serial.print("Position: ");
  Serial.print(udpPacket.value);
  Serial.print(" | ");
  Serial.print(udpPacket.buffer[0], HEX);
  Serial.print(" ");
  Serial.print(udpPacket.buffer[1], HEX);
  Serial.print(" ");
  Serial.print(udpPacket.buffer[2], HEX);
  Serial.print(" ");
  Serial.println(udpPacket.buffer[3], HEX);
  if (wifi_connected)  //only send data when connected
  {
    //Send a packet
    udp.beginPacket(udpAddress, udpPort);
    udp.write((const uint8_t *)udpPacket.buffer, 4);
    udp.endPacket();

    if (udp.parsePacket())
    {
      // code for encoder zero setting
    }
  }

  delayMicroseconds(1);
}
Base ESP32 Code

Code: Select all

/* BSD Socket API Example

   This example code is in the Public Domain (or CC0 licensed, at your option.)

   Unless required by applicable law or agreed to in writing, this
   software is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
   CONDITIONS OF ANY KIND, either express or implied.
*/
#include <string.h>
#include <sys/param.h>
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "esp_system.h"
#include "esp_wifi.h"
#include "esp_event.h"
#include "esp_log.h"
#include "nvs_flash.h"
#include "tcpip_adapter.h"
#include "protocol_examples_common.h"

#include "driver/gpio.h"
#include "driver/spi_slave.h"

#include "lwip/err.h"
#include "lwip/sockets.h"
#include "lwip/sys.h"
#include <lwip/netdb.h>

#define EXAMPLE_ESP_WIFI_SSID       "ublox"
#define EXAMPLE_ESP_WIFI_PASS       "furuta123"
#define WIFI_IP_ADDR                "10.0.0.1"
#define EXAMPLE_MAX_STA_CONN        2
#define PORT                        5050 
#define MAX_SOCKET_RETRIES          5

#define GPIO_MOSI 12
#define GPIO_MISO 13
#define GPIO_SCLK 15
#define GPIO_CS 14

#define UDP_PACKET_DELAY_THRESHOLD  1000U    // microseconds
#define X2_DATA_VALID_BIT           1U
#define X3_DATA_VALID_BIT           0U

static const char *WIFI_TAG = "WIFI";
static const char *UDP_TAG = "UDP";

typedef union {
    float value;
    char  buffer[sizeof(float)];
} udpPacket_t;

volatile udpPacket_t x2, x3;

typedef struct {
    float       x2;
    float       x3;
    uint8_t     dataValid;
} sensorData_S;

typedef union {
    sensorData_S sensorData;
    uint32_t bytes[sizeof(sensorData_S)];
} sendBuffer_t;

volatile sendBuffer_t sendBuffer;

uint64_t t0_x2, t0_x3;
uint64_t t1_x2, t1_x3;
uint64_t t_x2, t_x3;

static void wifi_event_handler(void* arg, esp_event_base_t event_base, int32_t event_id, void* event_data)
{
    if (event_id == WIFI_EVENT_AP_STACONNECTED) 
    {
        wifi_event_ap_staconnected_t* event = (wifi_event_ap_staconnected_t*) event_data;
        ESP_LOGI(WIFI_TAG, "station "MACSTR" join, AID=%d", MAC2STR(event->mac), event->aid);
    } 
    else if (event_id == WIFI_EVENT_AP_STADISCONNECTED) 
    {
        wifi_event_ap_stadisconnected_t* event = (wifi_event_ap_stadisconnected_t*) event_data;
        ESP_LOGI(WIFI_TAG, "station "MACSTR" leave, AID=%d", MAC2STR(event->mac), event->aid);
    }
}

void wifi_init_softap()
{
    tcpip_adapter_init();
    ESP_ERROR_CHECK(esp_event_loop_create_default());
    ESP_ERROR_CHECK(tcpip_adapter_dhcps_stop(TCPIP_ADAPTER_IF_AP)); // stop DHCP server
    // assign a static IP to the network interface
    tcpip_adapter_ip_info_t info;
    memset(&info, 0, sizeof(info));    
    info.ip.addr = inet_addr(WIFI_IP_ADDR);
    info.gw.addr = inet_addr(WIFI_IP_ADDR);
 
    IP4_ADDR(&info.netmask, 255, 255, 255, 0);
    ESP_ERROR_CHECK(tcpip_adapter_set_ip_info(TCPIP_ADAPTER_IF_AP, &info)); // hook the TCP adapter to the WiFi AP

    wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
    ESP_ERROR_CHECK(esp_wifi_init(&cfg));

    ESP_ERROR_CHECK(esp_event_handler_register(WIFI_EVENT, ESP_EVENT_ANY_ID, &wifi_event_handler, NULL));

    wifi_config_t wifi_config = 
    {
        .ap = 
        {
            .ssid = EXAMPLE_ESP_WIFI_SSID,
            .ssid_len = strlen(EXAMPLE_ESP_WIFI_SSID),
            .password = EXAMPLE_ESP_WIFI_PASS,
            .max_connection = EXAMPLE_MAX_STA_CONN,
            .authmode = WIFI_AUTH_WPA_WPA2_PSK
        },
    };
    if (strlen(EXAMPLE_ESP_WIFI_PASS) == 0) 
    {
        wifi_config.ap.authmode = WIFI_AUTH_OPEN;
    }

    ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_AP));
    ESP_ERROR_CHECK(esp_wifi_set_config(ESP_IF_WIFI_AP, &wifi_config));
    ESP_ERROR_CHECK(esp_wifi_start());

    ESP_LOGI(WIFI_TAG, "wifi_init_softap finished. SSID:%s password:%s", EXAMPLE_ESP_WIFI_SSID, EXAMPLE_ESP_WIFI_PASS);   
}

static __always_inline void process_data(unsigned char *recv_buf, const uint32_t addr_raw)
{
    if(0x0A00000A == addr_raw)
    {
        memcpy((void *)x2.buffer, recv_buf, 4);
        t1_x2 = esp_timer_get_time();
        t_x2 = t1_x2 - t0_x2;
        t0_x2 = t1_x2;
        sendBuffer.sensorData.x2 = x2.value;
        sendBuffer.sensorData.dataValid = (t_x2 <= UDP_PACKET_DELAY_THRESHOLD) ? (sendBuffer.sensorData.dataValid | (1 << X2_DATA_VALID_BIT)) : (sendBuffer.sensorData.dataValid & ~(1 << X2_DATA_VALID_BIT));
        // ESP_LOGI(UDP_TAG, "%lld", t_x2);
    }
    else if(0x0500000A == addr_raw)
    {
        memcpy((void *)x3.buffer, recv_buf, 4);
        t1_x3 = esp_timer_get_time();
        t_x3 = t1_x3 - t0_x3;
        t0_x3 = t1_x3;
        sendBuffer.sensorData.x3 = x3.value;
        sendBuffer.sensorData.dataValid = (t_x3 <= UDP_PACKET_DELAY_THRESHOLD) ? (sendBuffer.sensorData.dataValid | (1 << X3_DATA_VALID_BIT)) : (sendBuffer.sensorData.dataValid & ~(1 << X3_DATA_VALID_BIT));
        // ESP_LOGI(UDP_TAG, "%lld", t_x3);
    }
}

static void udp_server_task(void *pvParameters)
{
    int socket_retry_count = 0;

    while (socket_retry_count < MAX_SOCKET_RETRIES) 
    {
        struct sockaddr_in dest_addr;
        dest_addr.sin_addr.s_addr = inet_addr(WIFI_IP_ADDR);
        dest_addr.sin_family = AF_INET;
        dest_addr.sin_port = htons(PORT);

        int sock = socket(AF_INET, SOCK_DGRAM, IPPROTO_IP);
        if (sock < 0) 
        {
            ESP_LOGE(UDP_TAG, "Unable to create socket: errno %d", errno);
            break;
        }
        ESP_LOGI(UDP_TAG, "Socket created");

        int err = bind(sock, (struct sockaddr *)&dest_addr, sizeof(dest_addr));
        if (err < 0) 
        {
            ESP_LOGE(UDP_TAG, "Socket unable to bind: errno %d", errno);
        }
        ESP_LOGI(UDP_TAG, "Socket bound, port %d", PORT);

        t0_x2 = esp_timer_get_time();
        t0_x3 = esp_timer_get_time();

        // initialize the send buffer (struct that will be updated to be sent over)
        sendBuffer.sensorData.x2 = 0.0F;
        sendBuffer.sensorData.x3 = 0.0F;
        sendBuffer.sensorData.dataValid = 0U;

        while (1) 
        {
            char rx_buffer[7] = "";
            // ESP_LOGI(UDP_TAG, "Waiting for data");
            struct sockaddr_in source_addr; 
            socklen_t socklen = sizeof(source_addr);
            int len = recvfrom(sock, rx_buffer, sizeof(rx_buffer) - 1, 0, (struct sockaddr *)&source_addr, &socklen);

            // Error occurred during receiving
            if (len < 0) 
            {
                ESP_LOGE(UDP_TAG, "recvfrom failed: errno %d", errno);
                break;
            }
            // Data received
            else 
            {
                process_data((unsigned char *)rx_buffer, (uint32_t)source_addr.sin_addr.s_addr);
            }
        }

        if (sock != -1) 
        {
            ESP_LOGE(UDP_TAG, "Shutting down socket and restarting...");
            shutdown(sock, 0);
            close(sock);
        }

        socket_retry_count++;
    }
    vTaskDelete(NULL);
}

static void spi_comms_task(void *pvParameters)
{
    esp_err_t ret;
     //Configuration for the SPI bus
    spi_bus_config_t buscfg={
        .mosi_io_num=GPIO_MOSI,
        .miso_io_num=GPIO_MISO,
        .sclk_io_num=GPIO_SCLK
    };

    //Configuration for the SPI slave interface
    spi_slave_interface_config_t slvcfg={
        .mode=0,
        .spics_io_num=GPIO_CS,
        .queue_size=3,
        .flags=0,
        .post_setup_cb=NULL,
        .post_trans_cb=NULL
    };

    //Enable pull-ups on SPI lines so we don't detect rogue pulses when no master is connected.
    gpio_set_pull_mode(GPIO_MOSI, GPIO_PULLUP_ONLY);
    gpio_set_pull_mode(GPIO_SCLK, GPIO_PULLUP_ONLY);
    gpio_set_pull_mode(GPIO_CS, GPIO_PULLUP_ONLY);

    //Initialize SPI slave interface
    ret=spi_slave_initialize(HSPI_HOST, &buscfg, &slvcfg, 0);
    assert(ret==ESP_OK);

    uint8_t recvbuf[9] = {0x0000};
    // memset(recvbuf, 0, 33);
    spi_slave_transaction_t t;
    memset(&t, 0, sizeof(t));

    while(1) 
    {
        //Clear receive buffer, set send buffer to something sane
        memset(recvbuf, 0x00, 9);

        //Set up a transaction of 128 bytes to send/receive
        t.length = 9*8;
        t.tx_buffer = (const void *)sendBuffer.bytes;
        t.rx_buffer = &recvbuf;

        ret = spi_slave_transmit(HSPI_HOST, &t, portMAX_DELAY);
    }

}

void app_main()
{
    //Initialize NVS
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
      ESP_ERROR_CHECK(nvs_flash_erase());
      ret = nvs_flash_init();
    }
    ESP_ERROR_CHECK(ret);

    ESP_LOGI(WIFI_TAG, "ESP_WIFI_MODE_AP");
    wifi_init_softap();

    xTaskCreate(udp_server_task, "udp_server", 4096, NULL, 6, NULL);
    xTaskCreate(spi_comms_task, "spi_comms", 4096, NULL, 5, NULL);
}


vahidmousavi2017
Posts: 1
Joined: Tue Oct 13, 2020 10:23 am

Re: Improving UDP throughput

Postby vahidmousavi2017 » Thu Oct 15, 2020 10:24 am

Hello, thank you so much for sharing this code.
I am using two esp32 modules (esp32-wrover-B and esp-wroom-32) and esp-idf framework. I am gonna use them as a full duplex data streaming modem. Since, having real time data link factor is more important in my project even with the fact of probably loss of data in UDP socket, I have chosen UDP. I did try some example esp32_UDP_socket code but didn't work out. Data streaming stops after a short period of time. after several study I figured that I need to manage memory leak and decide when to release it during flow data. So I traced heap memory according to this link.

https://docs.espressif.com/projects/esp ... debug.html

Beside memory debug, there was a strange thing that every time I reset the AP side(esp32-wrover-B) just happened, data starts to stream but less data transmitted in compare with previous module run time. Unless I flash the module again. Let say at very first time (flashing module), streaming takes for example one minutes, after that by every time reset, time period decreased. All in all, I uploaded your code in both side. But client module couldn’t connect to Server_AP side. WiFi doesn't work. I think I should handle this part of your code.

Code: Select all

    ESP_ERROR_CHECK(esp_event_loop_create_default());
    ESP_ERROR_CHECK(tcpip_adapter_dhcps_stop(TCPIP_ADAPTER_IF_AP)); // stop DHCP server
    // assign a static IP to the network interface
    tcpip_adapter_ip_info_t info;
    memset(&info, 0, sizeof(info));    
    info.ip.addr = inet_addr(WIFI_IP_ADDR);
    info.gw.addr = inet_addr(WIFI_IP_ADDR);
 
    IP4_ADDR(&info.netmask, 255, 255, 255, 0);
    ESP_ERROR_CHECK(tcpip_adapter_set_ip_info(TCPIP_ADAPTER_IF_AP, &info)); // hook the TCP adapter to the WiFi AP
I would be appreciated if you guide me in this.

PeterR
Posts: 568
Joined: Mon Jun 04, 2018 2:47 pm

Re: Improving UDP throughput

Postby PeterR » Thu Oct 15, 2020 5:25 pm

100uS hard over Wifi sounds tough!
It takes me 15 uS just to read a few SPI bytes. Ok I could preallocate the request, forgive my lazy arse.
And then I also have someone jumping in every now and stealing 2mS (I blame Ethernet & I 'know' its bus because my SPI task is highest priority & no one else has that priority).
But even if you resolve that, what about Wifi buffer availability & ACKs etc?

The ESP32 has limited bus bandwidth & so any RT strategy needs to think that through, which will be hard given the size of the code base.

On a positive note I would suggest you look at core affinity and isolate your processes from the crowd.
Even so, 100uS over Wifi.... If I whispered would it fail?

Out of interest what is the control process (10KHz sounds fun)?
& I also believe that IDF CAN should be fixed.

Who is online

Users browsing this forum: Gianluca.Loi, Google [Bot], markevens and 37 guests