Google IoT client's memory limiter conflicts with ESP-IDF heap corruption detector

[dastoned]

Hi!

I'm using the excellent Google IoT Core client port from https://github.com/espressif/esp-google-iot. The CMakeLists.txt in this project enables the memory limiter (-DIOTC_MEMORY_LIMITER_ENABLED).

This option seems to conflict rather badly with ESP-IDF v 4.1 heap corruption detection (enabled by HEAP_POISONING_LIGHT or HEAP_POISONING_COMPREHENSIVE). When both the limiter and detector are enabled at the same time (as is the default), then under some circumstances freeing allocated heap memory triggers ESP-IDF heap corruption detector (which reboots the system).

I see that when memory is free()-d by the mbedtls library (e.g. when a TLS session dies due to an error) it fails with this message:

Code: Select all

D (18726) mbedtls: ssl_tls.c:6336 => handshake wrapup                                                                                                                                                                                                                         [439/134558]
D (18726) mbedtls: ssl_tls.c:6309 => handshake wrapup: final free
CORRUPT HEAP: Bad head at 0x3ffbdcc0. Expected 0xabba1234 got 0x3ffbdcf4
abort() was called at PC 0x40082aca on core 0
0x40082aca: lock_acquire_generic at /home/tarmo/myproject/lib/src/esp-idf/components/newlib/locks.c:143

ELF file SHA256: 3ed885ca66e80d50
Backtrace: 0x40086dbd:0x3ffd1210 0x40087151:0x3ffd1230 0x40082aca:0x3ffd1250 0x40082bed:0x3ffd1280 0x4019d901:0x3ffd12a0 0x40194e75:0x3ffd1560 0x40194d51:0x3ffd15b0 0x4008c42b:0x3ffd15e0 0x4008238e:0x3ffd1600 0x4008f3b9:0x3ffd1620 0x400e934d:0x3ffd1640 0x400e5355:0x3ffd1660 0x400e4
1b0:0x3ffd1680 0x40137c2a:0x3ffd16a0 0x4012c497:0x3ffd16c0 0x40130b8c:0x3ffd16e0 0x4012ef59:0x3ffd1700 0x4012f16d:0x3ffd1720 0x40123a5c:0x3ffd1740 0x40123a9d:0x3ffd1760 0x40123b70:0x3ffd1780 0x4012976e:0x3ffd17a0 0x40123979:0x3ffd17c0 0x401239cb:0x3ffd17e0 0x4011d649:0x3ffd1800 0x4
011ce79:0x3ffd1820 0x4011d0db:0x3ffd1840 0x4011d1ab:0x3ffd1870 0x4017925b:0x3ffd18a0 0x401b26a1:0x3ffd18c0 0x4011ef6f:0x3ffd18e0 0x4011f1b5:0x3ffd1910 0x400ea356:0x3ffd1930 0x400ea4d4:0x3ffd1950 0x400ea7a7:0x3ffd1980 0x400e21d5:0x3ffd19b0 0x400e22b3:0x3ffd1a40 0x400e2326:0x3ffd1a80
 0x40088399:0x3ffd1ab0
0x40086dbd: invoke_abort at /home/tarmo/myproject/lib/src/esp-idf/components/esp32/panic.c:157
0x40087151: abort at /home/tarmo/myproject/lib/src/esp-idf/components/esp32/panic.c:174
0x40082aca: lock_acquire_generic at /home/tarmo/myproject/lib/src/esp-idf/components/newlib/locks.c:143
0x40082bed: _lock_acquire_recursive at /home/tarmo/myproject/lib/src/esp-idf/components/newlib/locks.c:171
0x4019d901: _vfiprintf_r at /builds/idf/crosstool-NG/.build/xtensa-esp32-elf/src/newlib/newlib/libc/stdio/vfprintf.c:853 (discriminator 2)
0x40194e75: fiprintf at /builds/idf/crosstool-NG/.build/xtensa-esp32-elf/src/newlib/newlib/libc/stdio/fiprintf.c:48
0x40194d51: __assert_func at /builds/idf/crosstool-NG/.build/xtensa-esp32-elf/src/newlib/newlib/libc/stdlib/assert.c:58 (discriminator 8)
0x4008c42b: multi_heap_free at /home/tarmo/myproject/lib/src/esp-idf/components/heap/multi_heap_poisoning.c:266 (discriminator 1)
0x4008238e: heap_caps_free at /home/tarmo/myproject/lib/src/esp-idf/components/heap/heap_caps.c:272
0x4008f3b9: free at /home/tarmo/myproject/lib/src/esp-idf/components/newlib/heap.c:47
0x400e934d: iotc_bsp_mem_free at /home/tarmo/myproject/src/fv_comms/build/../components/gcp_iot_sdk/port/src/iotc_bsp_mem_posix.c:28
0x400e5355: __iotc_free at /home/tarmo/myproject/src/fv_comms/build/../components/gcp_iot_sdk/iot-device-sdk-embedded-c/src/libiotc/memory/iotc_allocator.c:39
0x400e41b0: iotc_memory_limiter_free at /home/tarmo/myproject/src/fv_comms/build/../components/gcp_iot_sdk/iot-device-sdk-embedded-c/src/libiotc/debug_extensions/memory_limiter/iotc_memory_limiter.c:341
0x40137c2a: mbedtls_free at /home/tarmo/myproject/lib/src/esp-idf/components/mbedtls/mbedtls/library/platform.c:71
0x4012c497: mbedtls_mpi_free at /home/tarmo/myproject/lib/src/esp-idf/components/mbedtls/mbedtls/library/bignum.c:111
0x40130b8c: mbedtls_ecp_point_free at /home/tarmo/myproject/lib/src/esp-idf/components/mbedtls/mbedtls/library/ecp.c:592
0x4012ef59: ecdh_free_internal at /home/tarmo/myproject/lib/src/esp-idf/components/mbedtls/mbedtls/library/ecdh.c:233
0x4012f16d: mbedtls_ecdh_free at /home/tarmo/myproject/lib/src/esp-idf/components/mbedtls/mbedtls/library/ecdh.c:265
0x40123a5c: mbedtls_ssl_handshake_free at /home/tarmo/myproject/lib/src/esp-idf/components/mbedtls/mbedtls/library/ssl_tls.c:8848
0x40123a9d: ssl_handshake_wrapup_free_hs_transform at /home/tarmo/myproject/lib/src/esp-idf/components/mbedtls/mbedtls/library/ssl_tls.c:6314
0x40123b70: mbedtls_ssl_handshake_wrapup at /home/tarmo/myproject/lib/src/esp-idf/components/mbedtls/mbedtls/library/ssl_tls.c:6387
0x4012976e: mbedtls_ssl_handshake_client_step at /home/tarmo/myproject/lib/src/esp-idf/components/mbedtls/mbedtls/library/ssl_cli.c:3626
0x40123979: mbedtls_ssl_handshake_step at /home/tarmo/myproject/lib/src/esp-idf/components/mbedtls/mbedtls/library/ssl_tls.c:8064
0x401239cb: mbedtls_ssl_handshake at /home/tarmo/myproject/lib/src/esp-idf/components/mbedtls/mbedtls/library/ssl_tls.c:8088
0x4011d649: esp_mbedtls_handshake at /home/tarmo/myproject/lib/src/esp-idf/components/esp-tls/esp_tls_mbedtls.c:107
0x4011ce79: esp_tls_handshake at /home/tarmo/myproject/lib/src/esp-idf/components/esp-tls/esp_tls.c:78
0x4011d0db: esp_tls_low_level_conn at /home/tarmo/myproject/lib/src/esp-idf/components/esp-tls/esp_tls.c:295 (discriminator 6)
0x4011d1ab: esp_tls_conn_new_sync at /home/tarmo/myproject/lib/src/esp-idf/components/esp-tls/esp_tls.c:346
0x4017925b: ssl_connect at /home/tarmo/myproject/lib/src/esp-idf/components/tcp_transport/transport_ssl.c:74
0x401b26a1: esp_transport_connect at /home/tarmo/myproject/lib/src/esp-idf/components/tcp_transport/transport.c:165
0x4011ef6f: esp_http_client_connect at /home/tarmo/myproject/lib/src/esp-idf/components/esp_http_client/esp_http_client.c:1023
0x4011f1b5: esp_http_client_open at /home/tarmo/myproject/lib/src/esp-idf/components/esp_http_client/esp_http_client.c:1179
0x400ea356: _http_connect at /home/tarmo/myproject/lib/src/esp-idf/components/esp_https_ota/src/esp_https_ota.c:98
0x400ea4d4: esp_https_ota_begin at /home/tarmo/myproject/lib/src/esp-idf/components/esp_https_ota/src/esp_https_ota.c:173
0x400ea7a7: esp_https_ota at /home/tarmo/myproject/lib/src/esp-idf/components/esp_https_ota/src/esp_https_ota.c:393
0x400e21d5: fvcomms::OtaHub::updateApp(char const*) at /home/tarmo/myproject/src/fv_comms/build/../main/OtaHub.cpp:177
0x400e22b3: fvcomms::OtaHub::run(bool const&) at /home/tarmo/myproject/src/fv_comms/build/../main/OtaHub.cpp:94 (discriminator 5)
0x400e2326: operator() at /home/tarmo/myproject/src/fv_comms/build/../main/OtaHub.cpp:36
 (inlined by) _FUN at /home/tarmo/myproject/src/fv_comms/build/../main/OtaHub.cpp:38
0x40088399: vPortTaskWrapper at /home/tarmo/myproject/lib/src/esp-idf/components/freertos/port.c:143

My hypothesis is that _multi_heap_free()_ (on multi_heap_poisoning.c:266) receives a pointer to a "heap poison" header structure generated by iotc_memory_limiter.c instead of its own poison. Since IOTC's header doesn't start with the magic *0xabba1234*, it declares heap corrupt.

Testing seems to indicate that disabling IOTC's memory limiter fixes the problem. I humbly suggest disabling it by default in https://github.com/espressif/esp-google-iot

[ESP_Angus]

Hi dastoned,

Thanks for the detailed report.

I see you've posted about the same issue on GitHub: https://github.com/espressif/esp-google-iot/issues/18

Someone will reply to you there soon.

[dastoned]

Yes, it took me a while to recognize that bug reports belong to an issue tracker, not forum Anyway, I guess this forum thread can be suspended.