Are there any tools to help debug watchdog timeouts

ESP_Sprite
Posts: 8921
Joined: Thu Nov 26, 2015 4:08 am

Re: Are there any tools to help debug watchdog timeouts

Postby ESP_Sprite » Tue Mar 13, 2018 2:43 am

Yes, the backtrace on panic is rubbish; the panic-on-watchdog-timeout-option is mostly meant to reboot the system when an abnormal issue happens. We are aware that this doesn't help when debugging - I actually ran into this a while ago. We actually have an internal request for a feature to stop in the active process when the task watchdog is called, should be in esp-idf master eventually.

rwel59
Posts: 97
Joined: Thu Oct 12, 2017 3:32 pm

Re: Are there any tools to help debug watchdog timeouts

Postby rwel59 » Thu Mar 15, 2018 1:50 am

So there is no way to figure out what the back trace is pointing to?

ESP_Sprite
Posts: 8921
Joined: Thu Nov 26, 2015 4:08 am

Re: Are there any tools to help debug watchdog timeouts

Postby ESP_Sprite » Thu Mar 15, 2018 11:09 am

Not using the serial port gdbstub. I seem to remember that jtag may be a better option; it either gives you a good backtrace out-of-the-box or you can get there by reading the exception registers (EPC1, if memory serves).

innomkr
Posts: 7
Joined: Wed Sep 22, 2021 4:29 am

Re: Are there any tools to help debug watchdog timeouts

Postby innomkr » Thu Nov 04, 2021 7:38 am

I am experiencing a similar issue but in my case esp_timer is the currently running task.
The funny thing is that I am not using esp_timer in my code and it must be some of libs my code depends on.
I have spent over a month trying to debug this issue but without any success. With some changes in my code now the task_wdt gets triggered after few days of uptime(it used to trigger within few hours before)

So, after this many years is there finally a better option to debug this?

Code: Select all

E (143150756) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (143150756) task_wdt:  - IDLE (CPU 0)
E (143150756) task_wdt: Tasks currently running:
E (143150756) task_wdt: CPU 0: esp_timer

ESP_Sprite
Posts: 8921
Joined: Thu Nov 26, 2015 4:08 am

Re: Are there any tools to help debug watchdog timeouts

Postby ESP_Sprite » Thu Nov 04, 2021 7:58 am

That's impossible to say without at least looking at your code, sorry.

ESP_igrr
Posts: 2067
Joined: Tue Dec 01, 2015 8:37 am

Re: Are there any tools to help debug watchdog timeouts

Postby ESP_igrr » Fri Nov 05, 2021 8:00 am

The corrupted backtrace on abort is fixed in the latest toolchain release, 2021r2. https://github.com/espressif/esp-idf/issues/6124

If you are using an IDF release older than 4.4, you can still try to use the gdb from the latest toolchain.

owenjames
Posts: 8
Joined: Fri Sep 02, 2022 6:24 pm

Re: Are there any tools to help debug watchdog timeouts

Postby owenjames » Fri Jun 02, 2023 4:44 am

SDK: ESP-IDF v5.0.1
I want to log the stack trace when I get a ESP_RST_TASK_WDT task watchdog timeout. I have a custom panic handler that copies the stack trace to RAM and startup code that interrogates this. I thought this would work when I enabled "Invoke panic handler on Task Watchdog timeout" in the config editor, but it does not. I simply get the restart reason as ESP_RST_TASK_WDT and no trace. The behaviour is the same regardless of the config setting, it seems to have no effect. Any idea as to how I can log the stack / determine the stuck task on a task timeout?

MicroController
Posts: 1136
Joined: Mon Oct 17, 2022 7:38 pm
Location: Europe, Germany

Re: Are there any tools to help debug watchdog timeouts

Postby MicroController » Fri Jun 02, 2023 8:21 pm

The stack trace may be a bit hard to come by as the watchdog is handled via an interrupt, i.e. the stack of the interrupted code may be in some "intermediate" state and the call stack prior to the interrupt not available.
You may want to note that you "can define the function esp_task_wdt_isr_user_handler in the user code, in order to receive the timeout event and extend the default behavior.", and maybe have a look at the implementation of task_wdt_isr(void*).

MicroController
Posts: 1136
Joined: Mon Oct 17, 2022 7:38 pm
Location: Europe, Germany

Re: Are there any tools to help debug watchdog timeouts

Postby MicroController » Sat Jun 03, 2023 7:07 pm

Also worth noting: The "offending" task for a WDT timeout may not even be at fault, esp. when it seems to be the IDLE task. The reason for one task to miss its watchdog may well be another, higher priority, task starving the lower priority task of the CPU.

A more manual approach to locate the problem is to do a sanity check on your code and spot sections where there may be significant time spent (in a loop) without using any vTaskDelay or other blocking functions (taking semaphores, reading from queues,...) of FreeRTOS. Busy-waiting or "spinning" on a flag are candidates for reconsideration. Busy spinning may also happen accidentally when trying to use timeout durations less than portTICK_PERIOD_MS (=10ms by default), e.g. a timeout (xTicksToWait) of 5ms is not possible and will result in a timeout of 0 (= (int)(5 / portTICK_PERIOD_MS)).

Who is online

Users browsing this forum: No registered users and 119 guests