Wierdly encoded characters when receiving data.

Feelfree
Posts: 4
Joined: Wed Feb 08, 2017 7:35 pm

Wierdly encoded characters when receiving data.

Postby Feelfree » Wed Feb 08, 2017 7:40 pm

Hi! I'm writing IRC bouncer on esp32. Implemented basic functionality, but when I receive data, I'm randomly getting characters replaced like ToggleSwitchWid�Ž�Žû?�gû?�, cherryh.freenode.net @�û?Í352 - Hmm, maybe its something with this buffer to string operation? Thanks!
https://gist.github.com/feelfreelinux/8 ... cbda0f463e

ESP_Angus
Posts: 2344
Joined: Sun May 08, 2016 4:11 am

Re: Wierdly encoded characters when receiving data.

Postby ESP_Angus » Wed Feb 08, 2017 10:16 pm

Hi Feelfree,

You don't seem to have tried fixing the buffer overflow issue that I mentioned on github (receiving the full size of the buffer doesn't leave space for a terminating null byte, so you can't treat it as a string). Are you sure that's not the problem?

Angus

Feelfree
Posts: 4
Joined: Wed Feb 08, 2017 7:35 pm

Re: Wierdly encoded characters when receiving data.

Postby Feelfree » Thu Feb 09, 2017 10:34 am

ESP_Angus wrote:Hi Feelfree,

You don't seem to have tried fixing the buffer overflow issue that I mentioned on github (receiving the full size of the buffer doesn't leave space for a terminating null byte, so you can't treat it as a string). Are you sure that's not the problem?

Angus
Forgot to update gist - resized buffer to 500, problem still persist. Looks like it's only seen when there's a lot of messages in short peroid of time(Like IRC motd).

ESP_igrr
Posts: 2067
Joined: Tue Dec 01, 2015 8:37 am

Re: Wierdly encoded characters when receiving data.

Postby ESP_igrr » Thu Feb 09, 2017 5:51 pm

Can you update the gist now? It still has the problem Angus has described, that is you aren't zero-terminating your string.
Second thing: if you increase buffer size, make sure you increase the stack size as well.

Feelfree
Posts: 4
Joined: Wed Feb 08, 2017 7:35 pm

Re: Wierdly encoded characters when receiving data.

Postby Feelfree » Fri Feb 10, 2017 11:06 am

Ok, updated gist.

Feelfree
Posts: 4
Joined: Wed Feb 08, 2017 7:35 pm

Re: Wierdly encoded characters when receiving data.

Postby Feelfree » Fri Feb 10, 2017 4:38 pm

And I ported this to PC, linux. This code works correctly here. Looks like its something with lwip/esp32. https://gist.github.com/feelfreelinux/9 ... 44a9ae6186

ESP_Angus
Posts: 2344
Joined: Sun May 08, 2016 4:11 am

Re: Wierdly encoded characters when receiving data.

Postby ESP_Angus » Tue Feb 14, 2017 3:10 am

The Linux version doesn't (yet) buffer overflow because it's receiving less than 500 bytes at a time (a combination of more CPU power and faster standard output as it's not speed limited by the serial port). The potential buffer overflow is still there on Linux, it just isn't triggered as easily (some combination of CPU load and network load would eventually trigger it).

EDIT: actually a better explanation of why the Linux version doesn't crash is that the first byte of data after the 500 byte buffer happens to be zero. You can't rely on this , though!

Here's the ESP32 main.cpp file modified to avoid the buffer overflow. This works for me, whereas the previous version triggers a fatal exception or produces garbage output: https://gist.github.com/projectgus/8596 ... a6f0b5fbfa

Some tips for things to read about in order to gain more skills with this kind of network code development:
  • Learn what a null-terminated string is in C, and what the difference between a string and a char buffer is in C. Knowing this difference will help you understand why the code above fixes the problem, whereas increasing the buffer size to 500 did not.
  • Read the 'man' pages for recv, connect, socket. Update your code to check results from these.
  • You'll probably want to adapt your code to use 'select' at some point.
  • In your code, remember that a single call to recv() can return any number of bytes (TCP is a byte stream not a packet stream) - so you can receive anything between 0 whole lines of output (ie a partial incomplete line) and multiple lines of output from the server in one hit. You'll need to accumulate that raw data into a buffer, and then parse that buffer for lines of output.
Enjoy!

Who is online

Users browsing this forum: Google [Bot], MasterOfReality and 113 guests