ESP-IDF example to download a file from the web

[fasani]

Hello dear forum,
I'm in the process to learn IDF coming from Arduino-framework since the beginning of 2020.

I've been searching for a way to do something similar to what WiFiClient does in Arduino-espressif32. What I would like to achieve:

To instantiate the WiFiClient and make an HTTP request. And read an image from the web. But not streaming the download to a _buffer[length] I want to do a client.read() and byte per byte, add it into a GFX buffer that is in turn rendering it on a epaper screen. This is the existing function that I'm trying to port to the IDF:
https://github.com/martinberlin/eink-ca ... n.cpp#L395
drawBitmapFrom_HTTP_ToBuffer(false) // Not using color

So far I read about httpClient, a PR made in arduino-esp32, and some IDF examples. But I still didn't found an easy way to replicate the byte per byte read stream of the WiFi client. Maybe someone can point me to the right sample of code?
What I understand by seeing the examples is that you always need to buffer the request somewhere with IDF, right?

[fasani]

A bit more research and partial success:

https://github.com/martinberlin/cale-id ... n/cale.cpp

I'm just trying to call the DATA event and dumping what comes using a buffer of 1024 bytes.
It's working but somehow I'm not receiving the whole downloaded Buffer. So for sure I'm doing something wrong!

I'm using esp_http_client* functions.

Can someone point me to the obvious mistake?
This is a Serial dump of what is going on:

[fasani]

Good so let's resume this with a funny story:
ESP32 - Get's a content-length of 60Kb and starts downloading
Backend - I have 60K image!
ESP32 - mhhh ok but I found only 11Kb!!! so I stop downloading
Backend - you fool...

So in this construction I got in the middle, and started to ask, why I get only 11Kb of a 60 Kb image. In turn I molest other developers that also start wondering what is going on.

And the truth is:
It's all my fault. I wrongly assumed that the other displays work alright with this.Only because their Firmware in arduino-esp32:
https://github.com/martinberlin/eink-calendar/ is really not aware of content-length. It just downloads byte per byte on a loop and streams it to the graphics buffer.

I really cannot afford to stream it to a new uint8_t _Buffer and then process this on a loop reading byte per byte. Both the GxEPD and my own own component CalEPD in IDF framework, have already a buffer defined that is calculated using:

display_width*display_heigth /8

That's only monochrome 1 bit buffer since we are dealing with epaper displays here, so on 1 byte we can store 8 black or white points.

So in resume this was my error as backend developer. Since I trusted a [function in PHP](https://www.php.net/manual/en/imagick.g ... length.php) that is called magick::getIMageLength that in certain situations like one person commented in the site, does not deliver what it promises to deliver: Returns the image length in bytes

"If you're planning to stream imagick images via http, pay attention that this function may return the uncompressed image size, so it's not directly suitable for setting the content-length http header."

So it's of course our jobs as developers, not to take anything from granted, but to double check the results and report or suggest alternatives if they do not match the expectations.

Content-length is not a light topic. If you would use getIMageLength value that wrongly is shorter than the image, the Firmware will say: Ok is done, we stop downloading and you will get part of the image (Even on the browser)
And if it's longer, you will also think that something wrong happens, while it does not finish downloading. Even on Chrome it will keep on trying to download since it does not understand why an image that says that had X length has smaller size.

So to resume:
It's very important not to trust any backend header info unless it's yours (And this one was mine)
And of course it's better not to have any content-length than to have a wrong content-length

[fasani]

Solved:

Found a way to fill the display buffer from a Bitmap image (8, 4 & 1 bits supported. Just if anyone wants to take a look, here is the code:
https://github.com/martinberlin/cale-id ... factor/oop