WIFI HTTP download file speed is slow

[Llgok]

Board

ESP32-S3-DevKitC-1

Device Description

nothing

Hardware Configuration

nothing

Version

v3.0.1

IDE Name

Arduino IDE

Operating System

Windows 10

Flash frequency

QIO 80Mhz

PSRAM enabled

no

Upload speed

115200

Description

I called the function stream->readBytes(); in the HTTPClient.h library to download resources from the internet, but found that the download speed is extremely slow, averaging between 100kb/s and 200kb/s. It has been confirmed that the current network status is good, and there is no speed limit on the resource download. Although the ESP32S3 has been tested with iperf and can achieve a download speed of around 5Mb, using the HTTPClient.h library for internet resource downloads has become very slow. Is this due to the inherent limitations of the ESP32S3, or is there something happening in the data acquisition function stream->readBytes() that is causing the download speed to be so slow?

Sketch

#include <WiFi.h>
#include <HTTPClient.h>

const char *ssid = "mywifi";
const char *password = "mywifi";

// const char *fileDownloadUrl = "https://code.visualstudio.com/docs/?dv=win64user";//vscode

static size_t CycleTime = 0;

void setup()
{
    Serial.begin(115200);

    WiFi.begin(ssid, password);
    while (WiFi.status() != WL_CONNECTED)
    {
        delay(1000);
        Serial.println("Connecting to WiFi...");
    }
    Serial.println("Successfully connected to WiFi");

    HTTPClient http;
    http.begin(fileDownloadUrl);

    const char *headerKeys[] = {"Location"};
    http.collectHeaders(headerKeys, 1);

    size_t startTime = millis();

    size_t uselessTime = 0;

    int httpCode = http.GET();

    while (httpCode == HTTP_CODE_MOVED_PERMANENTLY || httpCode == HTTP_CODE_FOUND)
    {
        String newUrl = http.header("Location");
        Serial.printf("Redirecting to: %s\n", newUrl.c_str());
        http.end(); 

        http.begin(newUrl);
        httpCode = http.GET();
    }

    if (httpCode == HTTP_CODE_OK)
    {
        size_t fileSize = http.getSize();
        Serial.printf("Starting file download...\n");
        Serial.printf("file size: %f Mb\n", fileSize / 1024.0 / 1024.0);

        WiFiClient *stream = http.getStreamPtr();

        size_t temp_count_s = 0;
        size_t temp_fileSize = fileSize;
        uint8_t *buf_1 = (uint8_t *)heap_caps_malloc(100 * 1024, MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT);
        // uint8_t buf_1[4096] = {0};
        CycleTime = millis() + 1000; 
        while (http.connected() && (fileSize > 0 || fileSize == -1))
        {
            size_t availableSize = stream->available();
            if (availableSize)
            {
                temp_fileSize -= stream->readBytes(buf_1, min(availableSize, (size_t)(100 * 1024)));

                if (millis() > CycleTime)
                {
                    size_t temp_time_1 = millis();
                    temp_count_s++;
                    Serial.printf("Download speed: %f Kb/s\n", ((fileSize - temp_fileSize) / 1024.0) / temp_count_s);
                    Serial.printf("Remaining file size: %f Mb\n\n", temp_fileSize / 1024.0 / 1024.0);

                    CycleTime = millis() + 1000;
                    size_t temp_time_2 = millis();

                    uselessTime = uselessTime + (temp_time_2 - temp_time_1);
                }
            }
            // delay(1);

            if (temp_fileSize == 0)
            {
                break;
            }
        }

        http.end();

        size_t endTime = millis();
        Serial.printf("Download completed!\n");
        Serial.printf("Total download time: %f s\n", (endTime - startTime - uselessTime) / 1000.0);
        Serial.printf("Average download speed: %f Kb/s\n", (fileSize / 1024.0) / ((endTime - startTime - uselessTime) / 1000.0));
    }
    else
    {
        Serial.printf("Failed to download\n");
        Serial.printf("Error httpCode: %d \n", httpCode);
    }
}

void loop()
{
}

Debug Message

Download speed: 144.199386 Kb/s
Remaining file size: 0.254036 Mb

Download speed: 144.062446 Kb/s
Remaining file size: 0.118031 Mb

Download completed!
Total download time: 38.091000 s
Average download speed: 139.327173 Kb/s

Other Steps to Reproduce

nothing

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[me-no-dev]

What happens if you read 1KB at a time, instead of 100KB?

[Llgok]

What happens if you read 1KB at a time, instead of 100KB?

The rate of reading 1k once and reading 100k once is the same.

Modification:
temp_fileSize -= stream->readBytes(buf_1, min(availableSize, (size_t)(1 * 1024)));

debug:
Download speed: 160.707065 Kb/s
Remaining file size: 0.631452 Mb

Download speed: 157.924121 Kb/s
Remaining file size: 0.556043 Mb

Download speed: 157.148973 Kb/s
Remaining file size: 0.425286 Mb

Download incomplete
Download time: 31.920000 s
Average download speed: 152.619617 Kb/s

@me-no-dev

[me-no-dev]

Use read instead of readBytes for now. Fix added in PR. readBytes currently reads byte by byte. I get about 340 KB/s with the change

[JAndrassy]

https://www.arduino.cc/reference/en/language/functions/communication/stream/streamreadbytes/

change to

temp_fileSize -= stream->read(buf_1, min(availableSize, (size_t)(100 * 1024)));

[SuGlider]

@Llgok - Does it improve if Stream::setTimeout(0); is applied?
*** This have been edited to fix the place where setTimeout() should be executed.***

    if (httpCode == HTTP_CODE_OK)
    {
        size_t fileSize = http.getSize();
        Serial.printf("Starting file download...\n");
        Serial.printf("file size: %f Mb\n", fileSize / 1024.0 / 1024.0);

        WiFiClient *stream = http.getStreamPtr();
        stream.setTimeout(0);   //   this shall improve readByte() time.

[Llgok]

@Llgok - Does it improve if Stream::setTimeout(0); is applied? *** This have been edited to fix the place where setTimeout() should be executed.***

    if (httpCode == HTTP_CODE_OK)
    {
        size_t fileSize = http.getSize();
        Serial.printf("Starting file download...\n");
        Serial.printf("file size: %f Mb\n", fileSize / 1024.0 / 1024.0);

        WiFiClient *stream = http.getStreamPtr();
        stream.setTimeout(0);   //   this shall improve readByte() time.

After testing with the addition of stream->setTimeout(0);, the results showed little difference compared to when it was not added.

Modification:
if (httpCode == HTTP_CODE_OK)
{
size_t fileSize = http.getSize();
Serial.printf("Starting file download...\n");
Serial.printf("file size: %f Mb\n", fileSize / 1024.0 / 1024.0);

WiFiClient *stream = http.getStreamPtr();
stream->setTimeout(0); // this shall improve readByte() time.
......

debug:
Download speed: 150.439453 Kb/s
Remaining file size: 0.334579 Mb

Download speed: 149.089442 Kb/s
Remaining file size: 0.232491 Mb

Download speed: 148.694029 Kb/s
Remaining file size: 0.100410 Mb

Download completed!
Total download time: 36.439000 s
Average download speed: 145.643715 Kb/s

@SuGlider

[Llgok]

Use read instead of readBytes for now. Fix added in PR. readBytes currently reads byte by byte. I get about 340 KB/s with the change

Using stream->read to read data is not much different from using stream->readBytes. However, reading one byte at a time with stream->read() is slower (about 30Kb/s).

Modification:
size_t availableSize = stream->available();
if (availableSize)
{
temp_fileSize -= stream->read(buf_1, min(availableSize, (size_t)(100 * 1024)));
......

debug:
Download speed: 162.999609 Kb/s
Remaining file size: 0.407347 Mb

Download speed: 164.102319 Kb/s
Remaining file size: 0.214785 Mb

Download speed: 162.855225 Kb/s
Remaining file size: 0.093500 Mb

Download completed!
Total download time: 33.807000 s
Average download speed: 156.982617 Kb/s

@JAndrassy @me-no-dev

[Llgok]

I found that increasing the value of length in the function virtual size_t readBytes(char *buffer, size_t length); can improve download speed. However, I can only set length to a maximum of 100 * 1024. If I set it to 500 * 1024, the ESP32S3 keeps restarting and reporting errors, but length is of the size_t type.


debug:
Guru Meditation Error: Core 1 panic'ed (StoreProhibited). Exception was unhandled.

Core 1 register dump:
PC : 0x4200a5d0 PS : 0x00060330 A0 : 0x820a2c1c A1 : 0x3fcebf60
A2 : 0x00000000 A3 : 0x00000000 A4 : 0x00000c96 A5 : 0x3fca4a94
A6 : 0x00000802 A7 : 0x00000000 A8 : 0x00000000 A9 : 0x3fcebf40
A10 : 0x00000049 A11 : 0x0000056c A12 : 0x3fcebf5c A13 : 0x1d600000
A14 : 0x3fc999e0 A15 : 0x00000064 SAR : 0x0000000a EXCCAUSE: 0x0000001d
EXCVADDR: 0x00000000 LBEG : 0x40056f5c LEND : 0x40056f72 LCOUNT : 0x00000000


Backtrace: 0x4200a5cd:0x3fcebf60 0x420a2c19:0x3fcebf80 0x42002f21:0x3fcebfa0 0x4200cd43:0x3fcec0d0




ELF file SHA256: b42dfae7624d220d

Rebooting...

@JAndrassy @me-no-dev @SuGlider

[me-no-dev]

@Llgok Client::read(buf, len) does not read byte by byte and is much faster than Stream::readBytes(buf, len) when you have good connection. I got (as I said) 340KB/s with read and 150KB/s with readBytes (which actually reads byte by byte). Until we make Client::readBytes() faster, I suggest you use read instead.

[me-no-dev]

If I set it to 500 * 1024, the ESP32S3 keeps restarting and reporting errors, but length is of the size_t type.

You do not have 500KB of continuous memory, so your malloc did not work and you are writing to NULL location. read with 64 * 1024 gives you plenty of buffer

[me-no-dev]

Test downloading 22MB zip and using readBytes with ESP32-S3

Test results are relative and actual speeds will vary based on the environment.

Download from HTTP server (no SSL):

// python3 -m http.server 8080 -d .
const char *fileDownloadUrl = "http://192.168.254.91:8080/esp32-3.0.1.zip";

New Method:

Download completed!
Total download time: 40.714000 s
Average download speed: 573.149578 Kb/s

Old Method:

Download completed!
Total download time: 63.890000 s
Average download speed: 365.240443 Kb/s

Download from HTTPS server (with SSL):

const char *fileDownloadUrl = "https://github.com/espressif/arduino-esp32/releases/download/3.0.1/esp32-3.0.1.zip";

New Method:

Download completed!
Total download time: 80.371000 s
Average download speed: 290.343680 Kb/s

Old Method:

Download completed!
Total download time: 162.368000 s
Average download speed: 143.718047 Kb/s

[Llgok]

Alright, thank you very much for your explanation. I moved the device to a place with a strong signal, and the download speed has indeed increased significantly. In future updates, is it possible that we can hope to reach download speeds of over 1 Mb/s？
@me-no-dev

[me-no-dev]

@Llgok I am pretty sure that you mistake b for B here. 1MB/s = 8Mb/s and in your example you should have B everywhere, because you measure bytes, not bits. 570KB/s = 4.4Mb/s