big_http_download sample localhost download


Lubos, Robert
 

Hi Brenton,

 

I don’t have exact answers for the issue you report, but some observations from my side, while I tried to find the root cause for the problem. The core reason http client disconnects is that we set the RX timeout on a socket (see the setsockopt calls). If you remove the timeout from a socket you’ll notice the connection remains, however the communication is extremely slow.

 

Now as for the reason of the sluggishness of the data transfer, I’ve noticed the following:

  1. The packets received by the e1000 driver tend to appear out of order, compared to what’s visible on wireshark (the server usually sends two packets at once after receiving ack)– this actually is not a big deal as Zephyr’s TCP implementation takes care of that,
  2. e1000 sometimes does not receive a packet, if a few of them were sent in a bulk – no RX irq is received at all for the lost packet. This also shouldn’t be a big deal, as retransmission at TCP level should kick in then. I’ve noticed however that the python HTTP server you use has some bizarre TCP retransmission delays (as if the delay was duplicated on each retransmission). This eventually leads to a timeout on a socket (if enabled).

 

Now, I’m not sure what is the reason that not all packets reach the e1000 driver. It could be something with RX IRQ handling, or it could be a qemu issue not handling heavy traffic well. Perhaps someone more familiar with this device or qemu emulation could elaborate on the problem.

 

Regards,

Robert

 

From: Brenton Chetty <brent7984@...>
Sent: środa, 3 listopada 2021 19:10
To: Lubos, Robert <Robert.Lubos@...>
Cc: users@...
Subject: Re: [Zephyr-users] big_http_download sample localhost download

 

Hi Robert

 

Thank you for your assistance. I managed to get qemu to begin downloading a file on localhost.

However, the file downloads up to a maximum of 2680 bytes, then it fails, and restarts the download.

 

Do you have any suggestions on why this only happens when downloading on localhost?

 

Regards

Brenton

 

On Wed, Nov 3, 2021 at 2:45 PM Lubos, Robert <Robert.Lubos@...> wrote:

Hi Brenton,

 

The DNS server configuration shouldn’t be an issue in this case, as we provide the IP address of the server – in such case the DNS query isn’t even sent as we already know the IP address.

 

I’ve reproduced the problem locally, and it turned out there was a problem with ARP retransmission handling, wrong source address was selected for the retransmission, I’ve posted a PR with a fix proposal:

https://github.com/zephyrproject-rtos/zephyr/pull/40038

 

Another thing is that the reason for the ARP retransmission even being sent  is the fact that e1000 driver does not process any incoming packets early after boot (for about 1 second). I have to little knowledge the particular device/driver to figure out why is this happening, but I’ve asked the question in the aforementioned PR, perhaps we’ll get some answers there.

 

Regards,

Robert

 

From: users@... <users@...> On Behalf Of Brenton Chetty via lists.zephyrproject.org
Sent: wtorek, 2 listopada 2021 20:11
To: users@...
Subject: [Zephyr-users] big_http_download sample localhost download

 

Hello,

I am attempting to download a file hosted on a localhost file server.

- I am hosting a file named "Zephyr" using "python3 -m http.server 8980"
- I am using a zeth interface and the "overlay-e1000.conf" to allow qemu to communicate with localhost. I successfully tested the LWM2M client example to ensure the connection was good.
- I can download the Zephyr file when I input "http://192.0.2.2:8980/zephyr" into a web browser.
- I attempted using the "overlay-ci.conf" file to perform the download.
    CONFIG_SAMPLE_BIG_HTTP_DL_URL="http://192.0.2.2:8980/zephyr"
 however I get the following error:
     Preparing HTTP GET request for http://192.0.2.2:8980/zephyr
     addrinfo @0x6da008: ai_family=1, ai_socktype=1, ai_protocol=6, sa_family=1, sin_port=1423
     Iteration 1 of 1:
     sock = 0
     [00:00:00.000,000] <inf> net_config: Initializing network
     [00:00:00.000,000] <inf> net_config: IPv4 address: 192.0.2.1
     [00:00:00.000,000] <inf> net_config: Running dhcpv4 client...
     Error: connect(sock, ai->ai_addr, ai->ai_addrlen)
     exit

I managed to get the example to download the online vmlinuz file, however I cannot get the sample to download via localhost.
Do you have any suggestions on what steps I could have missed?
Could it be a DNS issue? for downloading the online vmlinuz file I used 8.8.8.8 as CONFIG_DNS_SERVER1. For localhost, I left it at 192.0.2.2

With thanks
Brenton


Brenton Chetty
 

I forgot to mention.

After downloading up to 2680 bytes, I receive the following error:
     Timeout on reading responsees; 2 KiB; 0 MiB


Brenton Chetty
 

Hi Robert

Thank you for your assistance. I managed to get qemu to begin downloading a file on localhost.
However, the file downloads up to a maximum of 2680 bytes, then it fails, and restarts the download.

Do you have any suggestions on why this only happens when downloading on localhost?

Regards
Brenton


On Wed, Nov 3, 2021 at 2:45 PM Lubos, Robert <Robert.Lubos@...> wrote:

Hi Brenton,

 

The DNS server configuration shouldn’t be an issue in this case, as we provide the IP address of the server – in such case the DNS query isn’t even sent as we already know the IP address.

 

I’ve reproduced the problem locally, and it turned out there was a problem with ARP retransmission handling, wrong source address was selected for the retransmission, I’ve posted a PR with a fix proposal:

https://github.com/zephyrproject-rtos/zephyr/pull/40038

 

Another thing is that the reason for the ARP retransmission even being sent  is the fact that e1000 driver does not process any incoming packets early after boot (for about 1 second). I have to little knowledge the particular device/driver to figure out why is this happening, but I’ve asked the question in the aforementioned PR, perhaps we’ll get some answers there.

 

Regards,

Robert

 

From: users@... <users@...> On Behalf Of Brenton Chetty via lists.zephyrproject.org
Sent: wtorek, 2 listopada 2021 20:11
To: users@...
Subject: [Zephyr-users] big_http_download sample localhost download

 

Hello,

I am attempting to download a file hosted on a localhost file server.

- I am hosting a file named "Zephyr" using "python3 -m http.server 8980"
- I am using a zeth interface and the "overlay-e1000.conf" to allow qemu to communicate with localhost. I successfully tested the LWM2M client example to ensure the connection was good.
- I can download the Zephyr file when I input "http://192.0.2.2:8980/zephyr" into a web browser.
- I attempted using the "overlay-ci.conf" file to perform the download.
    CONFIG_SAMPLE_BIG_HTTP_DL_URL="http://192.0.2.2:8980/zephyr"
 however I get the following error:
     Preparing HTTP GET request for http://192.0.2.2:8980/zephyr
     addrinfo @0x6da008: ai_family=1, ai_socktype=1, ai_protocol=6, sa_family=1, sin_port=1423
     Iteration 1 of 1:
     sock = 0
     [00:00:00.000,000] <inf> net_config: Initializing network
     [00:00:00.000,000] <inf> net_config: IPv4 address: 192.0.2.1
     [00:00:00.000,000] <inf> net_config: Running dhcpv4 client...
     Error: connect(sock, ai->ai_addr, ai->ai_addrlen)
     exit

I managed to get the example to download the online vmlinuz file, however I cannot get the sample to download via localhost.
Do you have any suggestions on what steps I could have missed?
Could it be a DNS issue? for downloading the online vmlinuz file I used 8.8.8.8 as CONFIG_DNS_SERVER1. For localhost, I left it at 192.0.2.2

With thanks
Brenton


Lubos, Robert
 

Hi Brenton,

 

The DNS server configuration shouldn’t be an issue in this case, as we provide the IP address of the server – in such case the DNS query isn’t even sent as we already know the IP address.

 

I’ve reproduced the problem locally, and it turned out there was a problem with ARP retransmission handling, wrong source address was selected for the retransmission, I’ve posted a PR with a fix proposal:

https://github.com/zephyrproject-rtos/zephyr/pull/40038

 

Another thing is that the reason for the ARP retransmission even being sent  is the fact that e1000 driver does not process any incoming packets early after boot (for about 1 second). I have to little knowledge the particular device/driver to figure out why is this happening, but I’ve asked the question in the aforementioned PR, perhaps we’ll get some answers there.

 

Regards,

Robert

 

From: users@... <users@...> On Behalf Of Brenton Chetty via lists.zephyrproject.org
Sent: wtorek, 2 listopada 2021 20:11
To: users@...
Subject: [Zephyr-users] big_http_download sample localhost download

 

Hello,

I am attempting to download a file hosted on a localhost file server.

- I am hosting a file named "Zephyr" using "python3 -m http.server 8980"
- I am using a zeth interface and the "overlay-e1000.conf" to allow qemu to communicate with localhost. I successfully tested the LWM2M client example to ensure the connection was good.
- I can download the Zephyr file when I input "http://192.0.2.2:8980/zephyr" into a web browser.
- I attempted using the "overlay-ci.conf" file to perform the download.
    CONFIG_SAMPLE_BIG_HTTP_DL_URL="http://192.0.2.2:8980/zephyr"
 however I get the following error:
     Preparing HTTP GET request for http://192.0.2.2:8980/zephyr
     addrinfo @0x6da008: ai_family=1, ai_socktype=1, ai_protocol=6, sa_family=1, sin_port=1423
     Iteration 1 of 1:
     sock = 0
     [00:00:00.000,000] <inf> net_config: Initializing network
     [00:00:00.000,000] <inf> net_config: IPv4 address: 192.0.2.1
     [00:00:00.000,000] <inf> net_config: Running dhcpv4 client...
     Error: connect(sock, ai->ai_addr, ai->ai_addrlen)
     exit

I managed to get the example to download the online vmlinuz file, however I cannot get the sample to download via localhost.
Do you have any suggestions on what steps I could have missed?
Could it be a DNS issue? for downloading the online vmlinuz file I used 8.8.8.8 as CONFIG_DNS_SERVER1. For localhost, I left it at 192.0.2.2

With thanks
Brenton


Brenton Chetty
 

Hello,

I am attempting to download a file hosted on a localhost file server.

- I am hosting a file named "Zephyr" using "python3 -m http.server 8980"
- I am using a zeth interface and the "overlay-e1000.conf" to allow qemu to communicate with localhost. I successfully tested the LWM2M client example to ensure the connection was good.
- I can download the Zephyr file when I input "http://192.0.2.2:8980/zephyr" into a web browser.
- I attempted using the "overlay-ci.conf" file to perform the download.
    CONFIG_SAMPLE_BIG_HTTP_DL_URL="http://192.0.2.2:8980/zephyr"
 however I get the following error:
     Preparing HTTP GET request for http://192.0.2.2:8980/zephyr
     addrinfo @0x6da008: ai_family=1, ai_socktype=1, ai_protocol=6, sa_family=1, sin_port=1423
     Iteration 1 of 1:
     sock = 0
     [00:00:00.000,000] <inf> net_config: Initializing network
     [00:00:00.000,000] <inf> net_config: IPv4 address: 192.0.2.1
     [00:00:00.000,000] <inf> net_config: Running dhcpv4 client...
     Error: connect(sock, ai->ai_addr, ai->ai_addrlen)
     exit

I managed to get the example to download the online vmlinuz file, however I cannot get the sample to download via localhost.
Do you have any suggestions on what steps I could have missed?
Could it be a DNS issue? for downloading the online vmlinuz file I used 8.8.8.8 as CONFIG_DNS_SERVER1. For localhost, I left it at 192.0.2.2

With thanks
Brenton