requests: HTTP/1.1 with Content-Length and streaming raw

[pablogventura]

Summary

• Send HTTP/1.1 requests (was HTTP/1.0).
• Parse response status/headers via a small _http module.
• Support Content-Length response bodies without buffering in request().
• Preserve .raw as a live BodyStream wrapper; .content remains lazy.

Closes the approach in #1119 (which buffered the full body and closed the socket). Chunked responses, max_body, and relative redirects are intentionally left for follow-up PRs.

Testing

• micropython test_requests.py (unix port) — 13 mock tests including incremental .raw.read().
• CI: ruff, codespell, package tests.

Trade-offs and Alternatives

• Chunked response bodies still raise ValueError (unchanged from master until a follow-up PR).
• BodyStream adds a small wrapper object (~two fields) instead of buffering the body in request().
• stream=True is not implemented yet (issue requests: stream parameter is noop #777).

Generative AI

I used generative AI tools when creating this PR, but a human has checked the
code and is responsible for the code and the description above.

[pablogventura]

ESP32 hardware testing

Tested on device connected via /dev/ttyACM0:

Board	ESP32
Firmware	MicroPython v1.29.0-preview.337 (2026-06-01)
WiFi	LosMius (STA), device IP 192.168.1.151
Package deployed	/lib/requests/__init__.py + /lib/requests/_http.py from this PR

Mock tests (test_requests.py) — 13/13 OK

All existing mock tests pass on-device (outbound HTTP/1.1, Content-Length inbound, incremental .raw.read(), chunked response still raises ValueError).

Live tests (LAN HTTP server at 192.168.1.72:58080) — 4/4 OK

Test	Result
GET /get with Content-Length body	200, 11 bytes
GET /bytes/8 via .content	abcdefgh, socket closed after read
GET /bytes/10 incremental .raw.read(3) + .raw.read(3) + .raw.read()	3+3+4 bytes → 0123456789
GET /chunked	ValueError: Unsupported Transfer-Encoding: chunked (expected for v1)

Server log confirmed outbound requests use HTTP/1.1:

GET /get HTTP/1.1
GET /bytes/8 HTTP/1.1
GET /bytes/10 HTTP/1.1
GET /chunked HTTP/1.1

Notes

• httpbin.org returned 503 from this network, so live body tests used a local test server instead.
• Mock tests replace sys.modules["socket"]; live tests reload usocket afterwards before hitting the network.

[pablogventura]

Done — I moved everything from _http.py into __init__.py and dropped the extra module (commit 17f78a5).

Re-ran test_requests.py on the unix port locally (13/13). CI is green again on my side.

Let me know if anything else should change before you take another look.

[pablogventura]

Also re-tested on the same ESP32 as in my earlier comment (ESP32-D0WD on /dev/ttyACM0, MicroPython v1.29 preview, WiFi at home). After inlining _http.py into __init__.py, mock tests still pass on-device (test_requests.py, 13/13) and live HTTP over the LAN worked as before.

[pablogventura]

Thanks for the review, @dpgeorge — much appreciated.

I've inlined read_status_line, read_headers, and open_body directly into request() (commit 03a85e3). BodyStream remains as the only helper class. _header_get was folded into the body-setup block as well.

CI should re-run shortly. Happy to adjust anything else.

[pablogventura]

Re-tested after inlining the helpers and the ruff fix (commit d3c3628):

Unix port - micropython test_requests.py: 13/13 OK

ESP32 (ESP32-D0WD, /dev/ttyACM0, MicroPython v1.29 preview, WiFi at home):

• Mock tests (test_requests.py): 13/13 OK
• Live HTTP over LAN (local test server): 4/4 OK (Content-Length, incremental .raw.read(), chunked -> ValueError)

CI re-running on d3c3628.

[dpgeorge]

I don't think this check is needed. If line is empty then the len(parts) < 2 will catch that.

[pablogventura]

Thanks again for the detailed review, @dpgeorge.

Addressed the latest round in the new commit:

• Restored l-based status/header parsing (no line/parts rename)
• Removed the extra empty status-line check
• Restored reason = "" / if len(l) > 2: as before
• Restored resp_d = None when parse_headers is False
• Restored in-loop Transfer-Encoding / Location handling from master
• Kept only the minimal additions: HTTP/1.1, BodyStream, Content-Length limit

Re-tested on unix port: test_requests.py 13/13 OK.

Happy to adjust further if needed.

[pablogventura]

Thanks for the follow-up comments, @dpgeorge.

Addressed in commit 8b20e5c:

• Detect Content-Length while populating resp_d in the header loop (removed the extra scan at the end)
• Use Response(s) when there is no Content-Length (unchanged behaviour from master)
• Wrap in BodyStream only when Content-Length is present; BodyStream now always takes a byte count

Re-tested on unix port: test_requests.py 13/13 OK. CI green on 8b20e5c.

[pablogventura]

Good question. I checked CPython 3.12 with requests 2.31 / urllib3 2.0.7 on a response with Content-Length: 10 but only 3 body bytes before close:

• response.raw.read(n) with partial data available: returns the short data (e.g. read(10) -> b"abc"), no exception on that call.
• A later read once the socket is exhausted but bytes are still expected per Content-Length: urllib3 raises IncompleteRead (requests wraps it as ProtocolError / ChunkedEncodingError).
• .content (full read): raises IncompleteRead.

So streaming behaviour is: partial reads are fine; exception when you try to read past EOF with Content-Length still outstanding. That matches what BodyStream does (ValueError instead of IncompleteRead, for simplicity).

Applied your suggestion in 6372c93: the check is now if not data:.
Re-tested on unix port: test_requests.py 13/13 OK.

[dpgeorge]

Thanks for updating.

Looking at how requests is used by mip, mip will call n= response.raw.readinto(buf) and that will not work with the changes in this PR. To fix that, BodyStream will need to implement readinto(self, buf). Could you add that?

Also regarding the tests: you can see in #1128 that I enhanced the mock socket so you can call set_server_responses() to set the response packet(s) from the server. It might be a good idea to follow that approach, instead of patching socket.socket as done here.

(Eventually the test for requests should be rewritten as a unittest, but that's for a separate PR.)

[pablogventura]

Thanks for the follow-up, @dpgeorge.

Added BodyStream.readinto() in commit baa3d77. It delegates to _sock.readinto() on a bounded memoryview, respects Content-Length, returns 0 when the body is exhausted, and raises the same ValueError as read() on premature EOF.

This covers the mip path: _download_file() passes response.raw to _chunk(), which loops on readinto(buf).

Added test_raw_readinto_content_length, which mirrors that loop (small buffer, multiple reads until n == 0). Re-tested on unix port: test_requests.py 14/14 OK.

Left the set_server_responses() mock refactor for a follow-up, per your note on #1128.

[dpgeorge]

So that this doesn't unnecessarily create memoryview objects, I suggest only slicing if needed, ie:

if len(buf) > self._remaining:
    buf = memoryview(buf)[:self._remaining]
got = self._sock.readinto(buf)

[pablogventura]

Thanks for the review, @dpgeorge.

Addressed the latest round in f23463f and cf1492d:

• BodyStream.readinto(): only slices into a memoryview when the buffer is larger than the remaining body, as suggested.
• Test mock: Socket.readinto() now just returns self._read_buffer.readinto(buf).
• Tests: restored the original style with direct calls at the end and exact request-byte asserts. The existing tests only change HTTP/1.0 to HTTP/1.1; the five new cases are appended after them.
• README: removed (separate PR) from the follow-up section.

Re-tested on the unix port: test_requests.py 14/14 OK.

[dpgeorge]

Thanks for updating. This looks in good shape now. I tested with mip and it seems to work well.

[dpgeorge]

I squashed the commits into one. And changed the version in the manifest to 1.0.0 because this is a pretty big change to use HTTP/1.1.