github-featured/e2b-dev--e2b
2
0
Fork 0
Code Issues Pull Requests Deployments Models Agent Notes Packages Projects Releases Wiki Activity

feat(python-sdk): move template build-context uploads onto pyqwest (#1603)

Browse Source 
## What

Stacked on #1602 (which is stacked on #1601). Migrates the **template
build-context uploads** (streaming the build archive to S3 presigned
URLs in `build_api.upload_file`) onto
[pyqwest](https://github.com/curioswitch/pyqwest) via its
httpx-compatible transport adapter.

Originally deferred from #1601 because S3 presigned URLs reject chunked
transfer encoding and Content-Length framing through reqwest was
unverified. Verified at the wire level (raw-socket capture server):
httpx's Content-Length — derived from the spooled archive (sync) or set
explicitly on the async-iterator body (async) — is forwarded by the
adapter and reqwest keeps Content-Length framing for streamed bodies, no
chunked fallback.


> [!NOTE]
> Rebased onto #1601, which locks **pyqwest 0.9.0**. Two knock-on
changes here: the upload client uses the stock
`PyqwestTransport`/`AsyncPyqwestTransport` (0.9.0's adapter subsumes
what the SDK's transport subclasses did, so #1601 deleted them), and it
builds its proxy from `proxy_to_config(...)` following #1601's rename.

## How

- `e2b/template_sync/build_api.py` / `template_async/build_api.py`:
`upload_file` uses a one-off pyqwest transport instead of the generated
client's httpx transport.
- **Redirects stay with the httpx client.** pyqwest 0.9.0 makes
reqwest's internal redirect following configurable, so it's turned off
on the upload transport: otherwise reqwest would replay the entire
archive body against a new location without httpx knowing. The httpx
client inherits the API client's `follow_redirects` (off), matching the
httpx transport this replaced — so an unexpected hop surfaces as a
failed upload rather than a silent re-upload.
- `verify_ssl=False` on the generated client is no longer honored for
uploads (pyqwest has no insecure-TLS option), and `http2=False` is gone
(S3 negotiates HTTP/1.1 via ALPN anyway).
- The 1-hour upload timeout now bounds the entire upload rather than
each socket write — arguably the intended meaning for that endpoint.

## Testing

- `tests/{sync,async}/*/test_upload_file.py` (the #1243 regression tests
— Content-Length present and equal to the body, no chunked encoding)
pass through pyqwest; the capture handlers now compare header names
case-insensitively since hyper lowercases them where httpcore
title-cased.
- New in both mirrors: `test_upload_file_leaves_redirects_to_httpx` — a
307 on the upload URL surfaces as `FileUploadException` and the capture
server sees exactly one PUT, guarding against reqwest silently following
the hop and replaying the archive.
- Lint (`ruff`), typecheck (`ty`), upload-file suites: green (10/10).

## Usage example

No API changes — template builds upload their context exactly as before:

```python
from e2b import Template

template = Template().from_image("ubuntu:22.04").copy("data/", "/data")
Template.build(template, alias="my-template")   # archive upload now goes through pyqwest
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Mish Ushakov
2026-08-10 19:46:38 +02:00
committed by GitHub
parent 458c2c4362
commit b3a7c9f44a
5 changed files with 156 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.changeset/python-pyqwest-template-uploads.md
+12
View File
@@ -0,0 +1,12 @@
---
"@e2b/python-sdk": minor
---
Move template build-context uploads (to S3 presigned URLs) onto
[`pyqwest`](https://pypi.org/project/pyqwest/) via its httpx-compatible
transport adapter. Content-Length framing for the streamed archive body is
preserved (S3 rejects chunked transfer encoding), and redirects stay with the
httpx client instead of being followed inside the transport. The 1-hour upload
timeout now bounds the entire upload rather than each socket operation, and
`verify_ssl=False` on the client is no longer honored for uploads (pyqwest
has no insecure-TLS option).
packages/python-sdk/e2b/template_async/build_api.py
+22 -5
View File
@@ -4,8 +4,10 @@ from types import TracebackType
from typing import Callable, Optional, List, Union
import httpx
from pyqwest import HTTPTransport
from pyqwest.httpx import AsyncPyqwestTransport
from e2b.api import handle_api_exception
from e2b.api import handle_api_exception, proxy_to_config
from e2b.io_utils import aiter_io_chunks
from e2b.api.client.api.templates import (
post_v3_templates,
@@ -118,6 +120,7 @@ async def upload_file(
upload_timeout = (
request_timeout if request_timeout is not None else FILE_UPLOAD_TIMEOUT_SECONDS
)
upload_proxy = proxy_to_config(getattr(api_client, "_proxy", None))
try:
tar_file = tar_file_stream(
file_name, context_path, ignore_patterns, resolve_symlinks, gzip
@@ -125,16 +128,30 @@ async def upload_file(
try:
size = os.fstat(tar_file.fileno()).st_size
# Through the pyqwest adapter the upload timeout is a
# whole-request deadline for the entire transfer, not a per-write
# bound as with the httpx transport this replaced.
async with httpx.AsyncClient(
timeout=httpx.Timeout(upload_timeout),
verify=api_client._verify_ssl,
follow_redirects=api_client._follow_redirects,
proxy=getattr(api_client, "_proxy", None),
http2=False,
transport=AsyncPyqwestTransport(
HTTPTransport(
tls_include_system_certs=True,
proxy=(
upload_proxy.to_pyqwest()
if upload_proxy is not None
else None
),
# Redirects belong to the httpx client above, not to
# reqwest.
follow_redirects=False,
)
),
) as client:
# Stream the archive from disk via an async iterator. The
# explicit Content-Length suppresses chunked transfer
# encoding, which S3 presigned URLs reject.
# encoding, which S3 presigned URLs reject; reqwest keeps the
# Content-Length framing for the streamed body.
response = await client.put(
url,
content=aiter_io_chunks(tar_file),
packages/python-sdk/e2b/template_sync/build_api.py
+22 -5
View File
@@ -3,8 +3,10 @@ from types import TracebackType
from typing import Callable, Optional, List, Union
import httpx
from pyqwest import SyncHTTPTransport
from pyqwest.httpx import PyqwestTransport
from e2b.api import handle_api_exception
from e2b.api import handle_api_exception, proxy_to_config
from e2b.api.client.api.templates import (
post_v3_templates,
get_templates_template_id_files_hash,
@@ -116,21 +118,36 @@ def upload_file(
upload_timeout = (
request_timeout if request_timeout is not None else FILE_UPLOAD_TIMEOUT_SECONDS
)
upload_proxy = proxy_to_config(getattr(api_client, "_proxy", None))
try:
tar_file = tar_file_stream(
file_name, context_path, ignore_patterns, resolve_symlinks, gzip
)
try:
# Through the pyqwest adapter the upload timeout is a
# whole-request deadline for the entire transfer, not a per-write
# bound as with the httpx transport this replaced.
with httpx.Client(
timeout=httpx.Timeout(upload_timeout),
verify=api_client._verify_ssl,
follow_redirects=api_client._follow_redirects,
proxy=getattr(api_client, "_proxy", None),
http2=False,
transport=PyqwestTransport(
SyncHTTPTransport(
tls_include_system_certs=True,
proxy=(
upload_proxy.to_pyqwest()
if upload_proxy is not None
else None
),
# Redirects belong to the httpx client above, not to
# reqwest.
follow_redirects=False,
)
),
) as client:
# httpx streams the archive from disk in chunks and sets
# Content-Length from the file size—S3 presigned URLs reject
# chunked transfer encoding.
# chunked transfer encoding, and reqwest keeps the
# Content-Length framing for the streamed body.
response = client.put(url, content=tar_file)
response.raise_for_status()
finally:
packages/python-sdk/tests/async/template_async/test_upload_file.py
+50 -5
View File
@@ -1,11 +1,14 @@
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer
from typing import Any, Dict
from unittest import mock
import httpx
import pytest
from e2b.api.client.client import AuthenticatedClient
from e2b.template import utils as template_utils
from e2b.exceptions import FileUploadException
from e2b.template.consts import FILE_UPLOAD_TIMEOUT_SECONDS
from e2b.template_async.build_api import upload_file
@@ -22,14 +25,23 @@ from e2b.template_async.build_api import upload_file
def _make_server():
state = {"headers": None, "body_length": 0}
state: Dict[str, Any] = {"headers": None, "body_length": 0, "paths": []}
class Handler(BaseHTTPRequestHandler):
def do_PUT(self):
state["headers"] = dict(self.headers)
# hyper (pyqwest) sends lowercase header names where httpcore
# title-cased them; compare case-insensitively.
state["headers"] = {k.lower(): v for k, v in self.headers.items()}
state["paths"].append(self.path)
length = int(self.headers.get("Content-Length", 0))
body = self.rfile.read(length) if length else b""
state["body_length"] = len(body)
if self.path.startswith("/redirect"):
self.send_response(307)
self.send_header("Location", "/upload")
self.send_header("Content-Length", "0")
self.end_headers()
return
self.send_response(200)
self.end_headers()
@@ -72,15 +84,48 @@ async def test_upload_file_sets_content_length_and_no_chunked_encoding(tmp_path)
thread.join(timeout=5)
assert state["headers"] is not None
content_length = state["headers"].get("Content-Length")
content_length = state["headers"].get("content-length")
assert content_length is not None
assert int(content_length) > 0
assert int(content_length) == state["body_length"]
transfer_encoding = state["headers"].get("Transfer-Encoding")
transfer_encoding = state["headers"].get("transfer-encoding")
if transfer_encoding is not None:
assert "chunked" not in transfer_encoding.lower()
assert "Authorization" not in state["headers"]
assert "authorization" not in state["headers"]
async def test_upload_file_leaves_redirects_to_httpx(tmp_path):
# reqwest would otherwise follow redirects inside the transport, replaying
# the archive body against the new location without httpx knowing. The
# upload client follows the API client's setting (off), so an unexpected
# hop must surface as a failed upload rather than being retried behind
# httpx's back.
(tmp_path / "hello.txt").write_text("hello world")
server, thread, state = _make_server()
host, port = server.server_address
try:
client = AuthenticatedClient(base_url="http://test", token="test")
assert client._follow_redirects is False
with pytest.raises(FileUploadException, match="307"):
await upload_file(
api_client=client,
file_name="*.txt",
context_path=str(tmp_path),
url=f"http://{host}:{port}/redirect",
ignore_patterns=[],
resolve_symlinks=False,
gzip=True,
stack_trace=None,
)
finally:
server.shutdown()
server.server_close()
thread.join(timeout=5)
assert state["paths"] == ["/redirect"]
async def _capture_upload_timeout(tmp_path, request_timeout=None):
packages/python-sdk/tests/sync/template_sync/test_upload_file.py
+50 -5
View File
@@ -1,11 +1,14 @@
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer
from typing import Any, Dict
from unittest import mock
import httpx
import pytest
from e2b.api.client.client import AuthenticatedClient
from e2b.template import utils as template_utils
from e2b.exceptions import FileUploadException
from e2b.template.consts import FILE_UPLOAD_TIMEOUT_SECONDS
from e2b.template_sync.build_api import upload_file
@@ -18,14 +21,23 @@ from e2b.template_sync.build_api import upload_file
def _make_server():
state = {"headers": None, "body_length": 0}
state: Dict[str, Any] = {"headers": None, "body_length": 0, "paths": []}
class Handler(BaseHTTPRequestHandler):
def do_PUT(self):
state["headers"] = dict(self.headers)
# hyper (pyqwest) sends lowercase header names where httpcore
# title-cased them; compare case-insensitively.
state["headers"] = {k.lower(): v for k, v in self.headers.items()}
state["paths"].append(self.path)
length = int(self.headers.get("Content-Length", 0))
body = self.rfile.read(length) if length else b""
state["body_length"] = len(body)
if self.path.startswith("/redirect"):
self.send_response(307)
self.send_header("Location", "/upload")
self.send_header("Content-Length", "0")
self.end_headers()
return
self.send_response(200)
self.end_headers()
@@ -68,15 +80,48 @@ def test_upload_file_sets_content_length_and_no_chunked_encoding(tmp_path):
thread.join(timeout=5)
assert state["headers"] is not None
content_length = state["headers"].get("Content-Length")
content_length = state["headers"].get("content-length")
assert content_length is not None
assert int(content_length) > 0
assert int(content_length) == state["body_length"]
transfer_encoding = state["headers"].get("Transfer-Encoding")
transfer_encoding = state["headers"].get("transfer-encoding")
if transfer_encoding is not None:
assert "chunked" not in transfer_encoding.lower()
assert "Authorization" not in state["headers"]
assert "authorization" not in state["headers"]
def test_upload_file_leaves_redirects_to_httpx(tmp_path):
# reqwest would otherwise follow redirects inside the transport, replaying
# the archive body against the new location without httpx knowing. The
# upload client follows the API client's setting (off), so an unexpected
# hop must surface as a failed upload rather than being retried behind
# httpx's back.
(tmp_path / "hello.txt").write_text("hello world")
server, thread, state = _make_server()
host, port = server.server_address
try:
client = AuthenticatedClient(base_url="http://test", token="test")
assert client._follow_redirects is False
with pytest.raises(FileUploadException, match="307"):
upload_file(
api_client=client,
file_name="*.txt",
context_path=str(tmp_path),
url=f"http://{host}:{port}/redirect",
ignore_patterns=[],
resolve_symlinks=False,
gzip=True,
stack_trace=None,
)
finally:
server.shutdown()
server.server_close()
thread.join(timeout=5)
assert state["paths"] == ["/redirect"]
def _capture_upload_timeout(tmp_path, request_timeout=None):