Python BrokenPipeError

Feedback
1

Issue Description

After a recent deployment to Vercel production environment, approximately 5% of requests result in BrokenPipeError. The issue appears to be related to connection handling, particularly during concurrent requests.
On an older deployment I got no errors. No dependancies have been updated and no real code changes.

Key characteristics:

Requests complete successfully (200 status code) before the error occurs.
The error appears in logs but doesn’t affect the response to clients.
Error occurs across different endpoints, including health checks and OPTIONS requests.
More frequent when there are multiple concurrent requests.
Cannot be reproduced locally or in Vercel preview deployments.

Additional Context

The application is a API proxy that forwards requests using httpx.
All functions are asynchronous.
Issue appears to be environment-specific to Vercel production (I cannot reproduce it in preview).
Error occurs even on simple endpoints like health checks.
Same dependency versions are used as in previous working deployments.
Might be caused by this: [python] add support for streaming by QuiiBz · Pull Request #12557 · vercel/vercel · GitHub update?

Impact
While the errors don’t affect client responses (requests complete successfully), they pollute the logs and might indicate underlying issues with connection handling that could impact performance or stability.

Environment

Python 3.9
FastAPI 0.115.3
Starlette 0.41.0
uvicorn  0.30.6
httpx 0.24.0
pipenv
Node 18.x
Other dependencies: aiomysql 0.2.0, loguru 0.7.3, pydantic 2.7.3, sqlalchemy 2.0.37, uvicorn 0.30.6

Error log from POST request:

File "/var/lang/lib/python3.9/socketserver.py", line 683, in process_request_thread
self.finish_request(request, client_address)
File "/var/lang/lib/python3.9/socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/var/lang/lib/python3.9/socketserver.py", line 747, in __init__
self.handle()
File "/var/lang/lib/python3.9/http/server.py", line 433, in handle
self.handle_one_request()
File "/var/task/vc__handler__python.py", line 203, in handle_one_request
self.handle_request()
File "/var/task/vc__handler__python.py", line 376, in handle_request
loop.run_until_complete(asgi_task)
File "/var/lang/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "/var/task/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/var/task/starlette/applications.py", line 113, in __call__
await self.middleware_stack(scope, receive, send)
File "/var/task/starlette/middleware/errors.py", line 187, in __call__
raise exc
File "/var/task/starlette/middleware/errors.py", line 165, in __call__
await self.app(scope, receive, _send)
File "/var/task/starlette/middleware/base.py", line 189, in __call__
response_sent.set()
File "/var/lang/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/var/task/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/var/task/starlette/middleware/base.py", line 188, in __call__
await response(scope, wrapped_receive, send)
File "/var/task/starlette/middleware/base.py", line 225, in __call__
await send({"type": "http.response.body", "body": b"", "more_body": False})
File "/var/task/starlette/middleware/errors.py", line 162, in _send
await send(message)
File "/var/task/vc__handler__python.py", line 368, in send
self.wfile.write(event['body'])
File "/var/lang/lib/python3.9/socketserver.py", line 826, in write
self._sock.sendall(b)
BrokenPipeError: [Errno 32] Broken pipe

Error log from OPTIONS request:

.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/var/task/starlette/applications.py", line 113, in __call__
await self.middleware_stack(scope, receive, send)
File "/var/task/starlette/middleware/errors.py", line 187, in __call__
raise exc
File "/var/task/starlette/middleware/errors.py", line 165, in __call__
await self.app(scope, receive, _send)
File "/var/task/starlette/middleware/base.py", line 189, in __call__
response_sent.set()
File "/var/lang/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/var/task/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/var/task/starlette/middleware/base.py", line 188, in __call__
await response(scope, wrapped_receive, send)
File "/var/task/starlette/middleware/base.py", line 225, in __call__
await send({"type": "http.response.body", "body": b"", "more_body": False})
File "/var/task/starlette/middleware/errors.py", line 162, in _send
await send(message)
File "/var/task/vc__handler__python.py", line 368, in send
self.wfile.write(event['body'])
File "/var/lang/lib/python3.9/socketserver.py", line 826, in write
self._sock.sendall(b)
BrokenPipeError: [Errno 32] Broken pipe
----------------------------------------
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 60408)
+ Exception Group Traceback (most recent call last):
|   File "/var/task/starlette/_utils.py", line 76, in collapse_excgroups
|     yield
|   File "/var/task/starlette/middleware/base.py", line 189, in __call__
|     response_sent.set()
|   File "/var/task/anyio/_backends/_asyncio.py", line 767, in __aexit__
|     raise BaseExceptionGroup(
| exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
|   File "/var/lang/lib/python3.9/socketserver.py", line 683, in process_request_thread
|     self.finish_request(request, client_address)
|   File "/var/lang/lib/python3.9/socketserver.py", line 360, in finish_request
|     self.RequestHandlerClass(request, client_address, self)
|   File "/var/lang/lib/python3.9/socketserver.py", line 747, in __init__
|     self.handle()
|   File "/var/lang/lib/python3.9/http/server.py", line 433, in handle
|     self.handle_one_request()
|   File "/var/task/vc__handler__python.py", line 203, in handle_one_request
|     self.handle_request()
|   File "/var/task/vc__handler__python.py", line 376, in handle_request
|     loop.run_until_complete(asgi_task)
|   File "/var/lang/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
|     return future.result()
|   File "/var/task/fastapi/applications.py", line 1054, in __call__
|     await super().__call__(scope, receive, send)
|   File "/var/task/starlette/applications.py", line 113, in __call__
|     await self.middleware_stack(scope, receive, send)
|   File "/var/task/starlette/middleware/errors.py", line 187, in __call__
|     raise exc
|   File "/var/task/starlette/middleware/errors.py", line 165, in __call__
|     await self.app(scope, receive, _send)
|   File "/var/task/starlette/middleware/base.py", line 189, in __call__
|     response_sent.set()
|   File "/var/lang/lib/python3.9/contextlib.py", line 137, in __exit__
|     self.gen.throw(typ, value, traceback)
|   File "/var/task/starlette/_utils.py", line 82, in collapse_excgroups
|     raise exc
|   File "/var/task/starlette/middleware/base.py", line 188, in __call__
|     await response(scope, wrapped_receive, send)
|   File "/var/task/starlette/middleware/base.py", line 225, in __call__
|     await send({"type": "http.response.body", "body": b"", "more_body": False})
|   File "/var/task/starlette/middleware/errors.py", line 162, in _send
|     await send(message)

Maybe unrelated

I sometimes get this in the log:

/var/task/vc__handler__python.py:346: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
  app_queue = asyncio.Queue(loop=loop)
2

Hi, @vrejf! Thank you for your feedback.

I’d love to share this internally, it’d be super helpful to have a minimal reproducible example for us to show :smile:

1 Like
4

I have been trying to reproduce it and this is what I can come up with. Now every fourth request to the Health path will return the error.
It does not happen locally and did not give errors some weeks ago when deployed.

[packages]
fastapi = "==0.115.3"
uvicorn = "==0.30.6"


[requires]
python_version = "3.9"

# main.py

from fastapi import FastAPI
from starlette.middleware.base import BaseHTTPMiddleware
import asyncio


app = FastAPI()


class RequestCounterMiddleware(BaseHTTPMiddleware):
    def __init__(self, app):
        super().__init__(app)
        self.request_count = 0
        self.lock = asyncio.Lock()

    async def dispatch(self, request, call_next):
        async with self.lock:
            self.request_count += 1
            current_count = self.request_count

        response = await call_next(request)

        response.headers["X-Request-Count"] = str(current_count)
        return response


app.add_middleware(RequestCounterMiddleware)


@app.get("/")
async def root():
    return {"message": "Hello World"}


@app.head("/health")
async def health():
    return {"status": "ok"}