Problem: We are uploading videos to the Vercel Blob Store using using API requests in python. Most of the time, this works just fine. However, there are periods of time where the upload fails even after 3 retries. These cluster of failures seem uncorrelated to any changes on our side (including increased traffic). Is this known behavior? Any recommendations of how to solve this? For instance, should we just increase retries to 10, and then we will probabilistically be successful? Or, is there something else going on here.
We are a video generation platform. So, storing videos to the Vercel blob store is critical for our user experience.
Code:
"""Adopted from https://github.com/SuryaSekhar14/vercel_blob which requires Python >=3.11"""
import os
import time
import requests
from mimetypes import guess_type
_VERCEL_BLOB_API_BASE_URL = "https://blob.vercel-storage.com"
_API_VERSION = "7"
_PAGINATED_LIST_SIZE = 1000
_DEFAULT_CACHE_AGE = "31536000"
_MAX_RETRY_REQUEST_RETRIES = 3
def _guess_mime_type(url) -> str:
mime_type, _ = guess_type(url, strict=False)
if mime_type:
return mime_type
else:
return "application/octet-stream"
def _request_factory(
url: str, method: str, backoff_factor: int = 0.5, timeout: int = 10, **kwargs
) -> requests.Response | None:
for attempt in range(1, _MAX_RETRY_REQUEST_RETRIES + 1):
try:
response = requests.request(method, url, timeout=timeout, **kwargs)
if response.status_code not in (502, 503, 504):
return response
except requests.exceptions.RequestException as e:
print(f"Request failed on attempt {attempt} ({e})")
time.sleep(backoff_factor * attempt)
def _response_handler(resp: requests.Response) -> dict:
if resp is None:
raise Exception("Request failed after retries. Please try again.")
elif resp.status_code == 200:
return resp.json()
else:
raise Exception(f"An error occurred: {resp.json()}")
def put(path: str, data: bytes, options: dict = None) -> dict:
if options is None:
options = dict()
headers = {
"access": "public",
"authorization": f"Bearer {os.environ.get('BLOB_READ_WRITE_TOKEN')}",
"x-api-version": _API_VERSION,
"x-content-type": _guess_mime_type(path),
"x-cache-control-max-age": options.get(
"cacheControlMaxAge", _DEFAULT_CACHE_AGE
),
}
if options.get("addRandomSuffix") in ("false", False, "0"):
headers["x-add-random-suffix"] = "0"
resp = _request_factory(
f"{_VERCEL_BLOB_API_BASE_URL}/{path}",
"PUT",
headers=headers,
data=data,
)
return _response_handler(resp)
Project Information:
Python 3.10, The repo where we got this code from required python >=3.11. However, the above code works with Python 3.10, which is a requirement for other parts of our repo.