Sporadic Failures when Uploading Videos to the Blob Store via API

Problem: We are uploading videos to the Vercel Blob Store using using API requests in python. Most of the time, this works just fine. However, there are periods of time where the upload fails even after 3 retries. These cluster of failures seem uncorrelated to any changes on our side (including increased traffic). Is this known behavior? Any recommendations of how to solve this? For instance, should we just increase retries to 10, and then we will probabilistically be successful? Or, is there something else going on here.

We are a video generation platform. So, storing videos to the Vercel blob store is critical for our user experience.

Code:

"""Adopted from https://github.com/SuryaSekhar14/vercel_blob which requires Python >=3.11"""

import os
import time
import requests
from mimetypes import guess_type

_VERCEL_BLOB_API_BASE_URL = "https://blob.vercel-storage.com"
_API_VERSION = "7"
_PAGINATED_LIST_SIZE = 1000
_DEFAULT_CACHE_AGE = "31536000"
_MAX_RETRY_REQUEST_RETRIES = 3


def _guess_mime_type(url) -> str:
    mime_type, _ = guess_type(url, strict=False)

    if mime_type:
        return mime_type
    else:
        return "application/octet-stream"


def _request_factory(
    url: str, method: str, backoff_factor: int = 0.5, timeout: int = 10, **kwargs
) -> requests.Response | None:
    for attempt in range(1, _MAX_RETRY_REQUEST_RETRIES + 1):
        try:
            response = requests.request(method, url, timeout=timeout, **kwargs)
            if response.status_code not in (502, 503, 504):
                return response
        except requests.exceptions.RequestException as e:
            print(f"Request failed on attempt {attempt} ({e})")
            time.sleep(backoff_factor * attempt)


def _response_handler(resp: requests.Response) -> dict:
    if resp is None:
        raise Exception("Request failed after retries. Please try again.")
    elif resp.status_code == 200:
        return resp.json()
    else:
        raise Exception(f"An error occurred: {resp.json()}")


def put(path: str, data: bytes, options: dict = None) -> dict:
    if options is None:
        options = dict()

    headers = {
        "access": "public",
        "authorization": f"Bearer {os.environ.get('BLOB_READ_WRITE_TOKEN')}",
        "x-api-version": _API_VERSION,
        "x-content-type": _guess_mime_type(path),
        "x-cache-control-max-age": options.get(
            "cacheControlMaxAge", _DEFAULT_CACHE_AGE
        ),
    }

    if options.get("addRandomSuffix") in ("false", False, "0"):
        headers["x-add-random-suffix"] = "0"

    resp = _request_factory(
        f"{_VERCEL_BLOB_API_BASE_URL}/{path}",
        "PUT",
        headers=headers,
        data=data,
    )

    return _response_handler(resp)

Project Information:
Python 3.10, The repo where we got this code from required python >=3.11. However, the above code works with Python 3.10, which is a requirement for other parts of our repo.

Please note that we don’t support python officially so it may not work as it intended to.

For larger file such as video, have you tried using multipart upload?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.