Python Package Caching Strategy for Faster Deployments?

Hi Vercel team,

We’re running a Python FastAPI service with ~40 dependencies (including aiohttp, Flask-Cors, pymongo, etc.) on Vercel and facing long deployment times (10-15 minutes) due to package reinstallation on every deploy. Our dependencies rarely change between deployments, so reinstalling all packages each time seems inefficient.

Current situation:

  • ~40 Python packages with specific versions pinned
  • Each deployment takes 10-15 minutes, mostly spent on package installation
  • Dependencies rarely change between deployments
  • Build cache shows only “29.00 B” being uploaded, suggesting the Python package cache isn’t being preserved
  • Using Python 3.12 runtime

We’ve tried:

  • Using pip’s cache directory (–cache-dir)
  • Configuring custom install commands in vercel.json
  • Various .vercelignore configurations to preserve cache
  • Setting up wheel directories
  • Using Pipfile/Pipfile.lock instead of requirements.txt
  • Setting cacheDirectories in vercel.json:
{
  "cacheDirectories": [
    ".vercel/cache/python",
    ".vercel/cache/pip"
  ]
}

Specific questions:

  1. Is there a recommended way to cache Python packages between deployments?
  2. Is there a difference in caching behavior between requirements.txt and Pipfile approaches?
  3. Are there specific directories we should be caching for Python projects?
  4. Should we be using a different build command or configuration for optimal caching?
  5. Do certain Python package types (wheels vs source distributions) cache better than others?

Other frameworks (Node.js, etc.) seem to have built-in caching support, but we haven’t found clear documentation for Python projects. Our goal is to reduce deployment times by reusing previously installed packages when dependencies haven’t changed.

Any guidance on best practices for Python package caching would be greatly appreciated. We’re open to restructuring our project or changing our dependency management approach if it would help with caching.

Thank you!

Hi, @suhaim! Welcome to the Vercel Community :smile:

Vercel automatically caches dependencies for Python projects . For best results, use a requirements.txt file in your project’s root directory with pinned package versions. So it’s better to stick with requirements.txt rather than Pipfile for optimal caching performance on Vercel.

You don’t need to manually specify cache directories; Vercel handles this automatically for Python projects .

The default install command pip install -r requirements.txt should work well in most cases.

Remember, the build cache has a maximum size of 1 GB and is retained for one month. If you’re still facing issues after implementing these suggestions, don’t hesitate to reach out to Vercel support for more specific guidance.

I hope this helps streamline your deployment process! Let me know if you have any other questions.

Thanks for getting back to me so quickly.

I’m now wondering if we have an issue with our project structure, seeing as the build cache has only ever shown 29.00 B being uploaded (whereas we’d expect 50-100mb based on our dependencies).

Do you see anything wrong with our project structure and/or vercel.json? I appreciate your help – Vercel is new to us, and we’re enjoying it so far!

Project structure:

readiverse-service
├── .github
│   └── workflows
│       └── file1.yml
├── api
│   ├── api
│   │   ├── payment
│   │   │   └── razorpay
│   │   │       ├── file1.py
│   │   │       ├── file2.py
│   │   │       └── file3.py
│   │   ├── routes
│   │   │   └── file1.py
│   │   ├── file1.py
│   │   ├── file2.py
│   │   ├── file3.py
│   │   ├── file4.py
│   │   ├── file5.py
│   │   ├── file6.py
│   │   ├── file7.py
│   │   └── file8.py
│   ├── helpers
│   │   ├── file1.py
│   │   └── file2.py
│   ├── middleware
│   │   └── file1.py
│   ├── models
│   │   ├── file1.py
│   │   └── file2.py
│   ├── prompts
│   │   ├── file1.json
│   │   ├── file2.json
│   │   ├── file3.py
│   │   ├── file4.py
│   │   ├── file5.py
│   │   └── __pycache__
│   │       ├── file1.pyc
│   │       └── file2.pyc
│   ├── repositories
│   │   └── file1.py
│   ├── services
│   │   ├── file1.py
│   │   ├── file2.py
│   │   ├── file3.py
│   │   ├── file4.py
│   │   ├── file5.py
│   │   ├── file6.py
│   │   ├── file7.py
│   │   ├── file8.py
│   │   ├── file9.py
│   │   ├── file10.py
│   │   ├── file11.py
│   │   ├── file12.py
│   │   ├── file13.py
│   │   ├── file14.py
│   │   ├── file15.py
│   │   ├── file16.py
│   │   ├── file17.py
│   │   ├── file18.py
│   │   ├── file19.py
│   │   ├── file20.py
│   │   ├── file21.py
│   │   └── __pycache__
│   │       ├── file1.pyc
│   │       └── file2.pyc
│   ├── templates
│   │   └── email_templates
│   │       ├── file1.html
│   │       └── file2.html
│   ├── webhooks
│   │   ├── file1.py
│   │   └── file2.py
│   ├── __pycache__
│   │   ├── file1.pyc
│   │   └── file2.pyc
│   ├── file1.py
│   ├── file2.py
│   └── file3.py
├── .gitignore
├── README.md
├── requirements.txt
├── test_ai_generation.py
└── vercel.json

Vercel.json:

{
  "routes": [
    {
      "src": "/.*",
      "dest": "api/main.py"
    }
  ],
  "functions": {
    "api/**/*.py": {
      "maxDuration": 300
    }
  }
}

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.