Upload route returns cache path before cross-filesystem move completes

[vmikasa]

Describe the bug

When the operating-system temporary directory and Gradio's upload directory are on different filesystems or Windows volumes, /gradio_api/upload can return a content-addressed cache path before the uploaded file has been moved to that path.

Note

Two environments are intentionally reported:

• Original incident: Python 3.12.13, Gradio 5.50.0, Starlette 0.52.1, FaceFusion, and a 507 MB MP4. This is where the intermittent PermissionError and process exit were observed.
• Latest-version isolated reproduction: Python 3.13.5, Gradio 6.20.0, and a generated 32 MiB payload. This deterministically demonstrates that the underlying return-before-publication race still exists in the latest release. Its immediate read usually observes HTTP 403 before the background move starts, rather than reproducing the exact process-level crash.
• Candidate fix validation: based on Gradio main commit 01a06729c94916c2859886dca2971b4857c09741.

The main route calls upload_fn(..., force_move=False). If os.rename() fails across filesystems, upload_fn() returns the source and destination lists, the route schedules move_uploaded_files_to_cache() as a Starlette background task, and the response immediately exposes the destination path. A client can request that path while it is missing, partially copied, or still held by the copying process.

In a FaceFusion UI, an immediate browser Range request for a 507 MB MP4 intermittently overlapped that window and reached starlette.responses.FileResponse._handle_single_range -> anyio.open_file(path, "rb") -> PermissionError. The full service process then exited. The exact failure is timing-dependent, but the premature publication itself is deterministic.

Relevant source paths on main commit 01a06729c94916c2859886dca2971b4857c09741:

• gradio/route_utils.py, upload_fn() around lines 1399-1463
• gradio/routes.py, the main upload route around lines 1786-1813
• gradio/static_server.py around lines 98-116; this caller also passes force_move=False but discards the pending move lists

Deterministic 32 MiB probe results on released Gradio 6.20.0:

Observation	Baseline	Candidate fix
Upload response time	0.971 s	0.985 s
Destination exists at response	no	yes
Complete size at response	missing	33,554,432 bytes
Immediate 1 KiB Range GET	HTTP 403	HTTP 206

Expected behavior: once the upload endpoint returns a cache path, that path should name a complete, readable file. File-serving requests should never observe publication in progress.

Proposed fix: create multipart temporary files inside the configured Gradio upload directory, then close and atomically rename each file into its content-addressed subdirectory before returning. This keeps temporary and final files on the same filesystem, removes the background-copy visibility window, and preserves content-addressed cache reuse.

I prepared a candidate patch and focused tests. Seven upload/static-worker tests pass, along with formatting, syntax, and focused lint checks. A local bundle containing the standalone reproduction, raw baseline/fixed JSON evidence, validation record, and patch is ready and can be provided if useful.

Have you searched existing issues? 🔎

• I have searched and found no existing issues

Reproduction

Prerequisite: on Windows, keep TEMP/TMP on C: and choose a clean Gradio upload directory on D:. Install gradio==6.20.0 and httpx.

app.py:

import os
import shutil
import time
from pathlib import Path

os.environ["GRADIO_TEMP_DIR"] = r"D:\\gradio-upload-race"

import gradio as gr
import gradio.routes as gradio_routes

def delayed_existing_background_move(sources, destinations):
    # This only widens Gradio's existing background-task window.
    time.sleep(3)
    for source, destination in zip(sources, destinations, strict=False):
        Path(destination).parent.mkdir(parents=True, exist_ok=True)
        shutil.move(source, destination)

gradio_routes.move_uploaded_files_to_cache = delayed_existing_background_move

with gr.Blocks() as demo:
    gr.File()

demo.launch(server_name="127.0.0.1", server_port=7862)

probe.py:

import secrets
import time
from pathlib import Path
from urllib.parse import quote

import httpx

payload = Path("payload.bin")
payload.write_bytes(secrets.token_bytes(32 * 1024 * 1024))

with payload.open("rb") as f:
    response = httpx.post(
        "http://127.0.0.1:7862/gradio_api/upload",
        files={"files": (payload.name, f, "application/octet-stream")},
        timeout=120,
    )
response.raise_for_status()
destination = Path(response.json()[0])
print("at response:", destination.exists(), destination.stat().st_size if destination.exists() else None)

encoded = quote(str(destination), safe="")
range_response = httpx.get(
    f"http://127.0.0.1:7862/gradio_api/file={encoded}",
    headers={"Range": "bytes=0-1023"},
)
print("immediate Range GET:", range_response.status_code, len(range_response.content))

time.sleep(5)
print("after wait:", destination.exists(), destination.stat().st_size)

Run python app.py, then python probe.py in another terminal. On the affected release, the first line reports that the returned destination is missing, the immediate Range GET is HTTP 403, and the complete file appears after the background task finishes. The delay does not introduce the ordering bug; it only makes the existing return-before-publication interval deterministic.

Screenshot

Not applicable. This is a server-side upload publication race; the deterministic probe output and raw observations are described above. A local evidence bundle is available on request.

Logs

Original incident log context (FaceFusion, Python 3.12.13, Gradio 5.50.0, Starlette 0.52.1) while the browser requested the uploaded 507 MB MP4 preview:


ERROR: Exception in ASGI application
uvicorn h11_impl.run_asgi
  -> starlette middleware
    -> starlette.responses.FileResponse._handle_single_range
      -> anyio.open_file(path, "rb")
        -> PermissionError: [Errno 13] Permission denied:
           'D:\\...\\gradio\\<hash>\\video.mp4'


The deterministic minimal probe usually observes the earlier part of the same window and receives HTTP 403 because the destination does not yet exist. Once the cross-volume copy starts creating the destination, a concurrent Range read can instead overlap an incomplete or locked file.

System Info

Original intermittent incident

Operating System: Windows 11 Home 64-bit
Application: FaceFusion
Payload: 507 MB MP4
Python: 3.12.13
gradio: 5.50.0
starlette: 0.52.1

Latest-version isolated reproduction

Operating System: Windows 11 Home 64-bit (10.0.26200)
Python: 3.13.5
gradio: 6.20.0
gradio_client: 2.5.0
anyio: 4.14.2
fastapi: 0.141.1
httpx: 0.28.1
python-multipart: 0.0.32
starlette: 1.4.1
uvicorn: 0.52.1

Filesystem layout:
system TEMP/TMP: C:\Users\<user>\AppData\Local\Temp
GRADIO_TEMP_DIR: D:\gradio-upload-race

Candidate fix source base

gradio-app/gradio main commit
01a06729c94916c2859886dca2971b4857c09741

Severity

I can work around it

[SAUMustapure]

Closes #13722

I've prepared a fix for this issue. Here's the comparison: https://github.com/gradio-app/gradio/compare/main...SAUMustapure:fix/upload-atomic?expand=1

The changes include:

1. Make uploads atomic on the cross-filesystem fallback: when os.rename() fails across filesystems, the upload flow now copies the uploaded file into a temporary file inside the upload/cache directory, fsyncs and closes it, then atomically publishes it with os.replace() before returning the destination path.
2. Adds unit and integration tests:
   -----test/test_publish_atomic.py — unit test for the atomic publish helper.
   -----test/test_upload_integration.py — integration test that monkeypatches os.rename to simulate cross-filesystem failure and asserts the returned path is immediately readable.
3. Minor cleanup: removed duplicate/unused imports and applied repository formatting to changed files.

Because the upstream repository restricts PR creation to collaborators, I can't open a PR directly. A maintainer can open the PR from this comparison link when convenient: https://github.com/gradio-app/gradio/compare/main...SAUMustapure:fix/upload-atomic?expand=1

Local test & format commands I ran

1. Create/activate venv (Windows PowerShell): python -m venv .venv . ..venv\Scripts\Activate.ps1
2. Install (minimal/test deps): pip install -e .[dev] (or pip install -e .; pip install pytest fastapi httpx)
3. Focused tests: pytest -q test/test_publish_atomic.py pytest -q test/test_upload_integration.py
4. Formatting (please run before review): scripts\format_backend.bat

Notes for reviewers:
Tradeoff: cross-filesystem uploads are copied synchronously to ensure correctness (no publish-before-ready race); this increases latency for those cases but avoids partial/locked file reads and Windows PermissionError crashes.

This change addresses issue #13722; please let me know if you'd like a different approach.
Thanks — happy to iterate based on reviewer feedback.