Skip to content

feat(file-storage) File Storage Java SDK#99

Open
kowser-orkes wants to merge 1 commit into
mainfrom
feature/file-storage-wip
Open

feat(file-storage) File Storage Java SDK#99
kowser-orkes wants to merge 1 commit into
mainfrom
feature/file-storage-wip

Conversation

@kowser-orkes
Copy link
Copy Markdown

@kowser-orkes kowser-orkes commented Apr 16, 2026

Pull Request type

  • Bugfix
  • Feature
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • WHOSUSING.md
  • Other (please describe):

NOTE: Please remember to run ./gradlew spotlessApply to fix any format violations.

Changes in this PR

Summary

Adds file storage support to the Java SDK, letting workers upload and download large files through Conductor without passing bytes through the server.

  • FileHandler — developer-facing file reference; wraps a local file before upload (LocalFileHandler) or a server-assigned conductor://file/<fileId> after upload (ManagedFileHandler)
  • FileUploader — developer-facing upload API; implemented by WorkflowFileClient, which binds a workflowId at construction so callers never pass it per-call
  • FileClient — internal REST client composing ConductorClient + Map<StorageType, FileStorageBackend>; handles single-part and multipart upload flows, upload confirmation, and download URL fetching
  • FileStorageBackend — per-backend byte-transfer interface (S3, Azure Blob, GCS, Local); storage type is auto-selected from the server's response — no SDK-side type config needed
  • AnnotatedWorker changesFileHandler parameters are auto-resolved from conductor://file/ strings in inputData; FileHandler return values are stored in outputData for TaskRunner to upload
  • TaskRunner changes — after worker execution, scans outputData for LocalFileHandler entries (null fileHandleId), uploads each, and replaces with the resulting ManagedFileHandler; already-uploaded handlers are left untouched
  • Spring auto-configurationFileClient and FileClientProperties wired in ConductorClientAutoConfiguration; OrkesClients.getFileClient() follows the existing factory pattern

Key design decisions

  • Files are strictly owned by their creator workflow — download access requires the caller's workflow to be in the file's workflow family (self + ancestors + descendants), resolved by WorkflowFamilyResolver
  • ManagedFileHandler uses a predictable cache path (<cacheDir>/<fileId>_<fileName>) so two tasks on the same worker node sharing the same fileId skip re-download on the second access
  • Storage type is reported by the server in FileUploadResponse; the SDK validates it against registered backends at upload time and fails fast with a clear error if none matches
  • File bytes never pass through the Conductor server — all transfers go directly to the storage backend via presigned URLs

Test plan

  • FileClientTest — upload flow, multipart, backend mismatch fail-fast
  • ManagedFileHandlerTest — lazy download, cache hit, retry, concurrent access via ReentrantLock
  • FileHandlerConverterTest — DTO ↔ handler conversion
  • TaskRunnerFileUploadTest — auto-upload scan, skip already-uploaded, null fileClient guard
  • AnnotatedWorkerFileHandlerTestFileHandler input binding, return value passthrough
  • StubFileStorageBackend — in-memory backend for all SDK unit tests

Alternatives considered

Describe alternative implementation you have considered

@kowser-orkes kowser-orkes requested review from mp-orkes and v1r3n and removed request for mp-orkes April 16, 2026 16:19
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 16, 2026

Codecov Report

❌ Patch coverage is 58.74126% with 236 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...ductor/client/storage/AzureFileStorageBackend.java 1.78% 55 Missing ⚠️
...conductor/client/storage/S3FileStorageBackend.java 1.88% 52 Missing ⚠️
...onductor/client/storage/GcsFileStorageBackend.java 2.32% 42 Missing ⚠️
...ductoross/conductor/client/storage/HttpBodies.java 0.00% 26 Missing ⚠️
.../org/conductoross/conductor/client/FileClient.java 88.73% 12 Missing and 4 partials ⚠️
...ctoross/conductor/sdk/file/ManagedFileHandler.java 84.74% 5 Missing and 4 partials ⚠️
...ctoross/conductor/sdk/file/WorkflowFileClient.java 30.76% 9 Missing ⚠️
...ductor/client/storage/LocalFileStorageBackend.java 68.00% 8 Missing ⚠️
...ctoross/conductor/client/FileClientProperties.java 73.33% 4 Missing ⚠️
...netflix/conductor/client/automator/TaskRunner.java 83.33% 1 Missing and 2 partials ⚠️
... and 9 more
Files with missing lines Coverage Δ Complexity Δ
...lient/spring/ConductorClientAutoConfiguration.java 71.05% <100.00%> (ø) 10.00 <2.00> (?)
...ctoross/conductor/sdk/file/FileDownloadStatus.java 100.00% <100.00%> (ø) 1.00 <1.00> (?)
...ss/conductor/sdk/file/FileHandlerDeserializer.java 100.00% <100.00%> (ø) 4.00 <4.00> (?)
...ross/conductor/sdk/file/FileHandlerSerializer.java 100.00% <100.00%> (ø) 2.00 <2.00> (?)
...oross/conductor/sdk/file/FileStorageException.java 100.00% <100.00%> (ø) 2.00 <2.00> (?)
...uctoross/conductor/sdk/file/FileUploadOptions.java 100.00% <100.00%> (ø) 9.00 <9.00> (?)
.../spring/OrkesConductorClientAutoConfiguration.java 0.00% <0.00%> (ø) 0.00 <0.00> (?)
.../netflix/conductor/common/metadata/tasks/Task.java 88.81% <88.88%> (+<0.01%) 72.00 <4.00> (+4.00)
...n/java/io/orkes/conductor/client/OrkesClients.java 0.00% <0.00%> (ø) 0.00 <0.00> (ø)
...g/conductoross/conductor/sdk/file/FileHandler.java 90.90% <90.90%> (ø) 13.00 <13.00> (?)
... and 15 more

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@kowser-orkes kowser-orkes force-pushed the feature/file-storage-wip branch 11 times, most recently from 6b84835 to d9e5a2a Compare April 21, 2026 17:04
@kowser-orkes kowser-orkes force-pushed the feature/file-storage-wip branch 2 times, most recently from 8b668e1 to e9b37db Compare April 25, 2026 20:19
@kowser-orkes kowser-orkes marked this pull request as ready for review April 25, 2026 20:23

import com.netflix.conductor.client.http.ConductorClient;

public class OrkesFileClient extends FileClient {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename this to just FileClient please

@kowser-orkes kowser-orkes force-pushed the feature/file-storage-wip branch 2 times, most recently from 622beb1 to 121b369 Compare April 28, 2026 04:37
@kowser-orkes kowser-orkes changed the title feat(file-storage) Initial implementation of SDK support for file storage feat(file-storage) File storage Java SDK Apr 30, 2026
@kowser-orkes kowser-orkes changed the title feat(file-storage) File storage Java SDK feat(file-storage) File Storage Java SDK Apr 30, 2026
@kowser-orkes kowser-orkes force-pushed the feature/file-storage-wip branch from 121b369 to a05becc Compare May 4, 2026 16:56
@kowser-orkes kowser-orkes force-pushed the feature/file-storage-wip branch from a05becc to 1c5720f Compare May 5, 2026 01:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants