test(bench): enable TUS upload performance testing for files up to 1TB#477
Open
test(bench): enable TUS upload performance testing for files up to 1TB#477
Conversation
Benchmarks now support testing upload and pinning performance for files ranging from 1GB to 1TB. Tests measure upload duration, throughput, and pin status acquisition with configurable 30-minute timeouts. CPU and memory profiling available via TUS_BENCH_ENABLE_PROFILING.
This comment has been minimized.
This comment has been minimized.
Regular tests now exclude bench_tests directory. Benchmarks run in separate job with CPU/memory profiling enabled and upload results as artifacts.
![kody-ai[bot]](https://avatars.githubusercontent.com/in/413034?s=60&v=4){kind=small}
…estopts package Moved GetStandardTestOptions from internal/testing/util (which imports internal/plugin) to new internal/testing/testopts package to break circular dependency with test packages.
This comment has been minimized.
This comment has been minimized.
Remove go mod download from benchmark job since Go setup already handles module caching.
This comment has been minimized.
This comment has been minimized.
…d tests Update SetupTUSUpload to return testUser.ID along with protocol and requestID. Update all callers to use the dynamic userID instead of hardcoded value 1 for workflow options and pin status filtering.
This comment has been minimized.
This comment has been minimized.
Set TUS_BENCH_PIN_TIMEOUT=0 to disable timeout limit and allow unlimited wait time for IPFS pin status, which is necessary for multi-GB/TB uploads that may exceed default 30-minute timeout.
This comment has been minimized.
This comment has been minimized.
Check workflow status before entering pin wait loop. If workflow fails, log the status and return early instead of looping forever waiting for pins that will never be created.
Contributor
Kody Review CompleteGreat news! 🎉 Keep up the excellent work! 🚀 Kody Guide: Usage and ConfigurationInteracting with Kody
Current Kody ConfigurationReview OptionsThe following review options are enabled or disabled:
|
Contributor
|
Blacksmith runners detected OOM events on the following jobs:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Benchmarks now support testing upload and pinning performance for files ranging from 1GB to 1TB. Tests measure upload duration, throughput, and pin status acquisition with configurable 30-minute timeouts. CPU and memory profiling available via TUS_BENCH_ENABLE_PROFILING.
developThis pull request introduces comprehensive TUS upload performance benchmarking capabilities and refactors test utilities for better code reuse.
Key Changes:
TUS Upload Performance Benchmarking: Added benchmark tests to measure TUS upload performance for large files ranging from 1GB to 1TB. The benchmarks track upload duration, throughput (MB/s), and pin wait times to identify performance bottlenecks in the upload pipeline.
Optional Performance Profiling: Integrated CPU and memory profiling capabilities (controlled via environment variables) to enable detailed performance analysis during benchmark execution.
Test Utility Refactoring: Extracted TUS test helpers and standard test configuration options into shared packages (
internal/testing/tusandinternal/testing/util) to eliminate code duplication and enable reuse across test suites.Configurable Test Parameters: Added environment variable support for customizing pin timeout durations and enabling profiling output directories.
The changes enable systematic performance testing of the TUS upload workflow for enterprise-scale file sizes while improving the maintainability of the test codebase through better organization of shared testing utilities.