fix(relay): harden large download streaming#1382
Conversation
Inside the MITM TLS session, detect Connection: Upgrade + Upgrade: websocket before hitting the Apps Script relay. Establish a direct TLS connection to the real server (via upstream_socks5 if configured), relay the upgrade handshake, then splice both directions with copy_bidirectional. Apps Script cannot hold persistent connections so the bypass is the only viable path for wss://. Split request_timeout_secs (header/connect, 30s) from a new stream_timeout_secs (per-chunk body idle, default 300s) so large range downloads through Apps Script are not killed mid-transfer by the batch_timeout firing during the body drain phase.
Handle idempotent exit-node timeouts by falling back to direct Apps Script, route Range: bytes=N- requests through parallel resume streaming, and send TLS close_notify on shutdown so wget/curl can resume from the correct offset.
Preserve the new body-stream idle timeout across every config-facing path that can rewrite configuration. Desktop Save now round-trips stream_timeout_secs alongside request_timeout_secs, preventing hand-edited large-download timeout settings from being silently dropped. Document the split timeout model in the TOML examples and user guides: request_timeout_secs bounds relay connection and response-header arrival, while stream_timeout_secs bounds idle time between body chunks after headers have arrived. This keeps dead destinations bounded without aborting slow range downloads mid-stream. Update the guide's implementation matrix to reflect the active range-aware download path and its remaining quota cost. Large range-capable responses can stream in chunks and resume cleanly with Range: bytes=N-, but each chunk remains an Apps Script UrlFetchApp call. Add regression coverage for TOML parsing and defaults, JSON migration round-trip preservation, Range header start parsing, and validation of non-zero-offset resume probes.
|
im sorry did you just basically pull my commits and added a config update when i already added an automatic system for the configs to update based on what values are being used in #1317 ? even worst case scenario if it cant update the config it just uses the default and there is no need to change that in 90% of cases. there is no need to update the docs just to tell a normal user that the download now works normally, that shouldve been there in the first place and it was a bug not a feature to advertise, the docs are already packed with things that are not needed we dont need more clutter, we are not trying to confuse or scare away a new user its meant to be a guide not a world map |
|
just to add some context here: my concern isnt about blocking improvements, its about avoiding duplicate work. The behaviour this PR is touching is already handled automatically in my earlier PR (#1317), including sane defaults when config updates aren’t possible, so introducing a second, overlapping approach and extra docs for it could make things harder to maintain and more confusing for users in the long run. I’d prefer if we could either build on top of the existing implementation or consolidate the changes rather than having parallel solutions. and specifically in this case,the default and the UI update isnt needed, a lot of variables are missing from there, i am working on an overhauled version of the UI since it really needs one and a lot of things need to be changed around |
That's fair. I’ll close this PR and stop treating it as a separate source of truth. The download-resilience behavior should continue through #1346/#1317, and any useful pieces from my branch can be proposed later only as narrow follow-ups that do not overlap with your config migration or UI work. I also agree on the UI/docs point. I’ll avoid adding user-guide text for this bug fix and won’t touch the desktop UI for Thanks for clarifying the #1317 behavior. I should have checked that path more carefully before opening this. Sorry for the frustration. |
no frustration here mate dont get me wrong, the TOML move was quite a big PR i didnt mention every single detail i changed and made in the body, just wanted to say it here just so you know it already exists |
2 participants
Builds on CaptainMirage's relay download-resilience work with a clean branch based on current main.
This PR preserves the two reliability commits from #1346 and adds the missing config-facing completion layer:
stream_timeout_secsthrough desktop UI save round-trips;request_timeout_secsandstream_timeout_secsin TOML examples;The branch intentionally excludes #1346's unrelated compact logging/UI timestamp changes and release artifact churn so the relay behavior can be reviewed independently.
Verification:
git diff --checkpasses.winnow v0.7.15dependency when using--offline.