Skip to content

[Bug]: Seed file presigned URL expires on long-running resumed crawls #3576

Description

@tw4l

Browsertrix Version

v1.24.2-643de02

What did you expect to happen? What happened instead?

Crawls should be able to be paused and resumed as often as necessary, over any period of time.

Currently, however, the presigned URL passed into the crawl specification can expire for crawls that are paused and later resumed (or restart for other reasons after running for longer than the presigned URL expiration time).

To fix this, we should pass the seed file id to the crawl spec rather than the presigned URL so that the operator can generate a new presigned URL each time the crawler config is loaded.

Reproduction instructions

  1. Run a List of Pages crawl using a seed file
  2. Pause the crawl
  3. Resume the crawl over a week later and see that the crawl fails, with all data stored up to that point rejected, due to a fatal error in the crawl related to the seed file presigned URL being expired

Screenshots / Video

No response

Environment

No response

Additional details

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions