Description
Rather than adding additional overhead to our already high-overhead replication strategy, we will switch to using rclone copy to periodically sync (in an additive-only way) replica storage locations from primary storage.
Replica deletion will still be controlled on a per-file basis and subject to replica deletion delays, as currently implemented.
This will make file replication simpler and more consistent. See comments below for more details.
Original description (no longer the approach we will take, but kept for context)
Crawl and profile file replication works through background jobs, which have the possibility to fail. Despite superadmin email notices, at a large scale of activity it can become difficult to be sure that everything that should be replicated in fact is.
To add an extra level of guarantee around file replication, we will add a new background cron job that will run on each run will look for any missing file replicas and then start new jobs to create them.
We'll want the cron schedule for this job to be customizable like the seed file cleanup cron job. In addition, we should add a configurable limit to the number of file replication jobs that can be triggered each time the job runs to prevent overwhelming clusters with a large number of jobs being started simultaneously.
Context
No response
Description
Rather than adding additional overhead to our already high-overhead replication strategy, we will switch to using rclone copy to periodically sync (in an additive-only way) replica storage locations from primary storage.
Replica deletion will still be controlled on a per-file basis and subject to replica deletion delays, as currently implemented.
This will make file replication simpler and more consistent. See comments below for more details.
Original description (no longer the approach we will take, but kept for context)
Crawl and profile file replication works through background jobs, which have the possibility to fail. Despite superadmin email notices, at a large scale of activity it can become difficult to be sure that everything that should be replicated in fact is.
To add an extra level of guarantee around file replication, we will add a new background cron job that will run on each run will look for any missing file replicas and then start new jobs to create them.
We'll want the cron schedule for this job to be customizable like the seed file cleanup cron job. In addition, we should add a configurable limit to the number of file replication jobs that can be triggered each time the job runs to prevent overwhelming clusters with a large number of jobs being started simultaneously.
Context
No response