#744 confirmed that the problem are 2 hosts running inserts with the same data (only id and date fields are different).
Also, looks like inserts from different hosts are called at the same second (+-1), which is strange, unless the processes are restarted simultaneously during deploy.
As a fix I propose storing a lock key in redis with TTL of 55 seconds:
- All processes are trying to
redis.set('dashboard-probes:synced', true, { NX: true, EX: 55 } every 60 seconds
- If result is
null => probes are synced => return
- If result is
OK => probes are not synced => proceed with syncing
Previous logic with workers and SHOULD_SYNC_ADOPTIONS can be removed as not required.
#744 confirmed that the problem are 2 hosts running inserts with the same data (only
idand date fields are different).Also, looks like inserts from different hosts are called at the same second (+-1), which is strange, unless the processes are restarted simultaneously during deploy.
As a fix I propose storing a lock key in redis with TTL of 55 seconds:
redis.set('dashboard-probes:synced', true, { NX: true, EX: 55 }every 60 secondsnull=> probes are synced => returnOK=> probes are not synced => proceed with syncingPrevious logic with workers and
SHOULD_SYNC_ADOPTIONScan be removed as not required.