I think that it would be better if the data would be processed while uploading.
Currently this is ok for pictures(uploading first and processing) , but this could be made something like a proxy.
- The uploaded file would be split to chunks/shards while the uploading
- Certain endpoints would be configured. (like local, s3 , ftp or wahtever )
- The shards would be placed onto these endpoints.
The end result would be a solid uploader with different storage options.
In this case probably it would be better for the metadata to be stored on the disk, instead of database (maybe both for integrity check.)
Later this can be used for backup purposes.
Ex :
- kvm backup could be something like : snapshot, copy base disk , snapshot delete , then repeat. Most of the disk would be referenced and the changes would be only stored. This would shorten restore time for example, since there would be no actual incremental backup ..
I think that it would be better if the data would be processed while uploading.
Currently this is ok for pictures(uploading first and processing) , but this could be made something like a proxy.
The end result would be a solid uploader with different storage options.
In this case probably it would be better for the metadata to be stored on the disk, instead of database (maybe both for integrity check.)
Later this can be used for backup purposes.
Ex :