...

This process can be scaled horizontally. One can add as many processes as needed to handle the file uploads/downloads.

go-d-archive

This process can be scaled horizontally. One can add as many processes as needed to handle the jobs archiving.

Cleanup

Containers are cleanup by the software once job is over. It may be necessary however to clean images on nodes at regular interval if their number increase. docker-gc can be used on nodes for such task: https://github.com/spotify/docker-gc

...