Recently we had a problem with s3cmd giving errors while moving or copying (large) files across buckets in S3. This had worked fine perfectly before, but our file sizes were increasing.
This was the sort of error we were getting (files roughly 200 MB, 250 MB, or 300 MB):
WARNING: Retrying failed request: /abc/def (timed out)
WARNING: Waiting 3 sec…
Its understandable that s3cmd fails sometimes to upload large files – but moving/copying files between S3 buckets??? Something wrong here.
I then went into Amazon’s S3 web console and cut and paste the files – and that worked fine.
Hmmm, something clearly amiss. Something to do with s3cmd’s timeout settings maybe?
s3cmd –help doesn’t reveal much; there are no timeout settings in there.
So I edit .s3cfg, and find a esoteric setting called “socket_timeout”.
I change it from the default “10” to “100”, which I assume changes the S3 operation timeout to 100 seconds from 10 seconds.
Voila! s3cmd mv and cp commands work perfectly on large files.