s3cmd timeout problems moving large files on S3 (> 250MB)

19 Mar

Recently we had a problem with s3cmd giving errors while moving or copying (large) files across buckets in S3. This had worked fine perfectly before, but our file sizes were increasing.

This was the sort of error we were getting (files roughly 200 MB, 250 MB, or 300 MB):

WARNING: Retrying failed request: /abc/def (timed out)
WARNING: Waiting 3 sec…

Its understandable that s3cmd fails sometimes to upload large files – but moving/copying files between S3 buckets??? Something wrong here.

I then went into Amazon’s S3 web console and cut and paste the files – and that worked fine.

Hmmm, something clearly amiss. Something to do with s3cmd’s timeout settings maybe?

s3cmd –help doesn’t reveal much; there are no timeout settings in there.

So I edit .s3cfg, and find a esoteric setting called “socket_timeout”.

I change it from the default “10” to “100”, which I assume changes the S3 operation timeout to 100 seconds from 10 seconds.

Voila! s3cmd mv and cp commands work perfectly on large files.

Advertisements

5 Responses to “s3cmd timeout problems moving large files on S3 (> 250MB)”

  1. thenickdude April 21, 2014 at 2:23 am #

    Thanks, this was driving me nuts!

  2. Len Gabrielson May 27, 2014 at 9:30 pm #

    I wonder if using a multipart upload would work ok?

  3. mdomsch February 13, 2015 at 1:42 pm #

    The default timeout has been 300 (5 minutes) since April 2011. Consider upgrading to a newer release of s3cmd for this and many other bug fixes.

  4. Kipp Jones June 3, 2015 at 12:55 pm #

    Awesome! Thanks for the note, helped tremendously!

  5. vk May 22, 2017 at 3:52 am #

    Hi,

    I am still facing issue with s3 upload time. i wan’t upload file, having size of >2TB.
    I have set the socket_timeout to 86400.

    any help appreciated !

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: