All HowTo's

Rsync and the “Argument list too long” problem

If you try to rsync a subset of many files from a single directory, you might get the error “Argument list too long”. Actually, you can get this error with many bash commands. This article explains how to work around it.

rsync -avz /images/* cdn.example.com:/images/
-bash: /usr/bin/rsync: Argument list too long

The solution is to either remove the “*” from the source as in the following example:

rsync -avz /images/ cdn.example.com:/images/

Or do things a little smarter. Here’s the two step process. Here’s how to do it.

Create a list of the files (in this case we want pictures) you want to copy:

find /images/ -name *.png > /tmp/my_image_list.txt

And now we feed that into rsync:

rsync -avz --files-from=/tmp/my_image_list.txt / cdn.example.com:/images/

The last command needs an explanation. The “rsync -avz” says to use archive mode, be verbose and compress the session. The second part “–files-from=/tmp/my_image_list.txt” is how we feed the source list into rsync. Then we have the “/” by its self. This is the starting point (or reference point) and can be changed if you wanted to change where to get the files from. Finally we have the target of “cdn.example.com:/images/” on the remote machine.

6 comments

  1. The thing about –files-from is that it automatically implies -R (–relative), unlike giving files on the command line. So unless that’s really what you want (and it probably isn’t, unless you change the find command a bit), you probably want to put –no-R in the rsync command.

  2. FYI this method does not append a newline to the last line, so the last file will be ignored.
    May want to add the printf switch with “%p\n”

Leave a Reply to John Thacker Cancel reply

Your email address will not be published. Required fields are marked *