Rsync hangs: expand file_list pointer array to N bytes, did move

Considering that the rsync you're using is an open-source software, it's quite easy to get accesso to related source code.

After downloading the main .tar.gz and applied the Ubuntu patch (rsync_3.1.0-2ubuntu0.4.diff.gz), you end up with exactly the code underlying the rsync you're using. Something like this:

$ mkdir rsync
$ cd rsync/
$ wget http://archive.ubuntu.com/ubuntu/pool/main/r/rsync/rsync_3.1.0.orig.tar.gz
$ wget http://archive.ubuntu.com/ubuntu/pool/main/r/rsync/rsync_3.1.0-2ubuntu0.4.diff.gz
$ gzip -d rsync_3.1.0-2ubuntu0.4.diff.gz
$ tar zxvf rsync_3.1.0.orig.tar.gz 
$ cd rsync-3.1.0/
$ patch -p1 < ../rsync_3.1.0-2ubuntu0.4.diff

Now a simple grep can quickly tell us the context of your error message:

$ grep -r 'expand file_list pointer array to' 
flist.c:        rprintf(FCLIENT, "[%s] expand file_list pointer array to %s bytes, did%s move\n",

So you're lucky, as your error message is used in a single fragment of a single file. Nameli: flist.c.

Let's give a look:

It's relatively easy to guess that the routine containing the error message (lines 325, 326, 327, 328) is named flist_expand and sounds like something needed to ensure that the whole file list (to rsync) can be held in a properly sized in-memory structure (aka: the more files you need to rsync, the more memory is required to handle rsync-computations, and as such a list is not known "in advance", it need to be dinamically computed, by allocating proper chunks of memory to a "list" [more or less]).

So, I would bet that your problem rely NOT on the size of data you're rsync-ing, but on the number of files. I'd try splitting your rsync in multiple sub-rsync, by focusing on internal subfolders.

Actually, it would be nice to better investigate the:

1. line 328: (new_ptr == flist->files) ? " not" : "");
2. line 334: out_of_memory("flist_expand");

but this goes much beyond my initial goal :-)

Anyway, I would bet that checking your logs you would find some "out of memory" message.... :-)

HTH!