Fast, reliable data transfers from/to China

We are a small company and we will need to transfer rather large amounts of data (10GB+ each time) between Europe and China in the near future.

As many may have experienced, Internet connections to or from China can be rather unreliable and slow at times without any apparent reason. For example, while sending data to China via FTP generally works well, it can be painfully slow in the other direction.

Currently, we are investigating new ways to have high transfer rates in both directions. So far we have tried:

1. FTP (see above)
2. FTP over VPN services (generally slower than direct connections)
3. F2F (like Retroshare or Freenet - slow!!)
4. Aspera (fast but expensive!)
5. BitTorrent (unreachable end nodes, b/c of firewalls which we must not configure)

We would like to try:

1. Cloud storage (e.g. Amazon S3, Google Storage) - are those services always and reliably reachable from inside China?
2. Point-to-Point VPN (currently not possible, b/c of the network, see above)

I'd be especially grateful to hear from people who have already dealt with this kind of problem before.

Solution 1:

I would suggest using rsync to do these sort of transfers. It can use ssh to encrypt the traffic (so you don't need to use a separate VPN), compresses the stream (which can be quite effective depending on your type of data) and can restart transfers if they fail (it uses a rolling checksum to see whereabouts a transfer failed and where to re-start from).

I use the following which incorporates the above (as the rsync client on my/most machines use ssh by default):

rsync -avz <local file> username@hostname:/path/where/to/place/file/<remote file>

You can use --progress to see on the terminal how well the transfer is progressing. Usually it ssh's in and uses the local machine rsync client but there is also a rsync daemon. I have never used it but worth investigating if you want an actual daemon similar to ftpd.

As it uses SSH you can also alter the ports it uses if firewalls lock them down. i.e. you don't have to have ssh listening on port 22 (in fact I don't to avoid random people on the internet scanning my SSH).

It is usually used on Unix systems (which includes Macs) but you can install it with Cygwin as well if you need to use it on Windows.

If you want an actual service, I use rsync.net who, rather obviously, provide storage to which you can rsync data easily to. They have a European data centre in Sweden which would fit your purposes and don't limit in/out traffic speeds (though check the small print).

Solution 2:

I suggest you test a physical shipment of a disk or thumb drive back-forth to China to understand how long & how much it would take to send the data via a shipping company. The bandwidth is HUGE from a data perspective. You can encrypt and send multiple shipments to provide redundancy and protection and you have no infrastructure to maintain.

Solution 3:

Some of you have requested to know what we settled for, finally.

Though, we have tried and tested almost all of the suggested solutions above, using Aspera's proprietary protocol and software seem to do the best job for our purpose.

Maybe somebody out there else has still knows of something better and would like to share that?