Related links:
SUMMARY
Use simple encryption protocol. It contributes the most to speed of data transfer.
Use simple encryption protocol. It contributes the most to speed of data transfer.
Choice
|
Command
|
Pros
|
Cons
|
#1
|
scp
|
| |
#2
|
rsync
|
flexible and convenient for directory synchronization
|
possible, but not easy to configure to NOT to use encryption
|
#3
|
sftp
|
can be speed up choosing simple encryption
|
can't recursively copy directories
|
Notice: when running any tool it consumes about 5-10% of CPU at both sender and receiver machines, apparently, doing encryption/decryption.
TEST DETAILS
These are the actual commands and generated output by these tools.rsync
default "-a" --archive mode
|
"-z" compress
|
rsync -a --progress DIR2REPLICATE root@10.10.10.2:/tmp
411533312 0% 45.38MB/s 0:26:04
1407877120 1% 44.41MB/s 0:26:16
1716748288 2% 42.09MB/s 0:27:36
2002550784 2% 46.47MB/s 0:24:541
2382397440 3% 45.31MB/s 0:25:24
2762407936 3% 45.34MB/s 0:25:15
|
rsync -az --progress DIR2REPLICATE root@10.10.10.2:/tmp
991383915 100% 13.67MB/s 0:01:09
990955265 100% 14.02MB/s 0:01:07
202624740 100% 15.42MB/s 0:00:12
202771784 100% 15.87MB/s 0:00:12
91676674 100% 12.86MB/s 0:00:06
91628045 100% 11.76MB/s 0:00:07
1082301721 100% 16.86MB/s 0:01:01
1081744094 100% 17.14MB/s 0:01:00
444531263 100% 13.06MB/s 0:00:32
444311917 100% 12.97MB/s 0:00:32
25956199 100% 11.99MB/s 0:00:02
25387962 100% 16.94MB/s 0:00:01
94059363 100% 15.51MB/s 0:00:05
94189273 100% 14.61MB/s 0:00:06
369550738 100% 16.31MB/s 0:00:21
370924791 100% 15.96MB/s 0:00:22
143659839 100% 14.75MB/s 0:00:09
141681760 100% 14.58MB/s 0:00:09
74662680 100% 14.45MB/s 0:00:04
73882769 100% 12.73MB/s 0:00:05
1809543 100% 13.59MB/s 0:00:00
|
###
### "-c arcfour" cipher is defined in RFC 4253; it is plain RC4 with a 128-bit key
###
rsync -a -P -e "ssh -T -c arcfour -o Compression=no -x" DIR2REPLICATE root@10.10.10.2:/tmp
### "-c arcfour" cipher is defined in RFC 4253; it is plain RC4 with a 128-bit key
###
rsync -a -P -e "ssh -T -c arcfour -o Compression=no -x" DIR2REPLICATE root@10.10.10.2:/tmp
1081744094 100% 65.35MB/s 0:00:15
444531263 100% 56.34MB/s 0:00:07
444311917 100% 61.61MB/s 0:00:06
369550738 100% 53.94MB/s 0:00:06
370924791 100% 60.03MB/s 0:00:05
23319017231 100% 65.89MB/s 0:05:37
23308793162 100% 64.88MB/s 0:05:42
11951287020 100% 65.68MB/s 0:02:53
3453648896 28% 68.11MB/s 0:02:0
444531263 100% 56.34MB/s 0:00:07
444311917 100% 61.61MB/s 0:00:06
369550738 100% 53.94MB/s 0:00:06
370924791 100% 60.03MB/s 0:00:05
23319017231 100% 65.89MB/s 0:05:37
23308793162 100% 64.88MB/s 0:05:42
11951287020 100% 65.68MB/s 0:02:53
3453648896 28% 68.11MB/s 0:02:0
scp
default "-r" recursive
|
"-C" compress "-r" recursive
|
scp -r DIR2REPLICATE root@10.10.10.2:/tmp
100% 193MB 64.4MB/s 00:03
100% 424MB 60.6MB/s 00:07
100% 945MB 63.0MB/s 00:15
100% 945MB 59.1MB/s 00:16
100% 1032MB 64.5MB/s 00:16
100% 1032MB 60.7MB/s 00:17
100% 749MB 53.5MB/s 00:14
100% 1253MB 62.6MB/s 00:20
18% 4615MB 62.6MB/s 05:18
|
scp -Cr DIR2REPLICATE root@10.10.10.2:/tmp
100% 193MB 16.1MB/s 00:12
100% 424MB 14.6MB/s 00:29
100% 945MB 15.0MB/s 01:03
100% 945MB 14.8MB/s 01:04
100% 424MB 14.1MB/s 00:30
100% 352MB 17.6MB/s 00:20
100% 193MB 17.6MB/s 00:11
100% 135MB 16.9MB/s 00:08
100% 1032MB 17.8MB/s 00:58
100% 1032MB 17.8MB/s 00:58
100% 354MB 17.7MB/s 00:20
100% 749MB 18.3MB/s 00:41
100% 1253MB 18.4MB/s 01:08
6% 1518MB 17.7MB/s 21:43
|
"-c arcfour" cipher is defined in RFC 4253; it is plain RC4 with a 128-bit key.
| |
scp -c arcfour -r DIR2REPLICATE root@10.10.10.2:/tmp
100% 424MB 141.3MB/s 00:03
100% 945MB 135.0MB/s 00:07
100% 945MB 189.1MB/s 00:05
100% 424MB 141.2MB/s 00:03
100% 352MB 117.5MB/s 00:03
100% 1032MB 147.4MB/s 00:07
100% 1032MB 147.5MB/s 00:07
100% 749MB 149.8MB/s 00:05
100% 1253MB 156.6MB/s 00:08
100% 24GB 142.0MB/s 02:53
100% 595MB 119.1MB/s 00:05
100% 82GB 138.3MB/s 10:09
51% 9099MB 141.3MB/s 01:01
|
sftp
default behavior
|
"-R" to increase request queue length (default is 64)
"-B" to increase read/write request size (default is 32 KB)
|
sftp root@10.10.10.2:/tmp
10% 2363MB 57.8MB/s 05:43 ETA
15% 3349MB 58.1MB/s 05:25 ETA
32% 7311MB 59.3MB/s 04:11 ETA
35% 7803MB 60.6MB/s 03:58 ETA
43% 9594MB 62.1MB/s 03:23 ETA
69% 15GB 58.6MB/s 01:55 ETA
77% 17GB 62.1MB/s 01:20 ETA
|
sftp -R 128 -B 65536 root@10.10.10.2:/tmp
2% 551MB 58.9MB/s 06:08 ETA
8% 1806MB 62.3MB/s 05:28 ETA
41% 9170MB 60.6MB/s 03:35 ETA
56% 12GB 62.6MB/s 02:32 ETA
100% 22GB 62.5MB/s 05:56
|
"-c arcfour" cipher is defined in RFC 4253; it is plain RC4 with a 128-bit key.
| |
sftp -oCiphers=arcfour root@10.10.10.2:/tmp
3% 711MB 142.5MB/s 02:31 ETA
18% 4115MB 146.0MB/s 02:04 ETA
23% 5156MB 148.1MB/s 01:55 ETA
28% 6379MB 144.6MB/s 01:49 ETA
34% 7672MB 144.0MB/s 01:41 ETA
37% 8389MB 143.7MB/s 01:36 ETA
62% 14GB 143.8MB/s 00:58 ETA
85% 19GB 142.4MB/s 00:22 ETA
92% 20GB 142.3MB/s 00:12 ETA
100% 22GB 144.4MB/s 02:34
|
TEST ENVIRONMENT
The test was performed between two servers interconnected by private 10 Gbit link with 9000 MTU "jumbo frame". The files copies were large (100's GB) binary files.
iperf network bandwidth test between 10.10.10.2 and 10.10.10.1
|
Network interface configuration (10 Gbit, MTU 9000)
|
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.10.10.2 port 5001 connected with 10.10.10.1 port 57279
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 11.5 GBytes 9.89 Gbits/sec
[ 5] 0.0-30.0 sec 34.6 GBytes 9.89 Gbits/sec
[ 4] 0.0- 0.9 sec 1000 MBytes 9.80 Gbits/sec
[ 5] 0.0- 8.8 sec 9.77 GBytes 9.53 Gbits/sec
[ 4] 0.0- 8.7 sec 10.0 GBytes 9.89 Gbits/sec
[ 5] 0.0-86.8 sec 100 GBytes 9.89 Gbits/sec
|
[root@10.10.10.2]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE="bond0"
BOOTPROTO=none
IPADDR=10.10.10.2
NETMASK=255.255.255.192
ONBOOT="yes"
USERCTL=no
TXQUEUELEN=100000
MTU=8912
BONDING_OPTS="mode=1 miimon=200 primary=eth0"
|
ALTERNATIVE METHODS TO MOVE DATA FAST
Copy directories via netcat: tar | nc. Renders speed ~251 Mb/s ( = ~1 TB/hr).### On receiver ###
nc -v -l 5555 | tar -xvf -
### On Sender: test2del - large directory to move ###
time tar -cvf - test2del | nc -v 10.100.100.2 5555
### Output calculated ###
11GB in 27.465 s = 293 MB/s
42GB in 2m51.513s = 249 Mb/s (~1 TB/hr)
42GB in 2m50.630s = 251 Mb/s (~1 TB/hr)
