扶凯 » Rsync速度测试对比

[Linux 存储备份恢复] Rsync速度测试对比

Feb 21st, 2009

转载本站文章请注明，转载自：扶凯[http://www.php-oa.com]

本文链接: http://www.php-oa.com/2009/02/21/rsync-test.html

因为一直对rsync的速度有疑问,所以昨天测试了一下.测试的环境为了1000M的网络..二天机器各接了一个存储.同步一个大的文件,所以大量的小问题不在考虑的范围.主要是速度测试.

对比的项目.

1.rsync使用ssh和不使用ssh时的速度分别(其实默认也是ssh).
2.使用NFS挂到本地,然后在本地rsync的速度和rsync对比
3.使用NFS挂到本地,然后cp到本地的速度来对比
4.加优化的参数对比

使用rsync的ssh的方法来传送文件,看看会不会慢很多
time rsync -avlR -e ssh Pearl.Harbor.Blu-Ray.Remux.MPEG2.1080P.DTS.LPCM.DD51.Fanxy\@Silu.mkv 172.16.xxx.xxx:/data/
sending incremental file list
Pearl.Harbor.Blu-Ray.Remux.MPEG2.1080P.DTS.LPCM.DD51.Fanxy@Silu.mkv

sent 4370744200 bytes received 31 bytes 19821969.30 bytes/sec
total size is 4370210591 speedup is 1.00

real    3m40.864s
user    1m36.867s
sys     0m37.546s

去掉ssh的选项来对比
time rsync -avlR Pearl.Harbor.Blu-Ray.Remux.MPEG2.1080P.DTS.LPCM.DD51.Fanxy\@Silu.mkv 172.16.xxx.xxx:/data
sending incremental file list
Pearl.Harbor.Blu-Ray.Remux.MPEG2.1080P.DTS.LPCM.DD51.Fanxy@Silu.mkv

sent 4370744200 bytes received 31 bytes 19643794.30 bytes/sec
total size is 4370210591 speedup is 1.00

real    3m41.538s
user    1m37.270s
sys     0m37.984s

给nfs来mount到本地,使用rsync同步,看看是否会快些
time rsync -avlR Pearl.Harbor.Blu-Ray.Remux.MPEG2.1080P.DTS.LPCM.DD51.Fanxy\@Silu.mkv /data

sending incremental file list
Pearl.Harbor.Blu-Ray.Remux.MPEG2.1080P.DTS.LPCM.DD51.Fanxy@Silu.mkv

sent 4370744200 bytes received 31 bytes 19732479.60 bytes/sec
total size is 4370210591 speedup is 1.00

real    3m40.754s
user    0m28.250s
sys     0m22.297s

使用cp命令来测试
time cp Pearl.Harbor.Blu-Ray.Remux.MPEG2.1080P.DTS.LPCM.DD51.Fanxy\@Silu.mkv /data/
real    3m41.571s
user    0m0.444s
sys     0m5.324s

加入size-only参数后对比
time rsync -avl –size-only Pearl.Harbor.Blu-Ray.Remux.MPEG2.1080P.DTS.LPCM.DD51.Fanxy\@Silu.mkv 172.16.xxx.xxx:/data
sending incremental file list
Pearl.Harbor.Blu-Ray.Remux.MPEG2.1080P.DTS.LPCM.DD51.Fanxy@Silu.mkv

sent 4370744200 bytes received 31 bytes 21799223.10 bytes/sec
total size is 4370210591 speedup is 1.00

real    3m20.962s
user    1m37.645s
sys     0m37.688s

加入blocking-io参数对比
time rsync -avl –size-only –blocking-io Pearl.Harbor.Blu-Ray.Remux.MPEG2.1080P.DTS.LPCM.DD51.Fanxy\@Silu.mkv 172.16.xxx.xxx:/data
sending incremental file list
Pearl.Harbor.Blu-Ray.Remux.MPEG2.1080P.DTS.LPCM.DD51.Fanxy@Silu.mkv

sent 4370744200 bytes received 31 bytes 2xxx2973.19 bytes/sec
total size is 4370210591 speedup is 1.00

real    3m15.617s
user    1m37.587s
sys     0m37.793s

由上面可以看出来,使用nfs来mount和远程直接rsync速度没有分别,单线程的rsync都很快,打不满网络.如果是文件对比少的话,rsync和使用cp的时间基本一样.另外加入优化参数能提高rsync的速度,但是我不太明白blocking-io 表示什么意思,有达人知道,请告诉….

另外给大家放上我写的一个简单的perl程序来控制rsync多线程同步


#!/usr/bin/perl

my $path = "/data";              #本地目录
my $ip="172.16.xxx.xxx";     #远程目录
my $maxchild=5;                 #同时并发的个数

open FILE,"ls $path|";
while()
{

        chomp;
        my $filename = $_;
        my $i = 1;
        while($i<=1){
                my $un = `ps -ef |grep rsync|grep -v grep |grep avl|wc -l`;
                $i =$i+1;
                if( $un < $maxchild){
                        system("rsync -avl --size-only $path/$_   $ip:$path &") ;
                }else{
                        sleep 5;
                        $i = 1;
                }
        }
}

Tags: cp, nfs, rsync

rsync速度优化在squid中修改ACL相关的代码

jackboy
Feb 26th, 2009 at 10:21

Reply | Quote | #1

脚本能贴出来吗？上面的脚本是乱的！
admin
Feb 26th, 2009 at 10:46

Reply | Quote | #2

不好意思，更新完了
1288893
Apr 21st, 2009 at 13:00

Reply | Quote | #3

blocking-io 不使用ssh ,而是使用rsh ，所以速度要快。
Lonely
Apr 21st, 2009 at 16:15

Reply | Quote | #4

我正在镜像Debian，使用命令如下
rsync -a -H -rltvHSB8192 –timeout 3600 –stats –exclude Archive-Update-in-Progress-debian –exclude project/trace/debian –exclude Archive-Update-Required-debian -P –exclude Packages* –exclude Sources* –exclude Release* –exclude ls-lR* –exclude *Debian* –exclude *oldstable* –exclude *etch* –exclude *proposed-updates* –exclude .~tmp~/ –exclude binary-alpha/ –exclude disks-alpha/ –exclude installer-alpha/ –exclude Contents-alpha.gz –exclude Contents-alpha.bz2 –exclude Contents-alpha.diff/ –exclude arch-alpha.files –exclude arch-alpha.list.gz –exclude *_alpha.deb –exclude *_alpha.udeb –exclude binary-amd64/ –exclude disks-amd64/ –exclude installer-amd64/ –exclude Contents-amd64.gz –exclude Contents-amd64.bz2 –exclude Contents-amd64.diff/ –exclude arch-amd64.files –exclude arch-amd64.list.gz –exclude *_amd64.deb –exclude *_amd64.udeb –exclude binary-arm/ –exclude disks-arm/ –exclude installer-arm/ –exclude Contents-arm.gz –exclude Contents-arm.bz2 –exclude Contents-arm.diff/ –exclude arch-arm.files –exclude arch-arm.list.gz –exclude *_arm.deb –exclude *_arm.udeb –exclude binary-armel/ –exclude disks-armel/ –exclude installer-armel/ –exclude Contents-armel.gz –exclude Contents-armel.bz2 –exclude Contents-armel.diff/ –exclude arch-armel.files –exclude arch-armel.list.gz –exclude *_armel.deb –exclude *_armel.udeb –exclude binary-hppa/ –exclude disks-hppa/ –exclude installer-hppa/ –exclude Contents-hppa.gz –exclude Contents-hppa.bz2 –exclude Contents-hppa.diff/ –exclude arch-hppa.files –exclude arch-hppa.list.gz –exclude *_hppa.deb –exclude *_hppa.udeb –exclude binary-hurd-i386/ –exclude disks-hurd-i386/ –exclude installer-hurd-i386/ –exclude Contents-hurd-i386.gz –exclude Contents-hurd-i386.bz2 –exclude Contents-hurd-i386.diff/ –exclude arch-hurd-i386.files –exclude arch-hurd-i386.list.gz –exclude *_hurd-i386.deb –exclude *_hurd-i386.udeb –exclude binary-ia64/ –exclude disks-ia64/ –exclude installer-ia64/ –exclude Contents-ia64.gz –exclude Contents-ia64.bz2 –exclude Contents-ia64.diff/ –exclude arch-ia64.files –exclude arch-ia64.list.gz –exclude *_ia64.deb –exclude *_ia64.udeb –exclude binary-m68k/ –exclude disks-m68k/ –exclude installer-m68k/ –exclude Contents-m68k.gz –exclude Contents-m68k.bz2 –exclude Contents-m68k.diff/ –exclude arch-m68k.files –exclude arch-m68k.list.gz –exclude *_m68k.deb –exclude *_m68k.udeb –exclude binary-mipsel/ –exclude disks-mipsel/ –exclude installer-mipsel/ –exclude Contents-mipsel.gz –exclude Contents-mipsel.bz2 –exclude Contents-mipsel.diff/ –exclude arch-mipsel.files –exclude arch-mipsel.list.gz –exclude *_mipsel.deb –exclude *_mipsel.udeb –exclude binary-mips/ –exclude disks-mips/ –exclude installer-mips/ –exclude Contents-mips.gz –exclude Contents-mips.bz2 –exclude Contents-mips.diff/ –exclude arch-mips.files –exclude arch-mips.list.gz –exclude *_mips.deb –exclude *_mips.udeb –exclude binary-powerpc/ –exclude disks-powerpc/ –exclude installer-powerpc/ –exclude Contents-powerpc.gz –exclude Contents-powerpc.bz2 –exclude Contents-powerpc.diff/ –exclude arch-powerpc.files –exclude arch-powerpc.list.gz –exclude *_powerpc.deb –exclude *_powerpc.udeb –exclude binary-s390/ –exclude disks-s390/ –exclude installer-s390/ –exclude Contents-s390.gz –exclude Contents-s390.bz2 –exclude Contents-s390.diff/ –exclude arch-s390.files –exclude arch-s390.list.gz –exclude *_s390.deb –exclude *_s390.udeb –exclude binary-sh/ –exclude disks-sh/ –exclude installer-sh/ –exclude Contents-sh.gz –exclude Contents-sh.bz2 –exclude Contents-sh.diff/ –exclude arch-sh.files –exclude arch-sh.list.gz –exclude *_sh.deb –exclude *_sh.udeb –exclude binary-sparc/ –exclude disks-sparc/ –exclude installer-sparc/ –exclude Contents-sparc.gz –exclude Contents-sparc.bz2 –exclude Contents-sparc.diff/ –exclude arch-sparc.files –exclude arch-sparc.list.gz –exclude *_sparc.deb –exclude *_sparc.udeb http://ftp.tw.debian.org::debian /mirror/debian
我想问一下，如果我将上面的命令使用 & 转入后台，能不能再开一个相同的命令以达到多线程的目的？
Lonely
Apr 21st, 2009 at 16:18

Reply | Quote | #5

我同时在两个SSH窗口中输入同样的命令，得到的结果是两个窗口同时下载一个文件

receiving incremental file list
./
dists/sid/main/installer-i386/20080227/images/cdrom/gtk/
dists/sid/main/installer-i386/20080227/images/cdrom/gtk/initrd.gz
3932160 35% 1.71kB/s 1:10:55

receiving incremental file list
dists/sid/main/installer-i386/20080227/images/cdrom/gtk/
dists/sid/main/installer-i386/20080227/images/cdrom/gtk/initrd.gz
3768320 33% 6.63kB/s 0:18:43 ^C
1288893
Apr 21st, 2009 at 17:28

Reply | Quote | #6

抱歉，上面的解释存在想当然的失误，我详细了解了一下blocking-io 选项，
blocking-io ， Linux® 中最常用的输入/输出（I/O）模型是同步 I/O。在这个模型中，用户空间的应用程序执行一个系统调用，这会导致应用程序阻塞。这意味着应用程序会一直阻塞，直到系统调用完成为止（数据传输完成或发生错误）。调用应用程序处于一种不再消费 CPU 而只是简单等待响应的状态，因此从处理的角度来看，这是非常有效的。
admin
Apr 21st, 2009 at 22:46

Reply | Quote | #7

呵呵，非常多谢你研究这么深入，真不错
admin
Apr 21st, 2009 at 22:47

Reply | Quote | #8

如果处理相同的目录，不会起到并发的作用反到会更加慢。处理不同的目录才会。当然，是在你的硬盘速度快的情况下。
earp
Jun 8th, 2009 at 23:56

Reply | Quote | #9

大大,您的脚本我用不了,参照着用PHP改了一个将就着用呵.
http://blog.chinaunix.net/u2/70071/showart_1959123.html
admin
Jun 9th, 2009 at 13:00

Reply | Quote | #10

good
gyl
Jul 20th, 2009 at 13:54

Reply | Quote | #11

请教像新浪、网易等大型门户是使用什么软件做各区域服务器的静态网页文件同步的？
admin
Jul 21st, 2009 at 08:54

Reply | Quote | #12

不做同步,只做cache
wwwlh
Sep 24th, 2009 at 15:53

Reply | Quote | #13

加入z压缩参数会更快
admin
Sep 24th, 2009 at 16:44

Reply | Quote | #14

会更新慢.因为我是内网,所以速度主要在传送加密解密上,加了z只会更加慢,要是网络不好就会快很多

扶凯

[Linux 存储备份恢复] Rsync速度测试对比

Tags

Categories

Recent Posts

Recent Comments