RHEL5 hadoop2.4.1 rsync配置
时间:2014-12-30 23:12 来源:linux.it.net.cn 作者:IT
首先下载对应linux的rsync软件:http://rsync.samba.org/download.html
本例中,下载rsync-3.1.1-1.el5.rfx.i386.rpm 22-Jun-2014 21:58 420K RHEL5 and CentOS-5 x86 32bit
服务端配置
[root@master software]#pwd
/root/software
[root@master software]#ll
… …
-rwxr--r--1 root root 429855 Jul 18 17:52 rsync-3.1.1-1.el5.rfx.i386.rpm
[root@master software]# rpm -ivh rsync-3.1.1-1.el5.rfx.i386.rpm
warning: rsync-3.1.1-1.el5.rfx.i386.rpm:Header V3 DSA signature: NOKEY, key ID 6b8d79e6
Preparing...
########################################### [100%]
file /etc/xinetd.d/rsync from install of rsync-3.1.1-1.el5.rfx.i386 conflicts with file from package rsync-2.6.8-3.1.i386
file /usr/bin/rsync from install of rsync-3.1.1-1.el5.rfx.i386 conflicts with file from package rsync-2.6.8-3.1.i386
file /usr/share/man/man1/rsync.1.gz from install of rsync-3.1.1-1.el5.rfx.i386 conflicts with file from package rsync-2.6.8-3.1.i386
file /usr/share/man/man5/rsyncd.conf.5.gz from install of rsync-3.1.1-1.el5.rfx.i386 conflicts with file from package rsync-2.6.8-3.1.i386
从日志信息可以看出和本机已经安装的rsync-2.6.8-3.1.i386冲突了。需要先卸载该包。
[root@master software]# rpm -e rsync-2.6.8-3.1
warning: /etc/xinetd.d/rsync saved as /etc/xinetd.d/rsync.rpmsave
[root@master software]# rpm -qa|grep rsync
[root@master software]# rpm -ivh rsync-3.1.1-1.el5.rfx.i386.rpm
warning: rsync-3.1.1-1.el5.rfx.i386.rpm:Header V3 DSA signature: NOKEY, key ID 6b8d79e6
Preparing...
########################################### [100%]
1:rsync
########################################### [100%]
[root@master software]#rpm -qa|grep rsync
rsync-3.1.1-1.el5.rfx
[root@master software]#vi /etc/rsyncd.conf
#global options
strict modes=yes
port =873
log file=/var/log/rsyncd.log
pid file=/usr/local/rsync/rsyncd.pid
#mode options
[hadoop]
max connections=10
uid=root
gid =root
path = /home/hadoop/hadoop-2.4.1
ignore errors
read only = no
write only = no
hosts allow = *
#hosts deny =
auth users = root
screts file = /home/hadoop/rsync.pass
#(注意:这里的secrets单词写错了,导致后面权限验证通不过)
[root@master software]#vi /home/hadoop/rsync.pass
root:beijing
[root@master software]#chmod 777 /home/hadoop/rsync.pass
[root@master ~]# vi /etc/xinetd.d/rsync
# default: off
# description: The rsync server is a good addition to an ftp server, as it \
# allows crc checksumming etc.
service rsync
{
#disable = yes
disable = no(把这个参数改成no)
socket_type = stream
wait = no
user = root
server = /usr/bin/rsync
server_args = --daemon
log_on_failure += USERID
}
[root@master ~]#chkconfig --list|grep sync
rsync: on
[root@master software]#rsync --daemon
failed to create pid file /usr/local/rsync/rsyncd.pid:No such file or directory
[root@master software]#mkdir /usr/local/rsync/
[root@master software]#rsync –daemon
[root@master software]#netstat -an |grep 873
tcp 0
0 0.0.0.0:873
0.0.0.0:* LISTEN
tcp 0
0 :::873
:::* LISTEN
[root@master software]#ps -ef|grep rsync
root 29351 1 0 17:36 ? 00:00:00 rsync --daemon
root 31429 2714 0 17:41 pts/1 00:00:00 grep rsync
在客户端slave-5测试同步
[root@slave-5 ~]# rsync -vzrtopg --progress --delete root@master::hadoop /home/hadoop/testrsync
Password:
@ERROR: auth failed on module hadoop
rsync error: error starting client-server protocol (code 5) at main.c(1296) [receiver=2.6.8]
可见由于用户验证未通过,该操作失败了,下面用了几种方式进行调整并测试
方法一(not works):
虽然该方法对本例未奏效,但是建议chmod 600 /home/hadoop/rsync.pass
[root@master software]# ll /home/hadoop/rsync.pass
-rwxrwxrwx 1 root root 13 Jul 21 17:23 /home/hadoop/rsync.pass
[root@master software]# chmod 600 /home/hadoop/rsync.pass
[root@master software]# ll /home/hadoop/rsync.pass
-rw-------1 root root 13 Jul 21 17:23 /home/hadoop/rsync.pass
[root@slave-5 ~]# rsync -vzrtopg --progress --delete root@master::hadoop /home/hadoop/testrsync
Password:
@ERROR: auth failed on module hadoop
rsync error: error starting client-server protocol (code 5) at main.c(1296) [receiver=2.6.8]
方法二(not works):
在客户端更新跟服务器相同版本的rsync(后经测试,低版本的客户端也是能够正常从服务端同步文件的)
[root@slave-5 ~]# rpm -e rsync-2.6.8-3.1
[root@slave-5 software]# rpm -ivh rsync-3.1.1-1.el5.rfx.i386.rpm
warning:rsync-3.1.1-1.el5.rfx.i386.rpm: Header V3 DSA signature: NOKEY, key ID 6b8d79e6
Preparing... ###########################################[100%]
1:rsync ########################################### [100%]
[root@slave-5 software]# rsync --daemon
Failed to parse config file: /etc/rsyncd.conf
[root@slave-5 software]# touch /etc/rsyncd.conf
[root@slave-5 software]# rsync –daemon
[root@slave-5 software]# rsync -vzrtopg --progress --delete root@master::hadoop /home/hadoop/testrsync
Password:
@ERROR: auth failed on module hadoop
rsync error: error starting client-server protocol (code 5) at main.c(1635) [Receiver=3.1.1]
方法三(not works):
[root@slave-5 ~]# mkdir /home/hadoop/testrsync
[root@slave-5 ~]# rsync -vzrtopg --progress --delete root@master::hadoop /home/hadoop/testrsync
Password:
@ERROR: auth failed on module hadoop
rsync error: error starting client-server protocol (code 5) at main.c(1635) [Receiver=3.1.1]
方法四(not works):重启下服务端试试
[root@master software]# ps -ef|grep rsync
root 29351 1 0 17:36 ? 00:00:00 rsync --daemon
root 31614 2714 0 18:17 pts/1 00:00:00 grep rsync
[root@master software]# kill -9 29351
[root@master software]# ps -ef|grep rsync
root 31616 2714 0 18:18 pts/1 00:00:00 grep rsync
[root@master software]# rsync --daemon
failed to create pid file /usr/local/rsync/rsyncd.pid: File exists
[root@master software]# cd /usr/local/rsync/
[root@master rsync]# rsync --daemon
failed to create pid file /usr/local/rsync/rsyncd.pid: File exists
[root@master rsync]# ll
total 4
-rw-r--r-- 1 root root 6 Jul 21 17:36 rsyncd.pid
[root@master rsync]# cat rsyncd.pid
29351
[root@master rsync]# rm -rf rsyncd.pid
[root@master rsync]# rsync –daemon
[root@master rsync]# ps -ef|grep rsync
root 31629 1 0 18:19 ? 00:00:00 rsync --daemon
root 31634 2714 0 18:20 pts/1 00:00:00 grep rsync
方法五(works):查看下服务器端的日志,终于发现问题在哪里了
[root@master rsync]# tail -f /var/log/rsyncd.log
2014/07/21 18:20:35 [31636] params.c:Parameter() - Ignoring badly formed line in config file: ignore errors
2014/07/21 18:20:35 [31636] Unknown Parameter encountered:"screts file"
2014/07/21 18:20:35 [31636] IGNORING unknown parameter "screts file"
2014/07/21 18:20:35 [31636] connect from slave-5 (100.12.56.227)
2014/07/21 18:20:39 [31636] auth failed on module hadoop from slave-5 (100.12.56.227) for root: no matching rule
2014/07/21 18:23:11 [31651] params.c:Parameter() - Ignoring badly formed line in config file: ignore errors
2014/07/21 18:23:11 [31651] Unknown Parameter encountered: "screts file"
2014/07/21 18:23:11 [31651] IGNORING unknown parameter "screts file"
2014/07/21 18:23:11 [31651] connect from slave-5 (100.12.56.227)
2014/07/21 18:23:13 [31651] auth failed on module hadoop from slave-5 (100.12.56.227) for root: no secrets file
[root@master rsync]# vi /etc/rsyncd.conf
#global options
strict modes=yes
port=873
log file =/var/log/rsyncd.log
pid file =/usr/local/rsync/rsyncd.pid
#mode options
[hadoop]
max connections=10
uid=root
gid=root
path=/home/hadoop/hadoop-2.4.1
ignore errors
read only=no
write only=no
hosts allow=*
#hosts deny=
auth users=root
secrets file=/home/hadoop/rsync.pass
在客户端再进行测试
[root@slave-5 selinux]#rsync -vzrtopg --progress --delete root@master::hadoop /home/hadoop/testrsync
在服务端观察下日志如下:
[root@master rsync]# tail -f /var/log/rsyncd.log
2014/07/21 18:20:35 [31636] params.c:Parameter()- Ignoring badly formed line in config file: ignore errors
2014/07/21 18:20:35 [31636] Unknown Parameter encountered: "screts file"
2014/07/21 18:20:35 [31636] IGNORING unknown parameter "screts file"
2014/07/21 18:20:35 [31636] connect from slave-5 (100.12.56.227)
2014/07/21 18:20:39 [31636] auth failed on module hadoop from slave-5 (100.12.56.227) for root: no matching rule
2014/07/21 18:23:11 [31651] params.c:Parameter() - Ignoring badly formed line in config file: ignore errors
2014/07/21 18:23:11 [31651] Unknown Parameter encountered: "screts file"
2014/07/21 18:23:11 [31651] IGNORING unknown parameter "screts file"
2014/07/21 18:23:11 [31651] connect from slave-5 (100.12.56.227)
2014/07/21 18:23:13 [31651] auth failed on module hadoop from slave-5 (100.12.56.227) for root: no secrets file
2014/07/21 18:27:31 [31675] params.c:Parameter() - Ignoring badly formed line in config file: ignore errors
2014/07/21 18:27:31 [31675] connect from slave-5 (100.12.56.227)
2014/07/21 18:27:33 [31675] rsync on hadoop/ from root@slave-5 (100.12.56.227)
2014/07/21 10:27:33 [31675] building file list
2014/07/21 10:27:54 [31675] sent 143747859 bytes received 110760 bytes total size 238829369
在客户端查看同步的文件夹
[root@slave-5 selinux]#cd /home/hadoop/testrsync
[root@slave-5 testrsync]#ll
total 56
drwxr-xr-x 2 hadoop hadoop 4096 Jun 21 14:05 bin
drwxr-xr-x 3 hadoop hadoop 4096 Jun 21 14:05 etc
drwxr-xr-x 2 hadoop hadoop 4096 Jun 21 14:05 include
drwxr-xr-x 3 hadoop hadoop 4096 Jun 21 14:05 lib
drwxr-xr-x 2 hadoop hadoop 4096 Jun 21 14:05 libexec
-rw-r--r-- 1 hadoop hadoop 15458 Jun 21 14:38 LICENSE.txt
drwxrwxr-x 2 hadoop hadoop 4096 Jul 21 16:22 logs
-rw-r--r-- 1 hadoop hadoop 101 Jun 21 14:38 NOTICE.txt
-rw-r--r-- 1 hadoop hadoop 1366 Jun 21 14:38 README.txt
drwxr-xr-x 2 hadoop hadoop 4096 Jun 21 14:05 sbin
drwxr-xr-x 4 hadoop hadoop 4096 Jun 21 14:05 share
至此,测试ok
(责任编辑:IT)
首先下载对应linux的rsync软件:http://rsync.samba.org/download.html 本例中,下载rsync-3.1.1-1.el5.rfx.i386.rpm 22-Jun-2014 21:58 420K RHEL5 and CentOS-5 x86 32bit 服务端配置 [root@master software]#pwd /root/software
[root@master software]#ll -rwxr--r--1 root root 429855 Jul 18 17:52 rsync-3.1.1-1.el5.rfx.i386.rpm [root@master software]# rpm -ivh rsync-3.1.1-1.el5.rfx.i386.rpmwarning: rsync-3.1.1-1.el5.rfx.i386.rpm:Header V3 DSA signature: NOKEY, key ID 6b8d79e6 Preparing... ########################################### [100%]
file /etc/xinetd.d/rsync from install of rsync-3.1.1-1.el5.rfx.i386 conflicts with file from package rsync-2.6.8-3.1.i386 从日志信息可以看出和本机已经安装的rsync-2.6.8-3.1.i386冲突了。需要先卸载该包。 [root@master software]# rpm -e rsync-2.6.8-3.1 warning: /etc/xinetd.d/rsync saved as /etc/xinetd.d/rsync.rpmsave [root@master software]# rpm -qa|grep rsync [root@master software]# rpm -ivh rsync-3.1.1-1.el5.rfx.i386.rpm warning: rsync-3.1.1-1.el5.rfx.i386.rpm:Header V3 DSA signature: NOKEY, key ID 6b8d79e6
Preparing... [root@master software]#rpm -qa|grep rsync rsync-3.1.1-1.el5.rfx [root@master software]#vi /etc/rsyncd.conf
#global options log file=/var/log/rsyncd.log pid file=/usr/local/rsync/rsyncd.pid #mode options [hadoop] max connections=10 uid=root gid =root path = /home/hadoop/hadoop-2.4.1 ignore errors read only = no write only = no hosts allow = * #hosts deny = auth users = root screts file = /home/hadoop/rsync.pass #(注意:这里的secrets单词写错了,导致后面权限验证通不过) [root@master software]#vi /home/hadoop/rsync.pass root:beijing [root@master software]#chmod 777 /home/hadoop/rsync.pass [root@master ~]# vi /etc/xinetd.d/rsync # default: off # description: The rsync server is a good addition to an ftp server, as it \ # allows crc checksumming etc. service rsync
{
server = /usr/bin/rsync }
[root@master ~]#chkconfig --list|grep sync [root@master software]#rsync --daemon failed to create pid file /usr/local/rsync/rsyncd.pid:No such file or directory [root@master software]#mkdir /usr/local/rsync/ [root@master software]#rsync –daemon [root@master software]#netstat -an |grep 873
tcp 0
tcp 0 [root@master software]#ps -ef|grep rsync root 29351 1 0 17:36 ? 00:00:00 rsync --daemon root 31429 2714 0 17:41 pts/1 00:00:00 grep rsync 在客户端slave-5测试同步 [root@slave-5 ~]# rsync -vzrtopg --progress --delete root@master::hadoop /home/hadoop/testrsync Password: @ERROR: auth failed on module hadoop rsync error: error starting client-server protocol (code 5) at main.c(1296) [receiver=2.6.8] 可见由于用户验证未通过,该操作失败了,下面用了几种方式进行调整并测试 方法一(not works): 虽然该方法对本例未奏效,但是建议chmod 600 /home/hadoop/rsync.pass [root@master software]# ll /home/hadoop/rsync.pass -rwxrwxrwx 1 root root 13 Jul 21 17:23 /home/hadoop/rsync.pass [root@master software]# chmod 600 /home/hadoop/rsync.pass [root@master software]# ll /home/hadoop/rsync.pass -rw-------1 root root 13 Jul 21 17:23 /home/hadoop/rsync.pass [root@slave-5 ~]# rsync -vzrtopg --progress --delete root@master::hadoop /home/hadoop/testrsync Password: @ERROR: auth failed on module hadoop rsync error: error starting client-server protocol (code 5) at main.c(1296) [receiver=2.6.8] 方法二(not works): 在客户端更新跟服务器相同版本的rsync(后经测试,低版本的客户端也是能够正常从服务端同步文件的) [root@slave-5 ~]# rpm -e rsync-2.6.8-3.1 [root@slave-5 software]# rpm -ivh rsync-3.1.1-1.el5.rfx.i386.rpm warning:rsync-3.1.1-1.el5.rfx.i386.rpm: Header V3 DSA signature: NOKEY, key ID 6b8d79e6 Preparing... ###########################################[100%] 1:rsync ########################################### [100%] [root@slave-5 software]# rsync --daemon Failed to parse config file: /etc/rsyncd.conf [root@slave-5 software]# touch /etc/rsyncd.conf [root@slave-5 software]# rsync –daemon [root@slave-5 software]# rsync -vzrtopg --progress --delete root@master::hadoop /home/hadoop/testrsync Password: @ERROR: auth failed on module hadoop rsync error: error starting client-server protocol (code 5) at main.c(1635) [Receiver=3.1.1] 方法三(not works): [root@slave-5 ~]# mkdir /home/hadoop/testrsync [root@slave-5 ~]# rsync -vzrtopg --progress --delete root@master::hadoop /home/hadoop/testrsync Password: @ERROR: auth failed on module hadoop rsync error: error starting client-server protocol (code 5) at main.c(1635) [Receiver=3.1.1] 方法四(not works):重启下服务端试试 [root@master software]# ps -ef|grep rsync root 29351 1 0 17:36 ? 00:00:00 rsync --daemon root 31614 2714 0 18:17 pts/1 00:00:00 grep rsync [root@master software]# kill -9 29351 [root@master software]# ps -ef|grep rsync root 31616 2714 0 18:18 pts/1 00:00:00 grep rsync [root@master software]# rsync --daemon failed to create pid file /usr/local/rsync/rsyncd.pid: File exists [root@master software]# cd /usr/local/rsync/ [root@master rsync]# rsync --daemon failed to create pid file /usr/local/rsync/rsyncd.pid: File exists [root@master rsync]# ll total 4 -rw-r--r-- 1 root root 6 Jul 21 17:36 rsyncd.pid
[root@master rsync]# cat rsyncd.pid [root@master rsync]# rm -rf rsyncd.pid [root@master rsync]# rsync –daemon [root@master rsync]# ps -ef|grep rsync root 31629 1 0 18:19 ? 00:00:00 rsync --daemon root 31634 2714 0 18:20 pts/1 00:00:00 grep rsync 方法五(works):查看下服务器端的日志,终于发现问题在哪里了 [root@master rsync]# tail -f /var/log/rsyncd.log 2014/07/21 18:20:35 [31636] params.c:Parameter() - Ignoring badly formed line in config file: ignore errors 2014/07/21 18:20:35 [31636] Unknown Parameter encountered:"screts file" 2014/07/21 18:20:35 [31636] IGNORING unknown parameter "screts file" 2014/07/21 18:20:35 [31636] connect from slave-5 (100.12.56.227) 2014/07/21 18:20:39 [31636] auth failed on module hadoop from slave-5 (100.12.56.227) for root: no matching rule 2014/07/21 18:23:11 [31651] params.c:Parameter() - Ignoring badly formed line in config file: ignore errors 2014/07/21 18:23:11 [31651] Unknown Parameter encountered: "screts file" 2014/07/21 18:23:11 [31651] IGNORING unknown parameter "screts file" 2014/07/21 18:23:11 [31651] connect from slave-5 (100.12.56.227) 2014/07/21 18:23:13 [31651] auth failed on module hadoop from slave-5 (100.12.56.227) for root: no secrets file [root@master rsync]# vi /etc/rsyncd.conf #global options strict modes=yes port=873 log file =/var/log/rsyncd.log pid file =/usr/local/rsync/rsyncd.pid #mode options [hadoop] max connections=10 uid=root gid=root path=/home/hadoop/hadoop-2.4.1 ignore errors read only=no write only=no hosts allow=* #hosts deny= auth users=root secrets file=/home/hadoop/rsync.pass 在客户端再进行测试 [root@slave-5 selinux]#rsync -vzrtopg --progress --delete root@master::hadoop /home/hadoop/testrsync 在服务端观察下日志如下: [root@master rsync]# tail -f /var/log/rsyncd.log 2014/07/21 18:20:35 [31636] params.c:Parameter()- Ignoring badly formed line in config file: ignore errors 2014/07/21 18:20:35 [31636] Unknown Parameter encountered: "screts file" 2014/07/21 18:20:35 [31636] IGNORING unknown parameter "screts file" 2014/07/21 18:20:35 [31636] connect from slave-5 (100.12.56.227) 2014/07/21 18:20:39 [31636] auth failed on module hadoop from slave-5 (100.12.56.227) for root: no matching rule 2014/07/21 18:23:11 [31651] params.c:Parameter() - Ignoring badly formed line in config file: ignore errors 2014/07/21 18:23:11 [31651] Unknown Parameter encountered: "screts file" 2014/07/21 18:23:11 [31651] IGNORING unknown parameter "screts file" 2014/07/21 18:23:11 [31651] connect from slave-5 (100.12.56.227) 2014/07/21 18:23:13 [31651] auth failed on module hadoop from slave-5 (100.12.56.227) for root: no secrets file 2014/07/21 18:27:31 [31675] params.c:Parameter() - Ignoring badly formed line in config file: ignore errors 2014/07/21 18:27:31 [31675] connect from slave-5 (100.12.56.227) 2014/07/21 18:27:33 [31675] rsync on hadoop/ from root@slave-5 (100.12.56.227) 2014/07/21 10:27:33 [31675] building file list 2014/07/21 10:27:54 [31675] sent 143747859 bytes received 110760 bytes total size 238829369 在客户端查看同步的文件夹 [root@slave-5 selinux]#cd /home/hadoop/testrsync [root@slave-5 testrsync]#ll total 56 drwxr-xr-x 2 hadoop hadoop 4096 Jun 21 14:05 bin drwxr-xr-x 3 hadoop hadoop 4096 Jun 21 14:05 etc drwxr-xr-x 2 hadoop hadoop 4096 Jun 21 14:05 include drwxr-xr-x 3 hadoop hadoop 4096 Jun 21 14:05 lib drwxr-xr-x 2 hadoop hadoop 4096 Jun 21 14:05 libexec -rw-r--r-- 1 hadoop hadoop 15458 Jun 21 14:38 LICENSE.txt drwxrwxr-x 2 hadoop hadoop 4096 Jul 21 16:22 logs -rw-r--r-- 1 hadoop hadoop 101 Jun 21 14:38 NOTICE.txt -rw-r--r-- 1 hadoop hadoop 1366 Jun 21 14:38 README.txt drwxr-xr-x 2 hadoop hadoop 4096 Jun 21 14:05 sbin
drwxr-xr-x 4 hadoop hadoop 4096 Jun 21 14:05 share 至此,测试ok (责任编辑:IT) |