当前位置: > Linux集群 > Hadoop >

Hadoop2.2.0安装配置手册!完全分布式Hadoop集群搭建过程(3)

时间:2014-12-29 17:45来源:linux.it.net.cn 作者:IT


第三部分 Hadoop 2.2安装过程

由于hadoop集群中每个机器上面的配置基本相同,所以我们先在namenode上面进行配置部署,然后再复制到其他节点。所以这里的安装过程相当于在每台机器上面都要执行。但需要注意的是集群中64位系统和32位系统的问题。

1、 解压文件

将第一部分中下载的hadoop-2.2.tar.gz解压到/home/hduser路径下(或者将在64位机器上编译的结果存放在此路径下)。然后为了节省空间,可删除此压缩文件,或将其存放于其他地方进行备份。

注意:每台机器的安装路径要相同!!

2、 hadoop配置过程

配置之前,需要在cloud001本地文件系统创建以下文件夹:

~/dfs/name

~/dfs/data

~/temp

这里要涉及到的配置文件有7个:

~/hadoop-2.2.0/etc/hadoop/hadoop-env.sh

~/hadoop-2.2.0/etc/hadoop/yarn-env.sh

~/hadoop-2.2.0/etc/hadoop/slaves

~/hadoop-2.2.0/etc/hadoop/core-site.xml

~/hadoop-2.2.0/etc/hadoop/hdfs-site.xml

~/hadoop-2.2.0/etc/hadoop/mapred-site.xml

~/hadoop-2.2.0/etc/hadoop/yarn-site.xml

以上个别文件默认不存在的,可以复制相应的template文件获得。

配置文件1:hadoop-env.sh

修改JAVA_HOME值(export JAVA_HOME=/usr/java/jdk1.7.0_40)

配置文件2:yarn-env.sh

修改JAVA_HOME值(exportJAVA_HOME=/usr/java/jdk1.7.0_40)

配置文件3:slaves (这个文件里面保存所有slave节点)

写入以下内容:

cloud002

cloud003

配置文件4:core-site.xml

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://cloud001:9000</value>

</property>

<property>

<name>io.file.buffer.size</name>

<value>131072</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>file:/home/hduser/tmp</value>

<description>Abase for other temporary directories.</description>

</property>

<property>

<name>hadoop.proxyuser.hduser.hosts</name>

<value>*</value>

</property>

<property>

<name>hadoop.proxyuser.hduser.groups</name>

<value>*</value>

</property>

</configuration>

配置文件5:hdfs-site.xml

<configuration>

<property>

<name>dfs.namenode.secondary.http-address</name>

<value>cloud001:9001</value>

</property>

<property>

<name>dfs.namenode.name.dir</name>

<value>file:/home/hduser/dfs/name</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>file:/home/hduser/dfs/data</value>

</property>

<property>

<name>dfs.replication</name>

<value>3</value>

</property>

<property>

<name>dfs.webhdfs.enabled</name>

<value>true</value>

</property>

</configuration>

配置文件6:mapred-site.xml

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

<property>

<name>mapreduce.jobhistory.address</name>

<value>cloud001:10020</value>

</property>

<property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>cloud001:19888</value>

</property>

</configuration>

配置文件7:yarn-site.xml

<configuration>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<property>

<name>yarn.resourcemanager.address</name>

<value>cloud001:8032</value>

</property>

<property>

<name>yarn.resourcemanager.scheduler.address</name>

<value> cloud001:8030</value>

</property>

<property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value> cloud001:8031</value>

</property>

<property>

<name>yarn.resourcemanager.admin.address</name>

<value> cloud001:8033</value>

</property>

<property>

<name>yarn.resourcemanager.webapp.address</name>

<value> cloud001:8088</value>

</property> 

</configuration>

3、复制到其他节点

这里可以写一个shell脚本进行操作(有大量节点时比较方便)

cp2slave.sh

#!/bin/bash

scp–r /home/hduser/hadoop-2.2.0 hduser@cloud002:~/

scp–r /home/hduser/hadoop-2.2.0 hduser@cloud003:~/

注意:由于我们集群里面001是64bit 而002和003是32bit的,所以不能直接复制,而采用单独安装hadoop,复制替换相关配置文件:

Cp2slave2.sh

#!/bin/bash

scp /home/hduser/hadoop-2.2.0/etc/hadoop/slaveshduser@cloud002:~/hadoop-2.2.0/etc/hadoop/slaves

scp /home/hduser/hadoop-2.2.0/etc/hadoop/slaveshduser@cloud003:~/hadoop-2.2.0/etc/hadoop/slaves

scp /home/hduser/hadoop-2.2.0/etc/hadoop/core-site.xmlhduser@cloud002:~/hadoop-2.2.0/etc/hadoop/core-site.xml

scp /home/hduser/hadoop-2.2.0/etc/hadoop/core-site.xml hduser@cloud003:~/hadoop-2.2.0/etc/hadoop/core-site.xml

scp /home/hduser/hadoop-2.2.0/etc/hadoop/hdfs-site.xmlhduser@cloud002:~/hadoop-2.2.0/etc/hadoop/hdfs-site.xml

scp /home/hduser/hadoop-2.2.0/etc/hadoop/hdfs-site.xmlhduser@cloud003:~/hadoop-2.2.0/etc/hadoop/hdfs-site.xml

scp /home/hduser/hadoop-2.2.0/etc/hadoop/mapred-site.xmlhduser@cloud002:~/hadoop-2.2.0/etc/hadoop/mapred-site.xml

scp /home/hduser/hadoop-2.2.0/etc/hadoop/mapred-site.xmlhduser@cloud003:~/hadoop-2.2.0/etc/hadoop/mapred-site.xml

scp /home/hduser/hadoop-2.2.0/etc/hadoop/yarn-site.xmlhduser@cloud002:~/hadoop-2.2.0/etc/hadoop/yarn-site.xml

scp /home/hduser/hadoop-2.2.0/etc/hadoop/yarn-site.xmlhduser@cloud003:~/hadoop-2.2.0/etc/hadoop/yarn-site.xml

4、启动验证

4.1 启动hadoop

进入安装目录: cd ~/hadoop-2.2.0/

格式化namenode:./bin/hdfs namenode –format

启动hdfs: ./sbin/start-dfs.sh

此时在001上面运行的进程有:namenode secondarynamenode

002和003上面运行的进程有:datanode

启动yarn: ./sbin/start-yarn.sh

此时在001上面运行的进程有:namenode secondarynamenoderesourcemanager

002和003上面运行的进程有:datanode nodemanaget

查看集群状态:./bin/hdfs dfsadmin –report

 

查看文件块组成: ./bin/hdfsfsck / -files -blocks

查看HDFS: http://16.187.94.161:50070

查看RM: http:// 16.187.94.161:8088

4.2 运行示例程序:

先在hdfs上创建一个文件夹

./bin/hdfs dfs –mkdir /input

./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jarrandomwriter input


 
(责任编辑:IT)
------分隔线----------------------------