Hadoop1.2.1集群安装三
时间:2016-12-24 17:36 来源:linux.it.net.cn 作者:IT
配置Hadoop
1:下载hadoop-1.2.1.tar.gz
在/home/it 创建目录 mkdir hadoop
2:解压
[it@it.net.cn hadoop]$ ls
hadoop-1.2.1.tar.gz
[it@it.net.cn hadoop]$ tar zxf hadoop-1.2.1.tar.gz
[it@it.net.cn hadoop]$ ls
hadoop-1.2.1 hadoop-1.2.1.tar.gz
[it@it.net.cn hadoop]$
3: 修改hadoop-env.sh配置文件
[it@it.net.cn hadoop]$ cd hadoop-1.2.1
[it@it.net.cn hadoop-1.2.1]$ ls
bin hadoop-ant-1.2.1.jar ivy sbin
build.xml hadoop-client-1.2.1.jar ivy.xml share
c++ hadoop-core-1.2.1.jar lib src
CHANGES.txt hadoop-examples-1.2.1.jar libexec webapps
conf hadoop-minicluster-1.2.1.jar LICENSE.txt
contrib hadoop-test-1.2.1.jar NOTICE.txt
docs hadoop-tools-1.2.1.jar README.txt
[it@it.net.cn hadoop-1.2.1]$ cd conf
[it@it.net.cn conf]$ ls
capacity-scheduler.xml hadoop-policy.xml slaves
configuration.xsl hdfs-site.xml ssl-client.xml.example
core-site.xml log4j.properties ssl-server.xml.example
fair-scheduler.xml mapred-queue-acls.xml taskcontroller.cfg
hadoop-env.sh mapred-site.xml task-log4j.properties
hadoop-metrics2.properties masters
[it@it.net.cn conf]$ vi hadoop-env.sh
# Set Hadoop-specific environment variables here.
# The only required environment variable is JAVA_HOME. All others are
# optional. When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.
# The java implementation to use. Required.
export JAVA_HOME=/home/it/jdk1.7.0_45
# Extra Java CLASSPATH elements. Optional.
# export HADOOP_CLASSPATH=
# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=2000
# Extra Java runtime options. Empty by default.
# export HADOOP_OPTS=-server
# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
@
"hadoop-env.sh" 57L, 2436C 已写入
[it@it.net.cn conf]$ cat hadoop-env.sh
把# export JAVA_HOME 修改为“export JAVA_HOME=/home/it/jdk1.7.0_45”
4:修改core-site.xml文件
在hadoop目录下创建目录
[it@it.net.cn hadoop]$ mkdir tmp
[it@it.net.cn conf]$ vi core-site.xml
修改后如下：
[it@it.net.cn conf]$ cat core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://it.net.cn:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/it/hadoop/tmp</value>
</property>
</configuration>
5：修改hdfs-site.xml
修改后如下：
[it@it.net.cn conf]$ cat hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
<description></description>
</property>
</configuration>
6:修改mapred-site.xml文件
修改后如下：
[it@it.net.cn conf]$ cat mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>it.net.cn:9001</value>
<description>NameNode</description>
</property>
</configuration>
[it@it.net.cn conf]$
7：修改masters和slaves文件
修改后路下
[it@it.net.cn conf]$ cat masters
it.net.cn
[it@it.net.cn conf]$ cat slaves
it02
it03
[it@it.net.cn conf]$
8：先其它2个节点复制hadoop-1.2.1
[it@it.net.cn hadoop]$ scp -r ./hadoop-1.2.1 it02:/home/it/hadoop
[it@it.net.cn hadoop]$ scp -r ./hadoop-1.2.1 it03:/home/it/hadoop
9：格式化分布式文件系统
[it@it.net.cn hadoop-1.2.1]$ bin/hadoop namenode -format
14/07/24 10:29:43 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = it.net.cn/10.3.7.214
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.7.0_45
************************************************************/
14/07/24 10:29:43 INFO util.GSet: Computing capacity for map BlocksMap
14/07/24 10:29:43 INFO util.GSet: VM type = 64-bit
14/07/24 10:29:43 INFO util.GSet: 2.0% max memory = 932184064
14/07/24 10:29:43 INFO util.GSet: capacity = 2^21 = 2097152 entries
14/07/24 10:29:43 INFO util.GSet: recommended=2097152, actual=2097152
14/07/24 10:29:43 INFO namenode.FSNamesystem: fsOwner=it
14/07/24 10:29:43 INFO namenode.FSNamesystem: supergroup=supergroup
14/07/24 10:29:43 INFO namenode.FSNamesystem: isPermissionEnabled=true
14/07/24 10:29:43 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
14/07/24 10:29:43 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
14/07/24 10:29:43 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
14/07/24 10:29:43 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/07/24 10:29:43 INFO common.Storage: Image file /home/it/hadoop/tmp/dfs/name/current/fsimage of size 112 bytes saved in 0 seconds.
14/07/24 10:29:44 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/home/it/hadoop/tmp/dfs/name/current/edits
14/07/24 10:29:44 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/home/it/hadoop/tmp/dfs/name/current/edits
14/07/24 10:29:44 INFO common.Storage: Storage directory /home/it/hadoop/tmp/dfs/name has been successfully formatted.
14/07/24 10:29:44 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at it.net.cn/10.3.7.214
************************************************************/
[it@it.net.cn hadoop-1.2.1]$
10：启动hadoop
[it@it.net.cn hadoop-1.2.1]$ bin/start-all.sh
starting namenode, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-namenode-it.net.cn.out
it03: starting datanode, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-datanode-it03.out
it02: starting datanode, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-datanode-it02.out
The authenticity of host 'it.net.cn (10.3.7.214)' can't be established.
RSA key fingerprint is a8:9d:34:63:fa:c2:47:4f:81:10:94:fa:4b:ba:08:55.
Are you sure you want to continue connecting (yes/no)? yes
it.net.cn: Warning: Permanently added 'it.net.cn,10.3.7.214' (RSA) to the list of known hosts.
it@it.net.cn's password:
it@it.net.cn's password: it.net.cn: Permission denied, please try again.
it.net.cn: starting secondarynamenode, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-secondarynamenode-it.net.cn.out
starting jobtracker, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-jobtracker-it.net.cn.out
it03: starting tasktracker, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-tasktracker-it03.out
it02: starting tasktracker, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-tasktracker-it02.out
[it@it.net.cn hadoop-1.2.1]$
需要输入密码
11：检测守护进程
[it@it.net.cn hadoop-1.2.1]$ jps
4539 JobTracker
4454 SecondaryNameNode
4269 NameNode
4667 Jps
[it@it.net.cn hadoop-1.2.1]$
[it@it02 hadoop]$ jps
2734 TaskTracker
2815 Jps
2647 DataNode
[it@it02 hadoop]$
[it@it03 hadoop]$ jps
4070 Jps
3878 DataNode
3993 TaskTracker
[it@it03 hadoop]$
(责任编辑：IT)
| 配置Hadoop 1:下载hadoop-1.2.1.tar.gz 在/home/it 创建目录 mkdir hadoop 2:解压 [it@it.net.cn hadoop]$ ls hadoop-1.2.1.tar.gz [it@it.net.cn hadoop]$ tar zxf hadoop-1.2.1.tar.gz [it@it.net.cn hadoop]$ ls hadoop-1.2.1 hadoop-1.2.1.tar.gz [it@it.net.cn hadoop]$ 3: 修改hadoop-env.sh配置文件 [it@it.net.cn hadoop]$ cd hadoop-1.2.1 [it@it.net.cn hadoop-1.2.1]$ ls bin hadoop-ant-1.2.1.jar ivy sbin build.xml hadoop-client-1.2.1.jar ivy.xml share c++ hadoop-core-1.2.1.jar lib src CHANGES.txt hadoop-examples-1.2.1.jar libexec webapps conf hadoop-minicluster-1.2.1.jar LICENSE.txt contrib hadoop-test-1.2.1.jar NOTICE.txt docs hadoop-tools-1.2.1.jar README.txt [it@it.net.cn hadoop-1.2.1]$ cd conf [it@it.net.cn conf]$ ls capacity-scheduler.xml hadoop-policy.xml slaves configuration.xsl hdfs-site.xml ssl-client.xml.example core-site.xml log4j.properties ssl-server.xml.example fair-scheduler.xml mapred-queue-acls.xml taskcontroller.cfg hadoop-env.sh mapred-site.xml task-log4j.properties hadoop-metrics2.properties masters [it@it.net.cn conf]$ vi hadoop-env.sh # Set Hadoop-specific environment variables here. # The only required environment variable is JAVA_HOME. All others are # optional. When running a distributed configuration it is best to # set JAVA_HOME in this file, so that it is correctly defined on # remote nodes. # The java implementation to use. Required. export JAVA_HOME=/home/it/jdk1.7.0_45 # Extra Java CLASSPATH elements. Optional. # export HADOOP_CLASSPATH= # The maximum amount of heap to use, in MB. Default is 1000. # export HADOOP_HEAPSIZE=2000 # Extra Java runtime options. Empty by default. # export HADOOP_OPTS=-server # Command specific options appended to HADOOP_OPTS when specified export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS" @ "hadoop-env.sh" 57L, 2436C 已写入 [it@it.net.cn conf]$ cat hadoop-env.sh 把# export JAVA_HOME 修改为“export JAVA_HOME=/home/it/jdk1.7.0_45” 4:修改core-site.xml文件 在hadoop目录下创建目录 [it@it.net.cn hadoop]$ mkdir tmp [it@it.net.cn conf]$ vi core-site.xml 修改后如下： [it@it.net.cn conf]$ cat core-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>fs.default.name</name> <value>hdfs://it.net.cn:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/it/hadoop/tmp</value> </property> </configuration> 5：修改hdfs-site.xml 修改后如下： [it@it.net.cn conf]$ cat hdfs-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>dfs.replication</name> <value>1</value> <description></description> </property> </configuration> 6:修改mapred-site.xml文件 修改后如下： [it@it.net.cn conf]$ cat mapred-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>mapred.job.tracker</name> <value>it.net.cn:9001</value> <description>NameNode</description> </property> </configuration> [it@it.net.cn conf]$ 7：修改masters和slaves文件 修改后路下 [it@it.net.cn conf]$ cat masters it.net.cn [it@it.net.cn conf]$ cat slaves it02 it03 [it@it.net.cn conf]$ 8：先其它2个节点复制hadoop-1.2.1 [it@it.net.cn hadoop]$ scp -r ./hadoop-1.2.1 it02:/home/it/hadoop [it@it.net.cn hadoop]$ scp -r ./hadoop-1.2.1 it03:/home/it/hadoop 9：格式化分布式文件系统 [it@it.net.cn hadoop-1.2.1]$ bin/hadoop namenode -format 14/07/24 10:29:43 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = it.net.cn/10.3.7.214 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 1.2.1 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013 STARTUP_MSG: java = 1.7.0_45 ************************************************************/ 14/07/24 10:29:43 INFO util.GSet: Computing capacity for map BlocksMap 14/07/24 10:29:43 INFO util.GSet: VM type = 64-bit 14/07/24 10:29:43 INFO util.GSet: 2.0% max memory = 932184064 14/07/24 10:29:43 INFO util.GSet: capacity = 2^21 = 2097152 entries 14/07/24 10:29:43 INFO util.GSet: recommended=2097152, actual=2097152 14/07/24 10:29:43 INFO namenode.FSNamesystem: fsOwner=it 14/07/24 10:29:43 INFO namenode.FSNamesystem: supergroup=supergroup 14/07/24 10:29:43 INFO namenode.FSNamesystem: isPermissionEnabled=true 14/07/24 10:29:43 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100 14/07/24 10:29:43 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 14/07/24 10:29:43 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0 14/07/24 10:29:43 INFO namenode.NameNode: Caching file names occuring more than 10 times 14/07/24 10:29:43 INFO common.Storage: Image file /home/it/hadoop/tmp/dfs/name/current/fsimage of size 112 bytes saved in 0 seconds. 14/07/24 10:29:44 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/home/it/hadoop/tmp/dfs/name/current/edits 14/07/24 10:29:44 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/home/it/hadoop/tmp/dfs/name/current/edits 14/07/24 10:29:44 INFO common.Storage: Storage directory /home/it/hadoop/tmp/dfs/name has been successfully formatted. 14/07/24 10:29:44 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at it.net.cn/10.3.7.214 ************************************************************/ [it@it.net.cn hadoop-1.2.1]$ 10：启动hadoop [it@it.net.cn hadoop-1.2.1]$ bin/start-all.sh starting namenode, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-namenode-it.net.cn.out it03: starting datanode, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-datanode-it03.out it02: starting datanode, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-datanode-it02.out The authenticity of host 'it.net.cn (10.3.7.214)' can't be established. RSA key fingerprint is a8:9d:34:63:fa:c2:47:4f:81:10:94:fa:4b:ba:08:55. Are you sure you want to continue connecting (yes/no)? yes it.net.cn: Warning: Permanently added 'it.net.cn,10.3.7.214' (RSA) to the list of known hosts. it@it.net.cn's password: it@it.net.cn's password: it.net.cn: Permission denied, please try again. it.net.cn: starting secondarynamenode, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-secondarynamenode-it.net.cn.out starting jobtracker, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-jobtracker-it.net.cn.out it03: starting tasktracker, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-tasktracker-it03.out it02: starting tasktracker, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-tasktracker-it02.out [it@it.net.cn hadoop-1.2.1]$ 需要输入密码 11：检测守护进程 [it@it.net.cn hadoop-1.2.1]$ jps 4539 JobTracker 4454 SecondaryNameNode 4269 NameNode 4667 Jps [it@it.net.cn hadoop-1.2.1]$ [it@it02 hadoop]$ jps 2734 TaskTracker 2815 Jps 2647 DataNode [it@it02 hadoop]$ [it@it03 hadoop]$ jps 4070 Jps 3878 DataNode 3993 TaskTracker [it@it03 hadoop]$ (责任编辑：IT) |