当前位置: > Linux集群 > Hadoop >

Hadoop1.2.1集群安装三

时间:2016-12-24 17:36来源:linux.it.net.cn 作者:IT
配置Hadoop

1:下载hadoop-1.2.1.tar.gz

   在/home/it 创建目录   mkdir hadoop 

2:解压


[it@it.net.cn hadoop]$ ls 
hadoop-1.2.1.tar.gz 
[it@it.net.cn hadoop]$ tar zxf hadoop-1.2.1.tar.gz  
[it@it.net.cn hadoop]$ ls 
hadoop-1.2.1  hadoop-1.2.1.tar.gz 
[it@it.net.cn hadoop]$  

3: 修改hadoop-env.sh配置文件

[it@it.net.cn hadoop]$ cd hadoop-1.2.1 
[it@it.net.cn hadoop-1.2.1]$ ls 
bin          hadoop-ant-1.2.1.jar          ivy          sbin 
build.xml    hadoop-client-1.2.1.jar       ivy.xml      share 
c++          hadoop-core-1.2.1.jar         lib          src 
CHANGES.txt  hadoop-examples-1.2.1.jar     libexec      webapps 
conf         hadoop-minicluster-1.2.1.jar  LICENSE.txt 
contrib      hadoop-test-1.2.1.jar         NOTICE.txt 
docs         hadoop-tools-1.2.1.jar        README.txt 
[it@it.net.cn hadoop-1.2.1]$ cd conf 
[it@it.net.cn conf]$ ls 
capacity-scheduler.xml      hadoop-policy.xml      slaves 
configuration.xsl           hdfs-site.xml          ssl-client.xml.example 
core-site.xml               log4j.properties       ssl-server.xml.example 
fair-scheduler.xml          mapred-queue-acls.xml  taskcontroller.cfg 
hadoop-env.sh               mapred-site.xml        task-log4j.properties 
hadoop-metrics2.properties  masters 
[it@it.net.cn conf]$ vi hadoop-env.sh 
 
 
 
 
# Set Hadoop-specific environment variables here. 
 
# The only required environment variable is JAVA_HOME.  All others are 
# optional.  When running a distributed configuration it is best to 
# set JAVA_HOME in this file, so that it is correctly defined on 
# remote nodes. 
 
# The java implementation to use.  Required. 
export JAVA_HOME=/home/it/jdk1.7.0_45 
 
# Extra Java CLASSPATH elements.  Optional. 
# export HADOOP_CLASSPATH= 
 
# The maximum amount of heap to use, in MB. Default is 1000. 
# export HADOOP_HEAPSIZE=2000 
 
# Extra Java runtime options.  Empty by default. 
# export HADOOP_OPTS=-server 
 
# Command specific options appended to HADOOP_OPTS when specified 
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS" 

"hadoop-env.sh" 57L, 2436C 已写入                              
[it@it.net.cn conf]$ cat hadoop-env.sh 
把#  export JAVA_HOME 修改为“export JAVA_HOME=/home/it/jdk1.7.0_45”

4:修改core-site.xml文件

在hadoop目录下创建目录

[it@it.net.cn hadoop]$ mkdir tmp


[it@it.net.cn conf]$ vi core-site.xml 
修改后如下:

[it@it.net.cn conf]$ cat core-site.xml 
<?xml version="1.0"?> 
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 
 
<!-- Put site-specific property overrides in this file. --> 
 
<configuration> 
<property> 
<name>fs.default.name</name>   
  <value>hdfs://it.net.cn:9000</value>   
</property> 
 
<property> 
<name>hadoop.tmp.dir</name>   
  <value>/home/it/hadoop/tmp</value>   
</property> 
</configuration> 

5:修改hdfs-site.xml
修改后如下:

[it@it.net.cn conf]$ cat hdfs-site.xml 
<?xml version="1.0"?> 
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 
 
<!-- Put site-specific property overrides in this file. --> 
 
<configuration> 
<property>   
            <name>dfs.replication</name>   
            <value>1</value>   
            <description></description>   
</property>   
</configuration> 

6:修改mapred-site.xml文件
修改后如下:

[it@it.net.cn conf]$ cat  mapred-site.xml 
<?xml version="1.0"?> 
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 
 
<!-- Put site-specific property overrides in this file. --> 
 
<configuration> 
<property>   
  <name>mapred.job.tracker</name>   
  <value>it.net.cn:9001</value>   
  <description>NameNode</description>   
</property>   
</configuration> 
[it@it.net.cn conf]$  

7:修改masters和slaves文件
修改后路下


[it@it.net.cn conf]$ cat masters 
it.net.cn 
[it@it.net.cn conf]$ cat slaves 
it02 
it03 
[it@it.net.cn conf]$  

8:先其它2个节点复制hadoop-1.2.1
[it@it.net.cn hadoop]$ scp -r ./hadoop-1.2.1 it02:/home/it/hadoop
[it@it.net.cn hadoop]$ scp -r ./hadoop-1.2.1 it03:/home/it/hadoop

9:格式化分布式文件系统


[it@it.net.cn hadoop-1.2.1]$ bin/hadoop namenode -format 
14/07/24 10:29:43 INFO namenode.NameNode: STARTUP_MSG:  
/************************************************************ 
STARTUP_MSG: Starting NameNode 
STARTUP_MSG:   host = it.net.cn/10.3.7.214 
STARTUP_MSG:   args = [-format] 
STARTUP_MSG:   version = 1.2.1 
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013 
STARTUP_MSG:   java = 1.7.0_45 
************************************************************/ 
14/07/24 10:29:43 INFO util.GSet: Computing capacity for map BlocksMap 
14/07/24 10:29:43 INFO util.GSet: VM type       = 64-bit 
14/07/24 10:29:43 INFO util.GSet: 2.0% max memory = 932184064 
14/07/24 10:29:43 INFO util.GSet: capacity      = 2^21 = 2097152 entries 
14/07/24 10:29:43 INFO util.GSet: recommended=2097152, actual=2097152 
14/07/24 10:29:43 INFO namenode.FSNamesystem: fsOwner=it 
14/07/24 10:29:43 INFO namenode.FSNamesystem: supergroup=supergroup 
14/07/24 10:29:43 INFO namenode.FSNamesystem: isPermissionEnabled=true 
14/07/24 10:29:43 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100 
14/07/24 10:29:43 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 
14/07/24 10:29:43 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0 
14/07/24 10:29:43 INFO namenode.NameNode: Caching file names occuring more than 10 times  
14/07/24 10:29:43 INFO common.Storage: Image file /home/it/hadoop/tmp/dfs/name/current/fsimage of size 112 bytes saved in 0 seconds. 
14/07/24 10:29:44 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/home/it/hadoop/tmp/dfs/name/current/edits 
14/07/24 10:29:44 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/home/it/hadoop/tmp/dfs/name/current/edits 
14/07/24 10:29:44 INFO common.Storage: Storage directory /home/it/hadoop/tmp/dfs/name has been successfully formatted. 
14/07/24 10:29:44 INFO namenode.NameNode: SHUTDOWN_MSG:  
/************************************************************ 
SHUTDOWN_MSG: Shutting down NameNode at it.net.cn/10.3.7.214 
************************************************************/ 
[it@it.net.cn hadoop-1.2.1]$  

10:启动hadoop

[it@it.net.cn hadoop-1.2.1]$ bin/start-all.sh 
starting namenode, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-namenode-it.net.cn.out 
it03: starting datanode, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-datanode-it03.out 
it02: starting datanode, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-datanode-it02.out 
The authenticity of host 'it.net.cn (10.3.7.214)' can't be established. 
RSA key fingerprint is a8:9d:34:63:fa:c2:47:4f:81:10:94:fa:4b:ba:08:55. 
Are you sure you want to continue connecting (yes/no)? yes 
it.net.cn: Warning: Permanently added 'it.net.cn,10.3.7.214' (RSA) to the list of known hosts. 
it@it.net.cn's password:  
it@it.net.cn's password: it.net.cn: Permission denied, please try again. 
 
it.net.cn: starting secondarynamenode, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-secondarynamenode-it.net.cn.out 
starting jobtracker, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-jobtracker-it.net.cn.out 
it03: starting tasktracker, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-tasktracker-it03.out 
it02: starting tasktracker, logging to /home/it/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-it-tasktracker-it02.out 
[it@it.net.cn hadoop-1.2.1]$  
需要输入密码

11:检测守护进程


[it@it.net.cn hadoop-1.2.1]$ jps 
4539 JobTracker 
4454 SecondaryNameNode 
4269 NameNode 
4667 Jps 
[it@it.net.cn hadoop-1.2.1]$  


[it@it02 hadoop]$ jps 
2734 TaskTracker 
2815 Jps 
2647 DataNode 
[it@it02 hadoop]$  



[it@it03 hadoop]$ jps 
4070 Jps 
3878 DataNode 
3993 TaskTracker 
[it@it03 hadoop]$   




(责任编辑:IT)
------分隔线----------------------------