当前位置: > Linux集群 > Hadoop >

CentOS6.4下搭建hadoop2.2(64bit)注意事项

时间:2015-07-12 02:00来源:cnblogs.com 作者:IT

注:本文针对64位机器,32bit课直接tar -zxvf hadoop-2.2.0.tar.gz 解压配置即可。

Step1:安装jdk(6以上版本)

Step2:下载hadoop--->http://mirror.esocc.com/apache/hadoop/common/hadoop-2.2.0/选择hadoop-2.2.0-src.tar.gz 

进行源码编译安装。

  为什么选择源码编译安装而不是选择hadoop-2.2.0.tar.gz 安装,此时你可以使用如下命令查看

 
[root@cloud003 hduser]# file hadoop-2.2.0/lib/native/*           ##看到没官方提供的是32bit的,可我们的机器是64bit的所以我们必须使用源代码编译安装
hadoop-2.2.0/lib/native/libhadoop.a:        current ar archive
hadoop-2.2.0/lib/native/libhadooppipes.a:   current ar archive
hadoop-2.2.0/lib/native/libhadoop.so:       symbolic link to `libhadoop.so.1.0.0'
hadoop-2.2.0/lib/native/libhadoop.so.1.0.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped
hadoop-2.2.0/lib/native/libhadooputils.a:   current ar archive
hadoop-2.2.0/lib/native/libhdfs.a:          current ar archive
hadoop-2.2.0/lib/native/libhdfs.so:         symbolic link to `libhdfs.so.0.0.0'
hadoop-2.2.0/lib/native/libhdfs.so.0.0.0:   ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped
 

Step3:编译安装之后,在此查看

 
[root@cloud003 hduser]# file /usr/local/hadoop-2.2.0/lib/native/*         #看到没64bit了
/usr/local/hadoop-2.2.0/lib/native/libhadoop.a:        current ar archive
/usr/local/hadoop-2.2.0/lib/native/libhadooppipes.a:   current ar archive
/usr/local/hadoop-2.2.0/lib/native/libhadoop.so:       symbolic link to `libhadoop.so.1.0.0'
/usr/local/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
/usr/local/hadoop-2.2.0/lib/native/libhadooputils.a:   current ar archive
/usr/local/hadoop-2.2.0/lib/native/libhdfs.a:          current ar archive
/usr/local/hadoop-2.2.0/lib/native/libhdfs.so:         symbolic link to `libhdfs.so.0.0.0'
/usr/local/hadoop-2.2.0/lib/native/libhdfs.so.0.0.0:   ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
[root@cloud003 hduser]# 
 

Step4:加入环境变量以及更改配置

 
#加入HADOOP_HOME
[root@cloud003 hduser]# vi ~/.bash_profile 
## export JAVA_HOME JDK ##
export JAVA_HOME="/usr/java/jdk1.6.0_45"
## export JAVA_HOME JRE ##
##export JAVA_HOME="/usr/java/jre1.6.0_45"
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
export ANT_HOME=/usr/local/apache-ant-1.9.2
export FINDBUGS_HOME=/usr/local/findbugs-2.0.2
export HADOOP_HOME=/usr/local/hadoop-2.2.0
PATH=$JAVA_HOME/bin:$ANT_HOME/bin:$FINDBUGS_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export PATH
 

使添加的变量生效

[root@cloud003 hduser]# source ~/.bash_profile 

添加目录结构:

数据和编辑文件 /usr/local/hadoop-2.2.0/dfs/data
/usr/local/hadoop-2.2.0/dfs/name
存放数据 /usr/local/hadoop-2.2.0/mapred/local
/usr/local/hadoop-2.2.0/mapred/system

修改配置文件:

vi etc/hadoop/hdfs-site.xml             #参考官方单节点配置文档http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/SingleCluster.html
vi etc/hadoop/core-site.xml 
cp etc/hadoop/mapred-site.xml.template etc/hadoop/mapred-site.xml
vi etc/hadoop/mapred-site.xml
vi etc/hadoop/yarn-site.xml 

将hadoop-env.sh 中的JAVA_HOME制定到你系统的jdk路径

 
[root@cloud003 hadoop-2.2.0]# vi etc/hadoop/hadoop-env.sh 

# Copyright 2011 The Apache Software Foundation
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME.  All others are
# optional.  When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.

# The java implementation to use. 
export JAVA_HOME=/usr/java/jdk1.6.0_45           #修改为你的JDK路径

# The jsvc implementation to use. Jsvc is required to run secure datanodes.
#export JSVC_HOME=${JSVC_HOME}

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}
 

上述改完之后:格式化namenode

 
[root@cloud003 hadoop-2.2.0]# hdfs namenode -format
.....................
.....................
.....................
[root@cloud003 hadoop-2.2.0]# hadoop-daemon.sh start namenode
starting namenode, logging to /usr/local/hadoop-2.2.0/logs/hadoop-root-namenode-cloud003.out
[root@cloud003 hadoop-2.2.0]# hadoop-daemon.sh start datanode
starting datanode, logging to /usr/local/hadoop-2.2.0/logs/hadoop-root-datanode-cloud003.out
[root@cloud003 hadoop-2.2.0]# yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /usr/local/hadoop-2.2.0/logs/yarn-root-resourcemanager-cloud003.out
[root@cloud003 hadoop-2.2.0]# yarn-daemon.sh start manager
starting manager, logging to /usr/local/hadoop-2.2.0/logs/yarn-root-manager-cloud003.out
[root@cloud003 hadoop-2.2.0]# start-yarn.sh
starting yarn daemons
resourcemanager running as process 870. Stop it first.
The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is 45:9f:56:fd:79:b8:be:18:86:3e:3f:a7:bf:90:2f:6c.
Are you sure you want to continue connecting (yes/no)? y
Please type 'yes' or 'no': yes
localhost: Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
localhost: starting nodemanager, logging to /usr/local/hadoop-2.2.0/logs/yarn-root-nodemanager-cloud003.out
[root@cloud003 hadoop-2.2.0]# jps
1253 NodeManager
870 ResourceManager
1351 Jps
31069 DataNode
725 NameNode
 

测试:

查看resourcemanager上cluster运行状态(hadoop资源管理页面)
    Firefox进入:http://localhost:8088(node1为resourcemanager所在节点)

查看节点状态:

     FireFox进入: http://localhost:50070

(责任编辑:IT)
------分隔线----------------------------