CentOS6.5 安装 Hadoop

时间:2015-02-28 00:55 来源:www.it.net.cn 作者:IT网

Hadoop实现了一个分布式文件系统（Hadoop Distributed File System），简称HDFS。HDFS有高容错性的特点，并且设计用来部署在低廉的（low-cost）硬件上；而且它提供高吞吐量（high throughput）来访问应用程序的数据，适合那些有着超大数据集（large data set）的应用程序。HDFS放宽了（relax）POSIX的要求，可以以流的形式访问（streaming access）文件系统中的数据。

1、新建Hadoop用户配置免密码登陆

[root@itnetcn ~]# groupadd hadoop

[root@itnetcn ~]# useradd hadoop -g hadoop

[root@itnetcn ~]# passwd hadoop

[root@itnetcn ~]# mkdir /tools

[root@itnetcn ~]# chown hadoop:hadoop /tools/

##Ssh##

[root@itnetcn ~]# su - hadoop

[hadoop@itnetcn ~]$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

+--[ DSA 1024]----+

|BE* |

|.*.= |

|+ o o . . |

|. o . o + |

| . . S o . |

| = o . |

| o o |

| . |

| |

+-----------------+

[hadoop@itnetcn ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

[hadoop@itnetcn ~]$ chmod 0600 ~/.ssh/authorized_keys

2、JAVA.JDK 已安装（请参考:【CentOS JDK-1.8安装】）

[hadoop@itnetcn ~]$ java -version

java version "1.8.0_25"

Java(TM) SE Runtime Environment (build 1.8.0_25-b17)

Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)

3、下载并解包Hadoop

[hadoop@itnetcn ~]$ cd /tools/

[hadoop@itnetcn source]$ wget https://archive.apache.org/dist/hadoop/core/hadoop-2.2.0/hadoop-2.2.0.tar.gz

[hadoop@itnetcn source]$ tar zxf hadoop-2.2.0.tar.gz

[hadoop@itnetcn source]$ ln -s /tools/source/hadoop-2.2.0 /tools/hadoop

4、添加Hadoop环境变量

[hadoop@itnetcn source]$ cat >> ~/.bashrc <<EOF

export HADOOP_PREFIX="/tools/hadoop"

export PATH=\$PATH:\$HADOOP_PREFIX/bin

export PATH=\$PATH:\$HADOOP_PREFIX/sbin

export HADOOP_MAPRED_HOME=\${HADOOP_PREFIX}

export HADOOP_COMMON_HOME=\${HADOOP_PREFIX}

export HADOOP_HDFS_HOME=\${HADOOP_PREFIX}

export YARN_HOME=\${HADOOP_PREFIX}

####hadoop-env####

export JAVA_HOME="/tools/java"

export HADOOP_COMMON_LIB_NATIVE_DIR=\${HADOOP_PREFIX}/lib/native

export HADOOP_OPTS="-Djava.library.path=\$HADOOP_PREFIX/lib"

####yarn-env####

export HADOOP_COMMON_LIB_NATIVE_DIR=\${HADOOP_PREFIX}/lib/native

export HADOOP_OPTS="-Djava.library.path=\$HADOOP_PREFIX/lib"

EOF

[hadoop@itnetcn source]$ source ~/.bashrc

5、变更配置文件

[hadoop@itnetcn source]$ cd $HADOOP_PREFIX/etc/hadoop

[hadoop@itnetcn hadoop]$ vi core-site.xml

#-------------------------------------------------------#

<name>fs.default.name</name>

</property>

</configuration>

#-------------------------------------------------------#

[hadoop@itnetcn hadoop]$ vi hdfs-site.xml

#-------------------------------------------------------#

<name>dfs.namenode.name.dir</name>

<value>file:/hadoop/dfs/name</value>

</property>

<name>dfs.datanode.data.dir</name>

<value>file:/hadoop/dfs/data</value>

</property>

<name>dfs.permissions</name>

<value>false</value>

</property>

</configuration>

#-------------------------------------------------------#

[hadoop@itnetcn hadoop]$ cp mapred-site.xml.template mapred-site.xml

[hadoop@itnetcn hadoop]$ vi mapred-site.xml

#-------------------------------------------------------#

<name>mapreduce.framework.name</name>

</property>

<name>mapred.system.dir</name>

<value>file:/hadoop/mapred/system</value>

</property>

<name>mapred.local.dir</name>

<value>file:/hadoop/mapred/local</value>

</property>

</configuration>

#-------------------------------------------------------#

[hadoop@itnetcn hadoop]$ vi yarn-site.xml

#-------------------------------------------------------#

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

</configuration>

#-------------------------------------------------------#

6、启动HDFS

[hadoop@itnetcn hadoop]$ hdfs namenode -format

15/01/23 23:55:40 INFO namenode.FSImage: Saving image file /tools/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression

15/01/23 23:55:40 INFO namenode.FSImage: Image file /tools/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 198 bytes saved in 0 seconds.

15/01/23 23:55:40 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0

15/01/23 23:55:40 INFO util.ExitUtil: Exiting with status 0

15/01/23 23:55:40 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at itnetcn.me/10.211.55.40

************************************************************/

##Start All(namenode,datanode,yarn)###

[hadoop@itnetcn hadoop]$ cd $HADOOP_PREIFX/sbin

[hadoop@itnetcn sbin]$ start-all.sh

##Jps##

[hadoop@itnetcn sbin]$ jps

2656 Jps

2000 DataNode

2275 ResourceManager

1892 NameNode

2374 NodeManager

2141 SecondaryNameNode

访问HDFS NameNode Web

hadoop-hadoop-cluster

访问NameNode web UI

hadoop-namenode-info

访问 Resource Manager 接口

hadoop-node-manager

测试Hadoop

[hadoop@itnetcn hadoop]$ hdfs dfs -mkdir /user

[hadoop@itnetcn hadoop]$ hdfs dfs -put /tmp /test/logs

hadoop_test

(责任编辑：IT)