hadoop基本命令

1.start-all.sh开启线程 hadoop fs -ls hdfs://cloud4:9000/user (cloud4表示主机名9000代表端口号/就代表根目录 /user就是user文件夹) hadoop fs -ls /user（hdfs://cloud4:9000可省略） 2. -ls/-lsr查看目录下的文件信息 hadoop fs -ls / 查看根目录下文件...

本地的文件上传到hadoop注意编码

使用hadoop的时候要注意，本地的文件上传到hadoop之前， 看一下本地文件的编码（要使用GBK）编码， 否则在做reduce处理时，文本的第一行处理可能会有问题...

JobConf详解Job

...

hadoop配置多个输入路径，某个路径配置为空时

hadoop配置多个输入路径，某个路径配置为空时，运算时会把路径默认为用户目录下 即： user/用户名/null...

ERROR security.UserGroupInformation: Priviledge...

Failed to set permissions of path问题 参考文献：https://issues.apache.org/jira/browse/HADOOP-8089 错误信息如下： ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop cause:java.io.IOException Failed to set permissions...

hadoop，hbase，hive安装全记录(转)

操作系统：CentOS 5.5 Hadoop：hadoop-0.20.203.0 jdk1.7.0_01 namenode主机名:master，namenode的IP:10.10.102.15 datanode主机名:slave1，datanode的IP:10.10.106.8 datanode主机名:slave2，datanode的IP:10.10.106.9 一、hadoop 安装 1、建立用户 useradd...

Windows平台Hadoop编译、安装、配置与运行(转)

http://www.srccodes.com/p/article/38/build-install-configure-run-apache-hadoop-2.2.0-microsoft-windows-os...

Mapreduce实例-分组排重（group by distinct）

1 public class GroupComparator implements RawComparatorMyBinaryKey { 2 3 @Override 4 public int compare(MyBinaryKey o1, MyBinaryKey o2) { 5 return o1.toString().compareTo(o2.toString()); 6 } 7 8 @Override 9 public int compare( byte [] b1,...

Mapreduce实例-Top Key

1 public class TopK extends Configured implements Tool { 2 3 public static class TopKMapper extends MapperObject, Text, NullWritable, LongWritable { 4 5 public static final int K = 100 ; 6 private TreeMapLong, Long tm = new TreeMapLong, Lo...

hadoop MultipleInputs fails with ClassCastException （get fileName）

Following up on my comment, the Javadocs for TaggedInputSplit confirms that you are probably wrongly casting the input split to a FileSplit: /** * An {@link InputSplit} that tags another InputSplit with extra data for use * by {@link Deleg...