Mapreduce实例-Top Key

1 public class TopK extends Configured implements Tool { 2 3 public static class TopKMapper extends MapperObject, Text, NullWritable, LongWritable { 4 5 public static final int K = 100 ; 6 private TreeMapLong, Long tm = new TreeMapLong, Lo...

hadoop MultipleInputs fails with ClassCastException （get fileName）

Following up on my comment, the Javadocs for TaggedInputSplit confirms that you are probably wrongly casting the input split to a FileSplit: /** * An {@link InputSplit} that tags another InputSplit with extra data for use * by {@link Deleg...

hadoop safemode error

1、safemode bin / hadoopfs - put ./ inputinput put : org . apache . hadoop . hdfs . server . namenode . SafeModeException : Cannotcreatedirectory / user / root / input . Namenodeisinsafemode . 解决方法： NameNode在启动的时候首先进入安全模...

flume hdfs一些简单配置记忆

1 ############################################ 2 # producer config 3 ############################################ 4 #agent section 5 producer.sources = s 6 producer.channels = c c1 c2 7 producer.sinks = r h es 8 9 #source section 10 produc...

Speculative Execution in Hadoop

所谓的推测执行，就是当所有task都开始运行之后，Job Tracker会统计所有任务的平均进度，如果某个task所在的task node机器配置比较低或者CPU load很高（原因很多），导致任务执行比总体任务的平均执行要慢，此时Job Tracker会启动一个新的任务（duplicate ta...

Hadoop: Add third-party libraries to MapReduce job

Anybody working with Hadoop should have already faced a same common issue: How to add third-party libraries to your MapReduce job. Add libjars option The first solution, maybe the most common one, consists on adding libraries using -libjar...

在hadoop上运行java文件

hadoop 2.x版本 编译： javac -d . -classpath /usr/lib/hadoop/hadoop-common-2.2.0.2.0.6.0-102.jar TestGetPathMark.java (classpath多个jar包用分号分隔 /opt/1.jar:/opt/2.jar) 在com的同级目录上建立manifest.mf 在里面写上Main-Class: com.test.path.m...

MapReduce三种路径输入

目前为止知道MapReduce有三种路径输入方式。 1、第一种是通过一下方式输入： FileInputFormat.addInputPath(job, new Path(args[0])); FileInputFormat.addInputPath(job, new Path(args[1])); FileInputFormat.addInputPath(job, new Path(args[2])); FileIn...

使用hadoop eclipse plugin提交Job并添加多个第三方jar

通过 conf.set(tmpjars, jars); 可以设置第三方jar，之前一直只是添加一个jar，运行OK，今天打算添加多个jar的时候发现mapreduce在运行时找不到 class（ClassNotFoundException），跟踪代码发现jar文件的确上传到了HDFS中，所以甚是无解，后来上传jar到 hdfs...

hadoop lzo应用

几种压缩方式对比： LZO example: https://github.com/twitter/hadoop-lzo/blob/master/src/test/java/com/hadoop/mapreduce/TestLzoTextInputFormat.java 给lzo文件加索引的目的是为了让lzo支持 splitable，这样hadoop可以并行处理,所以这一步很关键，生成...