运行wordcount时显示Could not obtain block
时间:2014-11-04 18:32 来源:linux.it.net.cn 作者:it
该文章接上面hadoop运行wordcount时卡住不动，接着下面
hadoop@ubuntu118:~/hadoop-1.0.2$ bin/hadoop dfsadmin -safemode leave
Warning: $HADOOP_HOME is deprecated.
Safe mode is OFF
hadoop@ubuntu118:~/hadoop-1.0.2$ bin/hadoop jar hadoop-examples-1.0.2.jar wordcount file fileout66
Warning: $HADOOP_HOME is deprecated.
d****hdfs://192.168.0.118:9000/user/hadoop/file
12/09/10 18:35:52 INFO input.FileInputFormat: Total input paths to process : 2
12/09/10 18:35:53 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/09/10 18:35:53 WARN snappy.LoadSnappy: Snappy native library not loaded
12/09/10 18:35:53 INFO mapred.JobClient: Running job: job_201209101835_0001
12/09/10 18:35:54 INFO mapred.JobClient: map 0% reduce 0%
12/09/10 18:36:17 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000000_0, Status : FAILED
java.io.IOException: Could not obtain block: blk_-7132067374412134968_1005 file=/user/hadoop/file/file2
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2266)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2060)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2221)
at java.io.DataInputStream.read(DataInputStream.java:83)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
12/09/10 18:36:20 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000001_0, Status : FAILED
java.io.IOException: Could not obtain block: blk_-4697097477082743821_1006 file=/user/hadoop/file/file1
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2266)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2060)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2221)
at java.io.DataInputStream.read(DataInputStream.java:83)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
12/09/10 18:36:35 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000000_1, Status : FAILED
java.io.IOException: Could not obtain block: blk_-7132067374412134968_1005 file=/user/hadoop/file/file2
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2266)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2060)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2221)
at java.io.DataInputStream.read(DataInputStream.java:83)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
12/09/10 18:36:35 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000001_1, Status : FAILED
java.io.IOException: Could not obtain block: blk_-4697097477082743821_1006 file=/user/hadoop/file/file1
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2266)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2060)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2221)
at java.io.DataInputStream.read(DataInputStream.java:83)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
12/09/10 18:36:50 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000000_2, Status : FAILED
java.io.IOException: Could not obtain block: blk_-7132067374412134968_1005 file=/user/hadoop/file/file2
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2266)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2060)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2221)
at java.io.DataInputStream.read(DataInputStream.java:83)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
12/09/10 18:36:50 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000001_2, Status : FAILED
java.io.IOException: Could not obtain block: blk_-4697097477082743821_1006 file=/user/hadoop/file/file1
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2266)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2060)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2221)
at java.io.DataInputStream.read(DataInputStream.java:83)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
12/09/10 18:37:12 INFO mapred.JobClient: Job complete: job_201209101835_0001
12/09/10 18:37:12 INFO mapred.JobClient: Counters: 6
12/09/10 18:37:12 INFO mapred.JobClient: Job Counters
12/09/10 18:37:12 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=115005
12/09/10 18:37:12 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
12/09/10 18:37:12 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
12/09/10 18:37:12 INFO mapred.JobClient: Launched map tasks=8
12/09/10 18:37:12 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
12/09/10 18:37:12 INFO mapred.JobClient: Failed map tasks=1
执行文件file里面文件权限问题，查看里面的文件后发现其中的文件没有执行权限，但修改后发现还是没有执行权限。无奈执行重新上传文件上去hdfs，之后运行正常。
(责任编辑：IT)
该文章接上面hadoop运行wordcount时卡住不动，接着下面
hadoop@ubuntu118:~/hadoop-1.0.2$ bin/hadoop dfsadmin -safemode leave
Warning: $HADOOP_HOME is deprecated.
Safe mode is OFF
hadoop@ubuntu118:~/hadoop-1.0.2$ bin/hadoop jar hadoop-examples-1.0.2.jar wordcount file fileout66
Warning: $HADOOP_HOME is deprecated.
d****hdfs://192.168.0.118:9000/user/hadoop/file
12/09/10 18:35:52 INFO input.FileInputFormat: Total input paths to process : 2
12/09/10 18:35:53 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/09/10 18:35:53 WARN snappy.LoadSnappy: Snappy native library not loaded
12/09/10 18:35:53 INFO mapred.JobClient: Running job: job_201209101835_0001
12/09/10 18:35:54 INFO mapred.JobClient: map 0% reduce 0%
12/09/10 18:36:17 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000000_0, Status : FAILED
java.io.IOException: Could not obtain block: blk_-7132067374412134968_1005 file=/user/hadoop/file/file2
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2266)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2060)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2221)
at java.io.DataInputStream.read(DataInputStream.java:83)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
12/09/10 18:36:20 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000001_0, Status : FAILED
java.io.IOException: Could not obtain block: blk_-4697097477082743821_1006 file=/user/hadoop/file/file1
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2266)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2060)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2221)
at java.io.DataInputStream.read(DataInputStream.java:83)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
12/09/10 18:36:35 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000000_1, Status : FAILED
java.io.IOException: Could not obtain block: blk_-7132067374412134968_1005 file=/user/hadoop/file/file2
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2266)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2060)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2221)
at java.io.DataInputStream.read(DataInputStream.java:83)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
12/09/10 18:36:35 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000001_1, Status : FAILED
java.io.IOException: Could not obtain block: blk_-4697097477082743821_1006 file=/user/hadoop/file/file1
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2266)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2060)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2221)
at java.io.DataInputStream.read(DataInputStream.java:83)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
12/09/10 18:36:50 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000000_2, Status : FAILED
java.io.IOException: Could not obtain block: blk_-7132067374412134968_1005 file=/user/hadoop/file/file2
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2266)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2060)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2221)
at java.io.DataInputStream.read(DataInputStream.java:83)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
12/09/10 18:36:50 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000001_2, Status : FAILED
java.io.IOException: Could not obtain block: blk_-4697097477082743821_1006 file=/user/hadoop/file/file1
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2266)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2060)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2221)
at java.io.DataInputStream.read(DataInputStream.java:83)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
12/09/10 18:37:12 INFO mapred.JobClient: Job complete: job_201209101835_0001
12/09/10 18:37:12 INFO mapred.JobClient: Counters: 6
12/09/10 18:37:12 INFO mapred.JobClient: Job Counters
12/09/10 18:37:12 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=115005
12/09/10 18:37:12 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
12/09/10 18:37:12 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
12/09/10 18:37:12 INFO mapred.JobClient: Launched map tasks=8
12/09/10 18:37:12 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
12/09/10 18:37:12 INFO mapred.JobClient: Failed map tasks=1
执行文件file里面文件权限问题，查看里面的文件后发现其中的文件没有执行权限，但修改后发现还是没有执行权限。无奈执行重新上传文件上去hdfs，之后运行正常。
(责任编辑：IT) |