Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

cdh5 index the lzo file but get the error

cdh5 index the lzo file but get the error

New Contributor

I'm using the CHD5 and GPLEXTRAS, which version is 5.5.2-1.cdh5.5.2.p0.4. When I indexed the lzo files, index files were created, but the job was failed, all map tasks' state were SUCCEEDED. Here is the output of the command: 

 

# hadoop jar /opt/cloudera/parcels/GPLEXTRAS-5.5.2-1.cdh5.5.2.p0.4/lib/hadoop/lib/hadoop-lzo.jarcom.hadoop.compression.lzo.DistributedLzoIndexer /hiveDW/dm_sdk_master/device_install_app_master/day=20160205
16/03/31 10:13:40 INFO lzo.DistributedLzoIndexer: Adding LZO file hdfs://hiveDW/dm_sdk_master/device_install_app_master/day=20160205/001846_0.lzo to indexing list (no index currently exists) 16/03/31 10:13:40 INFO lzo.DistributedLzoIndexer: Adding LZO file hdfs:///hiveDW/dm_sdk_master/device_install_app_master/day=20160205/001847_0.lzo to indexing list (no index currently exists) 16/03/31 10:13:40 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 16/03/31 10:13:43 INFO input.FileInputFormat: Total input paths to process : 1848 16/03/31 10:13:46 INFO mapreduce.JobSubmitter: number of splits:1848 16/03/31 10:13:46 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1459389966841_0002 16/03/31 10:13:46 INFO impl.YarnClientImpl: Submitted application application_1459389966841_0002 16/03/31 10:13:46 INFO mapreduce.Job: The url to track the job: http://jh01-001:8088/proxy/application_1459389966841_0002/ 16/03/31 10:13:46 INFO lzo.DistributedLzoIndexer: Started DistributedIndexer job_1459389966841_0002 with 1848 splits for [/hiveDW/dm_sdk_master/device_install_app_master/day=20160205] 16/03/31 10:13:46 INFO mapreduce.Job: Running job: job_1459389966841_0002 16/03/31 10:13:52 INFO mapreduce.Job: Job job_1459389966841_0002 running in uber mode : false 16/03/31 10:13:52 INFO mapreduce.Job: map 0% reduce 0% 16/03/31 10:14:01 INFO mapreduce.Job: map 3% reduce 0% 16/03/31 10:14:02 INFO mapreduce.Job: map 10% reduce 0% 16/03/31 10:14:03 INFO mapreduce.Job: map 11% reduce 0% 16/03/31 10:14:04 INFO mapreduce.Job: map 12% reduce 0% 16/03/31 10:14:05 INFO mapreduce.Job: map 14% reduce 0% 16/03/31 10:14:06 INFO mapreduce.Job: map 25% reduce 0% 16/03/31 10:14:07 INFO mapreduce.Job: map 31% reduce 0% 16/03/31 10:14:08 INFO mapreduce.Job: map 34% reduce 0% 16/03/31 10:14:09 INFO mapreduce.Job: map 38% reduce 0% 16/03/31 10:14:10 INFO mapreduce.Job: map 41% reduce 0% 16/03/31 10:14:11 INFO mapreduce.Job: map 44% reduce 0% 16/03/31 10:14:12 INFO mapreduce.Job: map 46% reduce 0% 16/03/31 10:14:13 INFO mapreduce.Job: map 48% reduce 0% 16/03/31 10:14:14 INFO mapreduce.Job: map 52% reduce 0% 16/03/31 10:14:15 INFO mapreduce.Job: map 55% reduce 0% 16/03/31 10:14:16 INFO mapreduce.Job: map 59% reduce 0% 16/03/31 10:14:17 INFO mapreduce.Job: map 66% reduce 0% 16/03/31 10:14:18 INFO mapreduce.Job: map 70% reduce 0% 16/03/31 10:14:19 INFO mapreduce.Job: map 74% reduce 0% 16/03/31 10:14:20 INFO mapreduce.Job: map 78% reduce 0% 16/03/31 10:14:21 INFO mapreduce.Job: map 82% reduce 0% 16/03/31 10:14:22 INFO mapreduce.Job: map 86% reduce 0% 16/03/31 10:14:23 INFO mapreduce.Job: map 90% reduce 0% 16/03/31 10:14:24 INFO mapreduce.Job: map 95% reduce 0% 16/03/31 10:14:25 INFO mapreduce.Job: map 97% reduce 0% 16/03/31 10:14:26 INFO mapreduce.Job: map 98% reduce 0% 16/03/31 10:14:29 INFO mapreduce.Job: map 99% reduce 0% 16/03/31 10:14:34 INFO mapreduce.Job: map 100% reduce 0% 16/03/31 10:15:31 INFO mapreduce.Job: Job job_1459389966841_0002 completed successfully 16/03/31 10:15:31 INFO mapreduce.Job: Counters: 31 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=549794394 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=18274272 HDFS: Number of bytes written=17902824 HDFS: Number of read operations=3696 HDFS: Number of large read operations=0 HDFS: Number of write operations=7392 Job Counters Launched map tasks=1848 Data-local map tasks=1841 Rack-local map tasks=7 Total time spent by all maps in occupied slots (ms)=31082602 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=15541301 Total vcore-seconds taken by all map tasks=15541301 Total megabyte-seconds taken by all map tasks=31828584448 Map-Reduce Framework Map input records=2237853 Map output records=2237853 Input split bytes=293832 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=76084 CPU time spent (ms)=4223860 Physical memory (bytes) snapshot=686158041088 Virtual memory (bytes) snapshot=4659832168448 Total committed heap usage (bytes)=1866035888128 File Input Format Counters Bytes Read=17980440 File Output Format Counters Bytes Written=0 16/03/31 10:15:31 ERROR lzo.DistributedLzoIndexer: DistributedIndexer job_1459389966841_0002 failed. 1848 out of 1848 mappers failed.


 Why the index files created successed, but the job got the error...3Q 

2 REPLIES 2

Re: cdh5 index the lzo file but get the error

New Contributor

Maybe the host's conf files was corrupted. After I add this host into the cluster again, everything is ok. 

Re: cdh5 index the lzo file but get the error

Expert Contributor

good to hear that the problem is solved after re-adding the node. I agree that would seem to indicate a problem with that node's configuration. 

 

1848 out of 1848 mappers failed.

For your own future reference, you would need to look at the map task logs to see if they hold any clues to why the job failed.