Support Questions
Find answers, ask questions, and share your expertise

put hdfs (Java error)

put hdfs (Java error)

Explorer

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} span.s1 {font-variant-ligatures: no-common-ligatures} span.Apple-tab-span {white-space:pre}

2017-03-03 07:59:59,090 ERROR [Timer-Driven Process Thread-8] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=931441fa-015a-1000-d89f-19ccb802a271] Failed to write to HDFS due to java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.

2017-03-03 07:59:59,091 ERROR [Timer-Driven Process Thread-8] o.apache.nifi.processors.hadoop.PutHDFS

java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.

at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:139) ~[hadoop-common-2.7.3.jar:na]

at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:180) ~[hadoop-common-2.7.3.jar:na]

at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.getCompressionCodec(AbstractHadoopProcessor.java:398) ~[nifi-hdfs-processors-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]

at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:251) ~[nifi-hdfs-processors-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]

at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_101]

at javax.security.auth.Subject.doAs(Subject.java:360) [na:1.8.0_101]

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678) [hadoop-common-2.7.3.jar:na]

at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:230) [nifi-hdfs-processors-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]

at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]

at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]

at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]

at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]

at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_101]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_101]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_101]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]

at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]

Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found

at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) ~[hadoop-common-2.7.3.jar:na]

at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:132) ~[hadoop-common-2.7.3.jar:na]

... 19 common frames omitted

3 REPLIES 3

Re: put hdfs (Java error)

Expert Contributor

The problem is

Class com.hadoop.compression.lzo.LzoCodec not found

Do you have it enabled? In core-site.xml you need to add com.hadoop.compression.lzo.LzoCodec to io.compression.codecs and set io.compression.codec.lzo.class to com.hadoop.compression.lzo.LzoCodec.

If you have Ambari, see https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-reference/content/configure_core-..., otherwise you have to set these values manually in your core-site.xml.

Re: put hdfs (Java error)

Explorer

I have tried that but still the error is not resolved.Can you suggest another method?

Re: put hdfs (Java error)

Explorer

I resolved temporarily by performing below step.

i removed it from the core-site.xml file “com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec” from “io.compression.codecs” property in “core-site.xml” file.