Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

put hdfs (Java error)

Highlighted

put hdfs (Java error)

New Contributor

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} span.s1 {font-variant-ligatures: no-common-ligatures} span.Apple-tab-span {white-space:pre}

2017-03-03 07:59:59,090 ERROR [Timer-Driven Process Thread-8] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=931441fa-015a-1000-d89f-19ccb802a271] Failed to write to HDFS due to java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.

2017-03-03 07:59:59,091 ERROR [Timer-Driven Process Thread-8] o.apache.nifi.processors.hadoop.PutHDFS

java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.

at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:139) ~[hadoop-common-2.7.3.jar:na]

at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:180) ~[hadoop-common-2.7.3.jar:na]

at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.getCompressionCodec(AbstractHadoopProcessor.java:398) ~[nifi-hdfs-processors-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]

at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:251) ~[nifi-hdfs-processors-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]

at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_101]

at javax.security.auth.Subject.doAs(Subject.java:360) [na:1.8.0_101]

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678) [hadoop-common-2.7.3.jar:na]

at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:230) [nifi-hdfs-processors-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]

at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]

at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]

at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]

at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]

at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_101]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_101]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_101]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]

at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]

Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found

at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) ~[hadoop-common-2.7.3.jar:na]

at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:132) ~[hadoop-common-2.7.3.jar:na]

... 19 common frames omitted

3 REPLIES 3

Re: put hdfs (Java error)

Expert Contributor

The problem is

Class com.hadoop.compression.lzo.LzoCodec not found

Do you have it enabled? In core-site.xml you need to add com.hadoop.compression.lzo.LzoCodec to io.compression.codecs and set io.compression.codec.lzo.class to com.hadoop.compression.lzo.LzoCodec.

If you have Ambari, see https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-reference/content/configure_core-..., otherwise you have to set these values manually in your core-site.xml.

Re: put hdfs (Java error)

New Contributor

I have tried that but still the error is not resolved.Can you suggest another method?

Re: put hdfs (Java error)

New Contributor

I resolved temporarily by performing below step.

i removed it from the core-site.xml file “com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec” from “io.compression.codecs” property in “core-site.xml” file.

Don't have an account?
Coming from Hortonworks? Activate your account here