Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hadoop streaming create ORC+SNAPPY

hadoop streaming create ORC+SNAPPY

Explorer

HIVE can create ORC, but how hadoop streaming create ORC

i try streaming -outputformat org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat

But it does not work

java.lang.RuntimeException: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.io.NullWritable

how can i do?

4 REPLIES 4
Highlighted

Re: hadoop streaming create ORC+SNAPPY

Expert Contributor

@Fan WeiWen, Could you provide more details on how you "try streaming"? If you stream into table, please let us see "create table" statement

Highlighted

Re: hadoop streaming create ORC+SNAPPY

Explorer

yarn jar /usr/hdp/2.3.4.7-4/hadoop-mapreduce/hadoop-streaming.jar \ -libjars /usr/hdp/2.3.4.7-4/hive/lib/hive-exec.jar \ -mapper test_m.py \ -reducer test_r.py \ -input /user/hive/warehouse/test/ \ -output /user/mapred/test/ \ -numReduceTasks 2 \ -file test_m.py \ -file test_r.py \ -outputformat org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat

This ORC is don't work but remove

-outputformat org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat

it work

so ORC

What parameters are required?

Highlighted

Re: hadoop streaming create ORC+SNAPPY

Explorer

Who can help?

Highlighted

Re: hadoop streaming create ORC+SNAPPY

Expert Contributor
@Fan WeiWen

Usually ORC output format is being used along for Hive tables ORC formatted. But if you really want to use ORC output format from pure M/R job, you need to provide a schema. ORC is columnar storage format. In order to build those columns, output formatter needs to know how it should organize the data. First thing I noted from provided command line is missing attribute "orc.mapred.output.schema". For more information please refer to https://orc.apache.org/docs/mapreduce.html

Don't have an account?
Coming from Hortonworks? Activate your account here