Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Twitter data ingestion to Hive giving error.

Highlighted

Twitter data ingestion to Hive giving error.

Rising Star

Twitter data ingestion to Hive giving error as follow.

Failed with exception java.io.IOException:org.apache.avro.AvroRuntimeException: java.io.IOException: Block size invalid or too large for this implementation: -40

Please assist me on this.

Thanks in advance.

4 REPLIES 4
Highlighted

Re: Twitter data ingestion to Hive giving error.

@Raghu Ramamoorthi

Issue seems to be related to below Jira:

https://issues.apache.org/jira/browse/AVRO-1597

It works if you change the command to not use standard output, e.g.: random --schema-file schema.avsc --count 20 test.avro

Thanks and Regards,

Sindhu

Re: Twitter data ingestion to Hive giving error.

Rising Star
@SindhuThanks for the help.The avro file is getting loaded into hive table but while querying it is not giving any data.

For example if i type select * from table; it is throwing the above error.

Thank you

Highlighted

Re: Twitter data ingestion to Hive giving error.

Expert Contributor

Try to reproduce it as mentioned here -

https://qnalist.com/questions/5206657/invalid-file-with-avro-tools-random-and-tojson

Alternate approach is to use another flume source to save data in raw json instead.

Highlighted

Re: Twitter data ingestion to Hive giving error.

Rising Star

@Raghu Ramamoorthi

I was wondering, how did you know the data got loaded in the table without the "select * from" query returning anything? I suggest to

1.create your hive table with proper partitioning or

2. Re-import the avro formated data using flume and make sure the below parameter is set to

agent.sinks.hdfs.serializer=Text

Heres a detailed explanation of how to's in hive with avro:

https://thisdataguy.com/2014/08/04/avro-end-to-end-in-hdfs-part-3-hive/

Don't have an account?
Coming from Hortonworks? Activate your account here