question Re: read a AVRO file stored in HDFS in Archives of Support Questions (Read Only)

read a AVRO file stored in HDFS

lenovomi — Fri, 05 Feb 2016 19:36:14 GMT

Hi,

I want to read a metadata from avro file stored in HDFS using AVRO api ( https://avro.apache.org/docs/1.4.1/api/java/org/apache/avro/file/DataFileReader.html )

The avro DataFileReader accepts only File objects. Is it somehow possible to read data from file stored on hdfs instead of data stored on local fs?

Thank you

Re: read a AVRO file stored in HDFS

aervits — Fri, 05 Feb 2016 20:04:05 GMT

@John Smithcan you clarify, are you trying to do this programmatically using Java or in a pig script? You can look up schema using avro tools and pass getschema flag Link. I once kept schema in hdfs as XML but it can be any format even json ouut of avro tools and then process new records. Maybe what you suggest is better, to get schema. You can probably try reading it and passing hdfs scheme rather than file:///

Re: read a AVRO file stored in HDFS

lenovomi — Fri, 05 Feb 2016 21:05:45 GMT

Hi,

im trying to do this as part of the JAVA programme.

Re: read a AVRO file stored in HDFS

lenovomi — Fri, 05 Feb 2016 21:08:17 GMT

can you call

avro-tools-1.7.4.jar

within the pig script? and also is it possible to access files stored on HDFS using avro-tools?

Re: read a AVRO file stored in HDFS

aervits — Fri, 05 Feb 2016 21:36:35 GMT

@John Smith those are all valid questions :), I haven't tried as there was never a need. Try it out, post an article! As far as accessing from Pig, not sure that's possible? Again, try it out. You might be able to look at source code and write a UDF that does what avro-tools tries to do, I don't know. By the way, avro-tools coincides with the version of avro, so I'd suggest downloading the latest avro-tools available, which at this moment is 1.8.0.

Re: read a AVRO file stored in HDFS

aervits — Fri, 05 Feb 2016 21:37:56 GMT

@John Smith then look at how to infer schema in Java API. You don't need avro-tools in that case.

Re: read a AVRO file stored in HDFS

lenovomi — Fri, 05 Feb 2016 22:06:37 GMT

im trying to write sample java code... but

https://hadoop.apache.org/docs/r2.6.1/api/org/apache/hadoop/conf/Configuration.html

[root@sandbox deploy-4]# find / -name core-default.xml

[root@sandbox deploy-4]# find / -name core-site..xml

there are no such a files in sandbox. How can i go thru this step?

thanks

Re: read a AVRO file stored in HDFS

lenovomi — Fri, 05 Feb 2016 22:17:03 GMT

I created sample code, it works FINE.

BufferedInputStream inStream = null;
String inputF = "hdfs://CustomerData-20160128-1501807.avro";
org.apache.hadoop.fs.Path inPath = new org.apache.hadoop.fs.Path(inputF);
try {
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://sandbox.hortonworks.com:8020");
FileSystem fs = FileSystem.get(URI.create(inputF), conf);
inStream = new BufferedInputStream(fs.open(inPath));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
DataFileStream reader = new DataFileStream(inStream, new GenericDatumReader());
Schema schema = reader.getSchema();
System.out.println(schema.toString());

Re: read a AVRO file stored in HDFS

lenovomi — Fri, 05 Feb 2016 22:18:50 GMT

you should fix that FORUM website its pain to format text, paste code etc....

Re: read a AVRO file stored in HDFS

aervits — Fri, 05 Feb 2016 23:12:48 GMT

look in /etc/hadoop/conf directory @John Smith

Re: read a AVRO file stored in HDFS

aervits — Fri, 05 Feb 2016 23:13:59 GMT

@John Smith use "code" button to paste code

Re: read a AVRO file stored in HDFS

aervits — Fri, 05 Feb 2016 23:15:17 GMT

@John Smith I edited the answer to format the code.