Support Questions

lenovomi · ‎02-05-2016

Hi,

I want to read a metadata from avro file stored in HDFS using AVRO api ( https://avro.apache.org/docs/1.4.1/api/java/org/apache/avro/file/DataFileReader.html )

The avro DataFileReader accepts only File objects. Is it somehow possible to read data from file stored on hdfs instead of data stored on local fs?

Thank you

lenovomi · ‎02-05-2016

I created sample code, it works FINE.

BufferedInputStream inStream = null;
String inputF = "hdfs://CustomerData-20160128-1501807.avro";
org.apache.hadoop.fs.Path inPath = new org.apache.hadoop.fs.Path(inputF);
try {
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://sandbox.hortonworks.com:8020");
FileSystem fs = FileSystem.get(URI.create(inputF), conf);
inStream = new BufferedInputStream(fs.open(inPath));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
DataFileStream reader = new DataFileStream(inStream, new GenericDatumReader());
Schema schema = reader.getSchema();
System.out.println(schema.toString());

View solution in original post

aervits · ‎02-05-2016

@John Smith use "code" button to paste code

aervits · ‎02-05-2016

@John Smith I edited the answer to format the code.

Cloudera Community

Support Questions

read a AVRO file stored in HDFS

how to create and store the avro files in hive tab...

Read from HDFS in R script

Reading ORC files using Mapreduce

How to integrate a Feature Store on Cloudera Machi...

Read the flow file of ExecuteStreamCommand

Reading avro file from HDFS using Scala - Exceptio...

Spark throws "Invalid Sync" Error when trying to R...

How to Recover Accidentally deleted file (with -sk...

Analyze Small FIle in HDFS

Reading CSV File Spark - Issue with Backslash