Member since
01-07-2016
89
Posts
20
Kudos Received
6
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
9641 | 02-05-2016 02:17 PM | |
10884 | 02-05-2016 12:56 AM | |
3973 | 01-29-2016 03:24 AM | |
1892 | 01-20-2016 03:52 PM | |
1663 | 01-20-2016 08:48 AM |
02-05-2016
02:17 PM
2 Kudos
I created sample code, it works FINE. BufferedInputStream inStream = null;
String inputF = "hdfs://CustomerData-20160128-1501807.avro";
org.apache.hadoop.fs.Path inPath = new org.apache.hadoop.fs.Path(inputF);
try {
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://sandbox.hortonworks.com:8020");
FileSystem fs = FileSystem.get(URI.create(inputF), conf);
inStream = new BufferedInputStream(fs.open(inPath));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
DataFileStream reader = new DataFileStream(inStream, new GenericDatumReader());
Schema schema = reader.getSchema();
System.out.println(schema.toString());
... View more
02-05-2016
02:06 PM
im trying to write sample java code... but https://hadoop.apache.org/docs/r2.6.1/api/org/apache/hadoop/conf/Configuration.html [root@sandbox deploy-4]# find / -name core-default.xml
[root@sandbox deploy-4]# find / -name core-site..xml there are no such a files in sandbox. How can i go thru this step? thanks
... View more
02-05-2016
01:08 PM
can you call avro-tools-1.7.4.jar within the pig script? and also is it possible to access files stored on HDFS using avro-tools?
... View more
02-05-2016
11:36 AM
1 Kudo
Hi, I want to read a metadata from avro file stored in HDFS using AVRO api ( https://avro.apache.org/docs/1.4.1/api/java/org/apache/avro/file/DataFileReader.html ) The avro DataFileReader accepts only File objects. Is it somehow
possible to read data from file stored on hdfs instead of data stored on
local fs? Thank you
... View more
Labels:
- Labels:
-
Apache Hadoop
02-05-2016
09:03 AM
1 Kudo
this is odd: when i do grunt> b = limit sourceData 5; grunt>dump b; i works for me also, when i dont limit result set .. .and just executing dump sourceData; im occurring same error.
... View more
02-05-2016
08:43 AM
1 Kudo
then what kind of issue with environment it could be? I only executed menitoned command, nothing else.
... View more
02-05-2016
12:59 AM
you can find dataset here: https://drive.google.com/file/d/0B6RZ_9vVuTEcTHllU1dIR2VBY1E/view?usp=sharing \\thank you
... View more
02-05-2016
12:56 AM
1 Kudo
fyi https://issues.apache.org/jira/browse/PIG-4793 org.apache.pig.piggybank.storage.avro.AvroStorage is Deprecated, use AvroStorage('schema', '-d') This works.
... View more
02-05-2016
12:39 AM
1 Kudo
needles to say, this is insane. Yes, grunt by -x mapreduce, i tried -x tez but: 2016-02-05 00:37:42,172 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias sourceDataDetails at logfile: /home/hdfs/pig_1454632554431.log privileges are correct:
drwxr-xr-x - hdfs hdfs 0 2016-02-04 23:55 /src
delimiter is is ; any idea?
... View more