Support Questions
Find answers, ask questions, and share your expertise

convert text file into avro file..

convert text file into avro file..

Explorer

Hi All,

i have to convert the text file into avro file

following are the contents of my files

emp_avro.avsc

{
  "type" : "record",
  "name" : "emp_avro",
  "namespace" : "default",
  "fields" : [ {
    "name" : "id",
    "type" : [ "null", "int" ],
    "doc" : "\u0000\u0000\u0000",
    "default" : null
  }, {
    "name" : "name",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "sex",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "state",
    "type" : [ "null", "string" ],
    "default" : null
  } ]
}

 

 

 

 

emp_info.txt

1,amit,male,TX
2,manisg,male,CA
3,hsan,male,AZ
4,shnk,male,FL
5,manisha,female,TX
6,shana,female,AZ
7,sahana,female,CA
8,seema,female,FL

 

 

i am using the following command to convert the text file into avro file

 

java -jar avro-tools.jar fromtext /home/cloudera/Downloads/emp_avro.avsc /home/cloudera/Downloads/emp_info.txt > /home/cloudera/Downloads/emp_avro.avro

 

 

and i got this file

emp_avro.avro

Writing line = 1
Writing line = 20
Writing line = 22
Writing line = 26
Writing line = 16
Writing line = 18
Writing line = 31
Writing line = 33
Writing line = 20
Writing line = 6
Writing line = 20
Writing line = 34
Writing line = 20
Writing line = 6
Writing line = 19
Writing line = 34
Writing line = 20
Writing line = 6
Writing line = 21
Writing line = 34
Writing line = 20
Writing line = 5
Writing line = 1


so this file can't be accesible through hive tables;
any assistance will be helpful

 

3 REPLIES 3

Re: convert text file into avro file..

Master Guru
Are you having issues creating a table backed as AVRO type? What is your CREATE TABLE statement, and how are you loading the file into the table?

For more on creating and using Avro tables in Hive, see http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_hive.html

Re: convert text file into avro file..

Explorer

Hi Harsh,

Thanks for respond,

 

I have no problem in creating the table when backed table file is already in avro type.

But once I have created the table and loaded the data, it is done. After that if I have another text file, which I want to load in the same table, how will I do that?

I was trying to explicitly convert that text file into avro file (using avro tools)  the files contents in my previous question is that one.

 

so here it was not converting the text file into proper avro file which could be loaded by the hive.

 

 

Re: convert text file into avro file..

Super Collaborator

Why not creating an Hive table on top of the text file and then simply use an hive query to load the data into the avro table ?