Hi, Is it possible to sqoop from sql db to edge node/unix local path, instaed of HDFS?? Regards Mamta Chawal ... View more

Hi, I have a shell script in HDFS echo.sh, which contains echo Hello I want to execute the shell script from Unix shell as below. sh <HDFS_SCRIPT_PATH>/echo.sh fails saying no such file. Kindly let me know how can i do it. Regards Mamta Chawla ... View more

Hi,I have very simple AVSC file, And I generated the Avro using GitHub Code, xml to Avro converter, https://github.com/elodina/xml-avro/tree/master/src/ly/stealth/xmlavro/Converter.java But when I query the table I get below error. Avro - java.io.IOException: java.io.IOException: Not a data file. I can see the avro file inside the table folder. Kindly let me know what I am missing?? Below is my AVSC- CREATE TABLE embedded COMMENT "just drop the schema right into the HQL" ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' LOCATION 'hdfs://csaa-aap-qa/apps/hive/warehouse/reservemodel.db/embedded' TBLPROPERTIES ( 'avro.schema.literal'='{ "fields": [ { "name": "BillDate", "source": "element BillDate", "type": "string" }, { "name": "BillTime", "source": "element BillTime", "type": "string" }, { "name": "Remit_CompanyName", "source": "element Remit_CompanyName", "type": "string" }, { "name": "Remit_Addr", "source": "element Remit_Addr", "type": "string" }, { "name": "Remit_CityStZip", "source": "element Remit_CityStZip", "type": "string" }, { "name": "Remit_Phone", "source": "element Remit_Phone", "type": "string" }, { "name": "Remit_Fax", "source": "element Remit_Fax", "type": "string" }, { "name": "Remit_TaxID", "source": "element Remit_TaxID", "type": "string" }, { "name": "Previous_Balance", "source": "element Previous_Balance", "type": "string" }, { "name": "others", "type": { "type": "map", "values": "string" } } ], "name": "MetroBillType", "namespace": "ly.stealth.xmlavro", "protocol": "xml", "type": "record" } ') ; And the XML is. <?xml version="1.0" encoding="UTF-8" ?> <MetroBill xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" xs:noNamespaceSchemaLocation="Metrobill.xsd" > <BillDate>02/29/2016</BillDate> <BillTime>18:49:05</BillTime> <Remit_CompanyName>METROPOLITAN REPORTING BUREAU</Remit_CompanyName> <Remit_Addr>P.O. BOX 926, WILLIAM PENN ANNEX</Remit_Addr> <Remit_CityStZip>PHILADELPHIA, PA 19105-0926</Remit_CityStZip> <Remit_Phone>(800) 245-6686</Remit_Phone> <Remit_Fax>(800) 343-9047</Remit_Fax> <Remit_TaxID>23-1879730</Remit_TaxID> <Previous_Balance>1663</Previous_Balance> </MetroBill> The Avro genrated is with special chars- 02/29/201618:49:05:METROPOLITAN REPORTING BUREAU@P.O. BOX 926, WILLIAM PENN ANNEX6PHILADELPHIA, PA 19105-0926(800) 245-6686(800) 343-904723-18797301663 Regards Mamta Chawla ... View more

I have an avsc like below, in which record type is under record type. When I am trying to import in hive, I get table created with columns error schema and all.. Kindly suggest me how to import such avsc in hive. CREATE TABLE metro ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' TBLPROPERTIES ('avro.schema.literal'='{ "namespace": "ly.stealth.xmlavro", "protocol": "xml", "type" : "record", "name" : "MetroBillType", "fields" : [ { "name" : "BillDate", "type" : "string" }, { "name" : "BillTime", "type" : "string" }, { "name" : "Remit_CompanyName", "type" : "string" }, { "name" : "Remit_Addr", "type" : "string" }, { "name" : "Remit_CityStZip", "type" : "string" }, { "name" : "Remit_Phone", "type" : "string" }, { "name" : "Remit_Fax", "type" : "string" }, { "name" : "Remit_TaxID", "type" : "string" }, { "name" : "BillAcct_Break", "type" : { "type" : "record", "name" : "BillAcct_BreakType", "fields" : [ { "name" : "BillAcct", "type" : "string" }, { "name" : "Invoice_Number", "type" : "int" }, { "name" : "Acct_Break", "type" : { "type" : "record", "name" : "Acct_BreakType", "fields" : [ { "name" : "Acct", "type" : "string" }, { "name" : "Items", "type" : { "type" : "record", "name" : "ItemsType", "fields" : [ { "name" : "Item", "type" : { "type" : "array", "items" : { "type" : "record", "name" : "ItemType", "fields" : [ { "name" : "Account", "type" : "string" }, { "name" : "Claim_Number", "type" : "string" }, { "name" : "Insured_Name", "type" : "string" }, { "name" : "Price", "type" : "float" }, { "name" : "Control_Number", "type" : "int" }, { "name" : "State", "type" : "string" }, { "name" : "Report_Type_Code", "type" : "string" }, { "name" : "Report_Type_Desc", "type" : "string" }, { "name" : "Policy_Number", "type" : "string" }, { "name" : "Date_of_Loss", "type" : "string" }, { "name" : "Date_Received", "type" : "string" }, { "name" : "Date_Closed", "type" : "string" }, { "name" : "Days_to_Fill", "type" : "int" }, { "name" : "Police_Dept", "type" : "string" }, { "name" : "Attention", "type" : "string" }, { "name" : "RequestID", "type" : "int" }, { "name" : "ForceDup", "type" : "string" }, { "name" : "BillAcct", "type" : "string" }, { "name" : "BillCode", "type" : "string" } ] } } } ] } }, { "name" : "Acct_Total", "type" : "float" }, { "name" : "Acct_Count", "type" : "int" } ] } }, { "name" : "Bill_Total", "type" : "float" }, { "name" : "Bill_Count", "type" : "int" } ] } }, { "name" : "Previous_Balance", "type" : "int" } ] }'); Thanks Mamta ... View more

Hi, I want to calculate no. of business days between two dates using hive queries. How can I do it?? Thanks Mamta ... View more