Member since
06-14-2018
20
Posts
0
Kudos Received
0
Solutions
11-14-2018
12:43 AM
@Jay Kumar SenSharmaI can see the error as Exception caught in <EventBatchPublisher(Thread-5435, started daemon 140041392408320)> Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/cwlogs/threads.py", line 58, in run self._run() File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1403, in _run self._publish_event_batch() File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1210, in _publish_event_batch self.sequence_token = self._put_log_events(self.event_batch) File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1248, in _put_log_events response = self.logs_service.put_log_events(**params) File "/root/.local/lib/python2.7/site-packages/botocore/client.py", line 320, in _api_call return self._make_api_call(operation_name, kwargs) File "/root/.local/lib/python2.7/site-packages/botocore/client.py", line 623, in _make_api_call raise error_class(parsed_response, operation_name)
... View more
11-14-2018
12:43 AM
I am able to see something like this p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
span.s1 {font-variant-ligatures: no-common-ligatures} Exception caught in <EventBatchPublisher(Thread-5435, started daemon 140041392408320)> Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/cwlogs/threads.py", line 58, in run self._run() File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1403, in _run self._publish_event_batch() File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1210, in _publish_event_batch self.sequence_token = self._put_log_events(self.event_batch) File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1248, in _put_log_events response = self.logs_service.put_log_events(**params) File "/root/.local/lib/python2.7/site-packages/botocore/client.py", line 320, in _api_call return self._make_api_call(operation_name, kwargs) File "/root/.local/lib/python2.7/site-packages/botocore/client.py", line 623, in _make_api_call raise error_class(parsed_response, operation_name)
... View more
11-13-2018
11:59 PM
hi @Jay Kumar SenSharma Thanks for the suggestions. There were few build failures. I did correct them AND after the steps atlas got successfullt installed. p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff; min-height: 13.0px}
span.s1 {font-variant-ligatures: no-common-ligatures} ./bin/atlas_start.py configured for local elasticsearch. elasticsearch started. starting atlas on host localhost starting atlas on port 21000 .... Apache Atlas Server started!!! However when i am trying to use curl to check the atlas, its throwing an following Error: <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 503 </title> </head> <body> <h2>HTTP ERROR: 503</h2> <p>Problem accessing /api/atlas/admin/version. Reason: <pre>Service Unavailable</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.14.v20161028</a><hr/> </body> </html> I was checking the pom.xml file, but wasn't sure what to add or delete from the properties
... View more
11-13-2018
11:31 PM
hi @Jay Kumar SenSharma I dont have any HDP setup. I am installing it on single node ec2 instance. moreover in the distro, the target isnt created. I am definetly missing something
... View more
11-13-2018
11:05 PM
Hi , I have installed apache atlas on ec2 instance on single node. I did follow the steps as listed in the below link: http://atlas.apache.org/InstallationSteps.html I am unable to find te atlas_start.py to start the script. (I did find the script in distro/src/bin/) but couldnt run the script. I am expecting the file to be in distro/target. Could anyone help me if I am missing something?? Thanks,
... View more
Labels:
- Labels:
-
Apache Atlas
07-16-2018
02:44 AM
Hi @Vinicius Higa Murakami , Thanks for the reply. I was trying to use inline function with SELECT statement. but its throwing an Error. select strngname, inline(Customers.customer_details) from default.customer_details; BUT its throwing below Error: SemanticException [Error 10081]: UDTf's are not supported outside the SELECT clause , nor nested in expressions: Could you please help me with this. Thanks,
... View more
07-13-2018
08:30 PM
Hi All, I did load the below XML file into HIVE table” test_table.customer_details”
with a schema Hive> Create
table test_table.customer_details ( Customers
array<struct<customer_details:struct<first_name:string,last_name:string,DOB:string,Addr1:string,City:string,state:string,country:string>>>
) row format serde
'com.ibm.spss.hive.serde2.xml.XmlSerDe' with
serdeproperties ( “Column.xpath.customerdetails”
= “/FormResponse/Customers/customer_details” ) INPUTFORMAT 'com.ibm.spss.hive.serde2.xml.XmlInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat' TBLPROPERTIES ( "xmlinput.start"="<FormResponse>", "xmlinput.end"="</FormResponse>" ); XML file <FormResponse> <Customers> <customer_details> <First_name>Anji</First_name> <Last_name> Raju</Last_name> <DOB>06/24/1278<DOB> <Addr1> 14 duck st </Addr1> <City> boston </City> <State> OH </State> <Country> USA </Country> </customer_details> <customer_details> <First_name>Jeet</First_name> <Last_name> Anu</Last_name> <DOB>06/24/1279<DOB> <Addr1> tuttles groove </Addr1> <City> denver </City> <State> CA </State> <Country> USA </Country> </customer_details> <customer_details> <First_name>Test1</First_name> <Last_name>
Test_last</Last_name> <DOB>006/24/1280<DOB> <Addr1> Sleek street </Addr1> <City> cali </City> <State> MA </State> <Country> USA </Country> </customer_details> </Customers> </FormResponse> When I write the command: Hive> Select explode(customers.customer) from
test_table.customer_details; The output is {“First_Name”:”Anji”,”Last_name”:”Raju”,”DOB”:” 06/24/1278”,”Addr1”:” 14 duck st”,”City”:”boston”,”State”:”OH”,”Country”:”USA”} {“First_Name”:”Jeet”,”Last_name”:”Anu”,”DOB”:” 06/24/1279”,”Addr1”:” tuttles grove”,”City”:”denver”,”State”:”CA”,”Country”:”USA”} {“First_Name”:”Test1”,”Last_name”:”Test_last”,”DOB”:” 06/24/1280”,”Addr1”:” Sleek street”,”City”:”cali”,”State”:”MA”,”Country”:”USA”} I did create a new table create table test_table.new ( customer
array<struct<first_name:string,last_name:string,DOB:string, Addr1:string,
city:string, State:string, Country:string >>); I did try writing a hive command something like below to get
the expected output but I am getting an Error: hive> insert into test_table.new select explode(
customers.customer_details) from test_table.customer_details; FAILED: SemanticException [Error 10044]: Line 1:12 Cannot
insert into target table because column number/types are different
'test_mess1': Cannot convert column 1 from
array<struct<first_name:string,last_name:string,DOB:string,Addr1:string,city:string,State:string,Country:string
>> to array<struct<first_name:string,last_name:string,DOB:string,
Addr1:string, city:string, State:string, Country:string >> The Expected output should be:
First_name
Last_name
DOB
Addr1
City
State
Country
Anji
Raju
06/24/1278
14 duck st
boston
OH
USA
Jeet
Anu
06/24/1279
tuttles groove
denver
CA
USA
Test1
Test_lat
06/24/1280
Sleek street
cali
MA
USA
Could someone help me to resolve this issue???
... View more
Labels:
- Labels:
-
Apache Hive
06-20-2018
01:59 PM
Thanks @Vinicius Higa Murakami , I was able to figure out the Typo Mistake. Thanks for the help
... View more
06-18-2018
08:14 PM
Thanks @Vinicius Higa Murakami. I was able to make correction and was able to achieve it. There is one more help I need . I am trying to load the complex structure of XML> i am able to load all the fields but there is one section which I am trying in many way to load the data but upon the loading of data, i still see the values to be NULL> Attached the files. a) XPATH what i am writing b) the XML file I am loading xpath.txt nestedxml.xml Output: 0 Success A [{"message":{"severity":null,"statuscode":"0","pagenumber":"1","filename":"File.pdf","layoutfileid":"ACORD_125_2016_03","layoutpageid":"page1_COMMERCIAL_INSURANCE_APPLICATION","layoutidentifertext":null,"text":"Page Successfully Processed"}},{"message":{"severity":null,"statuscode":"0","pagenumber":"2","filename":"File.pdf","layoutfileid":"ACORD_125_2016_03","layoutpageid":"page2_CONTACT_INFORMATION","layoutidentifertext":null,"text":"Page Successfully Processed"}},{"message":{"severity":null,"statuscode":"0","pagenumber":"3","filename":"File.pdf","layoutfileid":"ACORD_125_2016_03","layoutpageid":"page3_GENERAL_INFORMATION","layoutidentifertext":null,"text":"Page Successfully Processed"}},{"message":{"severity":null,"statuscode":"0","pagenumber":"4","filename":"File.pdf","layoutfileid":"ACORD_125_2016_03","layoutpageid":"page4_PRIOR_CARRIER_INFORMATION","layoutidentifertext":null,"text":"Page Successfully Processed"}}] {"transactionid":"80c32800-65cb-11e8-8f5a-005056a3433861521442784784","productversion":"2.06.3.00064","requestdate":"2018-06-01","processingtime":"4 sec","filesize":"464888 bytes","extractiontechnique":"Text"} {"lobcd":"CPKGE","filesreceivedcnt":"1","pagesreceivedcnt":"4","pagesprocessedcnt":"4","formsprocessedcnt":"1","forms":[{"form":{"formname":"ACORD_125_2016_03","totalfields":"533","fieldswithdata":"82","ratebearingfields":"12","ratebearingfieldswithdata":"9"}}]} NULL Could you please help me with this Thanks, Harish
... View more
06-14-2018
03:41 PM
Hi All, I am trying to load nested XML file into HIVE table, but i see multiple lines getting generated for the values in Array. XML File : <?xml version="1.0" encoding="UTF-8"?>
<FormServerResponse>
<Status>
<StatusCode>0</StatusCode>
<StatusDesc> Success</StatusDesc>
<ConfidenceIndex> D</ConfidenceIndex>
<Messages>
<Message severity="Success">
<StatusCode> 0</StatusCode>
<PageNumber> 1</PageNumber>
<FileName> Filxxxxe.pdf</FileName>
<LayoutFileId> xxxx</LayoutFileId>
<LayoutPageId> xxxxx</LayoutPageId>
<LayoutIdentiferText/>
<Text> xx xxxxxx xx</Text>
</Message>
</Messages>
</Status>
</FormServerResponse> Command I used to load data into HIVE: create table ViewTable( StatusCode string,StatusDesc string,ConfidenceIndex string,Messages struct<Message: array<struct< severity:string,statuscode:string,pagenumber:string,filename:string,layoutfileid:string, layoutpageid:string, layoutidentifertext:string,text:string>>> )
row format serde 'com.ibm.spss.hive.serde2.xml.XmlSerDe'
with serdeproperties
(
"column.xpath.StatusCode" = "/Status/StatusCode/text()", "column.xpath.StatusDesc" = "/Status/StatusDesc/text()","column.xpath.ConfidenceIndex" = "/Status/ConfidenceIndex/text()", "column.xpath.Messages" = "/Status/Messages/Message"
)
stored as
inputformat 'com.ibm.spss.hive.serde2.xml.XmlInputFormat'
outputformat 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
tblproperties
(
"xmlinput.start" = "<Status>"
,"xmlinput.end" = "</Status>"
); Result : select * from viewtable;
OK
0 Success D {"message":[{"severity":null,"statuscode":"0","pagenumber":null,"filename":null,"layoutfileid":null,"layoutpageid":null,"layoutidentifertext":null,"text":null},{"severity":null,"statuscode":null,"pagenumber":"1","filename":null,"layoutfileid":null,"layoutpageid":null,"layoutidentifertext":null,"text":null},{"severity":null,"statuscode":null,"pagenumber":null,"filename":"Filxxxxe.pdf","layoutfileid":null,"layoutpageid":null,"layoutidentifertext":null,"text":null},{"severity":null,"statuscode":null,"pagenumber":null,"filename":null,"layoutfileid":"xxxx","layoutpageid":null,"layoutidentifertext":null,"text":null},{"severity":null,"statuscode":null,"pagenumber":null,"filename":null,"layoutfileid":null,"layoutpageid":"xxxxx","layoutidentifertext":null,"text":null},{"severity":null,"statuscode":null,"pagenumber":null,"filename":null,"layoutfileid":null,"layoutpageid":null,"layoutidentifertext":null,"text":null},{"severity":null,"statuscode":null,"pagenumber":null,"filename":null,"layoutfileid":null,"layoutpageid":null,"layoutidentifertext":null,"text":"xx xxxxxx xx"}]}
Time taken: 0.423 seconds, Fetched: 1 row(s) Result expected: 0 Success D {"message":[{"severity":success,"statuscode":"0","pagenumber":1,"filename":Filxxxe.pdf,"layoutfileid":xxxx,"layoutpageid":xxxxx,"layoutidentifertext":null,"text":xx xxxxx xx}]} Could someone help me where I am making mistake?????
... View more
Labels:
- Labels:
-
Apache Hive