Hive query giving null values when trying to pull the data from json which has @date column.below is the sample data and the query iam using. {"name":"jai","@date":"2015-06-15"} {"name":"pri","@date":"2017-08-25"} CREATE TABLE json_obj( json string) LOCATION '/user/*************/' select get_json_object(json_obj.json,'$.name') as name, get_json_object(json_obj.json,'$.@date') as date from json_obj; ouput: name date jai null pri null expected output: name date jai 2015-06-15 pri 2017-08-25 ... View more

Is there any possibility where we can check the count of vesions of a column family in hbase ... View more

Hi, I have a use case where i need to load json data to hbase using pyspark with row key and 3 column families,Can anyone please help me how to do this. Below is the json i want to load. { "ticid": "1496", "ticlocation": "vizag", "custnum": "222", "Comments": { "comment": [{ "commentno": "1", "desc": "journey", "passengerseat": { "intele": "09" }, "passengerloc": { "intele": "s15" } }, { "commentno": "5", "desc": " food", "passengerseat": { "intele": "09" }, "passengerloc": { "intele": "s15" } }, { "commentno": "12", "desc": " service", "passengerseat": { "intele": "09" }, "passengerloc": { "intele": "s15" } }] }, "Rails": { "Rail": [{ "Traino": "AP1545", "startcity": "vizag", "passengerseat": "5" }, { "Traino": "AP1555", "startcity": "HYD", "passengerseat": "15A" }] } } ticid is the row key ticlocation ,custnum need to be in column family 1 Comments needs to be column family 2 Rails needs to be column family 3 ... View more

Hi , I am trying to create a hive table on top of son data file but getting the below error.Below are my json file,hive ddl and error. Json file:::::::: { "purchaseid": { "ticketnumber": "23546852222", "location": "vizag", "Travelerhistory": { "trav": { "fname": "ramu", "lname": "gogi", "travelingarea": { "destination": { "stationid": "KAJKL", "stationname": "hyd" } }, "food": { "foodpref": [{ "foodcode": "CK567", "foodcodeSegment": "NOVEG" }, { "foodcode": "MM98", "foodcodeSegment": "VEG" } ] } } } } } Hive DDL:: add jar /home/**********/json-serde-1.3.7-jar-with-dependencies.jar; CREATE external TABLE ds1414( ticketnumber string, location string, Travelerhistory ARRAY<struct<trav :struct<fname:string,lname:string, travelingarea :ARRAY<struct<destination :struct<stationid:string,stationname:string>>>, food :ARRAY<struct<foodpref :struct<foodcode:string,foodcodeSegment:string>>> >>> ) ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' location '/user/***********/json2/' ; Error::: select * from ds1414; OK Failed with exception java.io.IOException:org.apache.hadoop.hive.serde2.SerDeException: org.codehaus.jackson.JsonParseException: Unexpected end-of-input: expected close marker for OBJECT (from [Source: java.io.ByteArrayInputStream@205df5dc; line: 1, column: 0]) at [Source: java.io.ByteArrayInputStream@205df5dc; line: 1, column: 3] Time taken: 0.212 seconds ... View more

Hi , I am trying do filter on rowkey which is combination of 3 columns in my case which are separated by a space.i want to query on filter conditions on 2 columns..below is how my data looks in hbase and hbase queries i am using. i want to query on rowkey which contains values row1 and rad.how can i do that hbase(main):003:0> scan 'testspaces' ROW COLUMN+CELL row1 gud column=cf:a2, timestamp=1523548834897, value=value1e row1 rad column=cf:a2, timestamp=1523548716606, value=value1e row1 sec column=cf:a2, timestamp=1523548822010, value=value1e row2 rad exam column=cf:a3, timestamp=1523548741273, value=vale1e 4 row(s) in 0.0150 seconds scan 'testspaces',{FILTER =>"(PrefixFilter('row1') OR PrefixFilter('sec')"} scan 'testspaces', { LIMIT => 3, FILTER => "org.apache.hadoop.hbase.filter.RowFilter( =, 'row1') AND ValueFilter( =, 'sec'} scan 'testspaces', {FILTER => org.apache.hadoop.hbase.filter.RowFilter.new(CompareFilter::CompareOp.valueOf('EQUAL'),SubstringComparator.new("sec")) AND org.apache.hadoop.hbase.filter.RowFilter.new(CompareFilter::CompareOp.valueOf('EQUAL'),SubstringComparator.new("row1"))} ... View more

Hi I have an xml with below format,Can anyone help me how to create a hive DDL on top of this xml. <code><root><root1><id>4545482361</id>`enter code here` <joiningdate>1/3/2010</joiningdate><Segments><Segmentxse:type="manager"><cityworked>Hyd</cityworked><reports>john</reports><salary>150000</salary><datestarted>1/3/2012</datestarted></Segment><Segmentxse:type="manager"><cityworked>Hyd</cityworked><reports>mike</reports><salary>225000</salary><datestarted>1/9/2014</datestarted></Segment><Segmentxse:type="VP"><cityworked>mumbai</cityworked><datestarted>1/9/2014</datestarted><subemployees><Fname>ram</Fname><Lname>Achanta</Lname><Desgination>Director of IT</Desgination></subemployees></Segment><Segmentxse:type="SVP"><Staus>currentposition</status><numberofemployees>10</numberofemployees></Segment></Segments></root1></root> ... View more

Hi I am new to hadoop i am trying to add solr service in ambari by using a recepi and executing using chef command ,my code works fine and adds solr compnent but installation is failing and getting the error: stderr: /var/lib/ambari-agent/data/errors-670.txt Repository HDP-UTILS-1.1.0.21 is listed more than once in the configuration Repository HDP-UTILS-1.1.0.21 is listed more than once in the configuration Repository HDP-UTILS-1.1.0.21 is listed more than once in the configuration Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/SOLR/5.5.2.2.5/package/scripts/solr.py", line 101, in <module> Solr().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/SOLR/5.5.2.2.5/package/scripts/solr.py", line 15, in install import params File "/var/lib/ambari-agent/cache/common-services/SOLR/5.5.2.2.5/package/scripts/params.py", line 12, in <module> import status_params File "/var/lib/ambari-agent/cache/common-services/SOLR/5.5.2.2.5/package/scripts/status_params.py", line 13, in <module> solr_config_pid_file = format('{solr_config_pid_dir}/solr-{solr_config_port}.pid') File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py", line 95, in format return ConfigurationFormatter().format(format_string, args, **result) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py", line 59, in format result_protected = self.vformat(format_string, args, all_params) File "/usr/lib64/python2.6/string.py", line 549, in vformat result = self._vformat(format_string, args, kwargs, used_args, 2) File "/usr/lib64/python2.6/string.py", line 582, in _vformat result.append(self.format_field(obj, format_spec)) File "/usr/lib64/python2.6/string.py", line 599, in format_field return format(value, format_spec) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__ raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!") resource_management.core.exceptions.Fail: Configuration parameter 'solr-config-env' was not found in configurations dictionary! ... View more