sivasai2006
Explorer
My Accepted Solutions
Show All

JSON is null in DataFrame

by Explorer in Support Questions
‎05-08-2019 02:53 PM
‎05-08-2019 02:53 PM
I have a valid JSON file [[{"eventEndTimestamp":"2019-05-07T03:48:01Z","eventStartTimestamp":"2019-05-07T03:48:01Z","sourceSystem":"X1","transactionCode":"asdf","transactionSuccessIndicator":"Y"}],[{"eventEndTimestamp":"2019-05-07T03:48:04Z","eventStartTimestamp":"2019-05-07T03:48:04Z","sourceSystem":"X2","transactionCode":"qwerty","transactionSuccessIndicator":"Y"}]] But when I print it in spark console, I'm getting the dataframe but with null values. The dataframe columns are inferred from this file correctly with null values. The code is working fine with other JSON files but not this one. Output: Batch: 0 ------------------------------------------- +-----------------+-------------------+----------------+--------------------------------+---------------------------+ |eventEndTimestamp|eventStartTimestamp|sourceSystem|transactionCode|transactionSuccessIndicator| +-----------------+-------------------+----------------+--------------------------------+---------------------------+ | null| null| null| null| null| +-----------------+-------------------+----------------+--------------------------------+---------------------------+ ... View more
Labels:

Re: Unable to load data to hive table

by Explorer in Support Questions
‎10-18-2018 02:32 AM
‎10-18-2018 02:32 AM
bash-4.1$ hdfs dfs -ls /etc/hive/data/weather -rw-r--r-- 1 cloudera supergroup 220 2018-10-17 13:03 /etc/hive/data/weather ... View more

Unable to load data to hive table

by Explorer in Support Questions
‎10-18-2018 01:57 AM
‎10-18-2018 01:57 AM
CREATE TABLE weather (wban INT, date STRING, precip INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION '/etc/hive/data/weather'; Getting error while creating the table, FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:hdfs://quickstart.cloudera:8020/etc/hive/data/weather is not a directory or unable to create one) ... View more
Labels:

Re: Hive table is loading with NULL values

by Explorer in Support Questions
‎10-18-2018 01:52 AM
1 Kudo
‎10-18-2018 01:52 AM
1 Kudo
Thank you @Jonathan Sneep I resolved the issue. It is due to typing the query in text editor and copied it to hive cli, the single quote was malformed. ... View more

Editing a sqoop job

by Explorer in Support Questions
‎10-17-2018 07:28 AM
‎10-17-2018 07:28 AM
I have an existing sqoop job 'InsertOrder'. Now I need to edit it. I couldn't find a way to edit, but getting only way to delete that job and re-create it. Is it possible to edit the existing job ... View more
Labels:

Hive table is loading with NULL values

by Explorer in Support Questions
‎10-17-2018 04:32 AM
‎10-17-2018 04:32 AM
CREATE TABLE IF NOT EXISTS employee (eid int, name String, salary String) STORED AS TEXTFILE; LOAD DATA LOCAL INPATH '/user/hdadmin/sample.txt' OVERWRITE INTO TABLE employee; Sample.txt 1201^AGopal^A45000 1202^AManisha^A45000 1203^AMasthanvali^A40000 1204^AKiran^A40000 1205^AKranthi^A30000 Both queries were executed successfully, but the table have only NULL values, Output: hive> select * from employee; OK NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL Time taken: 0.148 seconds, Fetched: 5 row(s) Please help me to fix this ... View more
Labels:

Unable to create a table in hive

by Explorer in Support Questions
‎10-17-2018 04:32 AM
‎10-17-2018 04:32 AM
CREATE TABLE IF NOT EXISTS employee (eid int, name String, salary String) ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘\t’ LINES TERMINATED BY ‘\n’ STORED AS TEXTFILE; I'm getting the below error: MismatchedTokenException(26!=298) at org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617) at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115) at org.apache.hadoop.hive.ql.parse.HiveParser.tableRowFormatFieldIdentifier(HiveParser.java:31777) at org.apache.hadoop.hive.ql.parse.HiveParser.rowFormatDelimited(HiveParser.java:30730) at org.apache.hadoop.hive.ql.parse.HiveParser.tableRowFormat(HiveParser.java:31006) at org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:4677) at org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:2138) at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1392) at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1030) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:199) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:417) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1026) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1091) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:962) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:952) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:269) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:221) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:431) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:800) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:694) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:633) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) FAILED: ParseException line 3:44 mismatched input 't' expecting StringLiteral near 'BY' in table row format's field separator Even iIf I use '\073' , I'm getting the error: MismatchedTokenException(287!=298) at org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617) at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115) at org.apache.hadoop.hive.ql.parse.HiveParser.tableRowFormatFieldIdentifier(HiveParser.java:31777) at org.apache.hadoop.hive.ql.parse.HiveParser.rowFormatDelimited(HiveParser.java:30730) at org.apache.hadoop.hive.ql.parse.HiveParser.tableRowFormat(HiveParser.java:31006) at org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:4677) at org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:2138) at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1392) at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1030) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:199) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:417) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1026) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1091) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:962) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:952) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:269) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:221) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:431) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:800) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:694) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:633) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) FAILED: ParseException line 3:44 mismatched input '073' expecting StringLiteral near 'BY' in table row format's field separator Please let me know how to fix this. ... View more
Labels:

Hive table need to be stored with a different deli...

by Explorer in Support Questions
‎09-21-2018 02:53 PM
‎09-21-2018 02:53 PM
I have a text file with values separated by ','. And I have to save the file as a text file in hive with delimiter '|'. How to apply this in query? ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎05-13-2019 11:51 AM
Public Statistics
Date Registered ‎09-21-2018 01:32 PM
Last Visited ‎05-13-2019 11:51 AM
Total Messages Posted 11
Total Kudos Received 1