Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

JSON is null in DataFrame

Highlighted

JSON is null in DataFrame

New Contributor

I have a valid JSON file


[[{"eventEndTimestamp":"2019-05-07T03:48:01Z","eventStartTimestamp":"2019-05-07T03:48:01Z","sourceSystem":"X1","transactionCode":"asdf","transactionSuccessIndicator":"Y"}],[{"eventEndTimestamp":"2019-05-07T03:48:04Z","eventStartTimestamp":"2019-05-07T03:48:04Z","sourceSystem":"X2","transactionCode":"qwerty","transactionSuccessIndicator":"Y"}]]


But when I print it in spark console, I'm getting the dataframe but with null values. The dataframe columns are inferred from this file correctly with null values. The code is working fine with other JSON files but not this one.


Output:

Batch: 0

-------------------------------------------

+-----------------+-------------------+----------------+--------------------------------+---------------------------+

|eventEndTimestamp|eventStartTimestamp|sourceSystem|transactionCode|transactionSuccessIndicator|

+-----------------+-------------------+----------------+--------------------------------+---------------------------+

| null| null| null| null| null|

+-----------------+-------------------+----------------+--------------------------------+---------------------------+


1 REPLY 1

Re: JSON is null in DataFrame

Community Manager

The above was originally posted in the Community Help track. On Sat May 11 03:19 UTC 2019, the HCC moderation staff moved it to the Data Processing track. The Community Help track is intended for questions about using the HCC site itself.