Member since
05-05-2015
7
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
85073 | 09-01-2015 12:14 AM |
11-02-2015
01:49 PM
hi, I think you missed to add <jar> under spark action. If you have <jar> but still facing the issue, please post your worlflow xml. Thanks ! Siva
... View more
09-01-2015
12:14 AM
2 Kudos
hi, Use "nohup spark-submit <parameters> 2>&1 < /dev/null &" nohup will not print the Spark stremaing logs on Terminal and it runs as a background process.. Thanks ! Siva
... View more
05-07-2015
09:12 AM
hi Chris and Michael Thanks for your quick response and solution.. Let me try this solution in our data model.. I will update this thread about the outcome.. Many thanks again.. explodeArray would be great option.. Nevetheless this solution is really cool.. Thanks again for you time.. Thanks ! Siva
... View more
05-05-2015
09:58 PM
We have a nested parquet file with below sample strucutre.. department_id String department_name String Employees Array<Struct<first_name String, last_name String, email String>> We want to flatten above structure using explode API of data frames. Whatever samples that we got from the documentation and git is talking about exploding a String by splitting but here we have an Array strucutre. We did not get any examples for this in web also. Or I could be missing something.. Could anyone please help in using explode method with nested array strucutre.. Thanks in advance.. I referrerd below API and github links https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.DataFrame https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala Thanks ! Siva
... View more
Labels:
- Labels:
-
Apache Spark