Member since
02-22-2017
7
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3500 | 01-11-2017 05:36 PM |
01-11-2017
05:36 PM
I changed to the call to include the args below and it worked. --num-executors 4 \ --executor-memory 512M \ --driver-memory 512M \
... View more
01-11-2017
05:31 PM
I changed the code to read from Hive instead and still got the same error. val sparkConf = new SparkConf().setAppName("test-spark").set("spark.eventLog.enabled", "true")
val sc = new SparkContext(sparkConf)
val sql = new HiveContext(sc)
sql.sql("SELECT * FROM foodmart.customer").show() I added these args to the call but also no luck. --num-executors 4 \ --executor-memory 1G \ --driver-memory 1G \
... View more
01-10-2017
10:30 PM
1 Kudo
I'm using spark to import data from a postgres database on my host machine into hdp sandbox. I get this error everytime. Here is the command call spark-submit \
--packages org.apache.spark:spark-streaming-kafka_2.10:1.6.0,com.databricks:spark-csv_2.10:1.5.0,com.databricks:spark-avro_2.10:2.0.1 \
--master yarn \
--deploy-mode client \
--jars ${L_ALL_JARS} \
--class ${L_CLASS} \
bdi.spark.jar ${HDM_CUSTOMER_KEY} yarn-client scala val sparkConf = new SparkConf().setAppName(args(2))
val sc = new SparkContext(sparkConf)
val hive = new HiveContext(sc)
val df = sql.read.format("jdbc").options(opts).load.cache
df.show
The error doesn't occur util I call an action on the dataframe show().
... View more
Labels:
- Labels:
-
Apache Spark