I have installed Spark and configure Hive to use it as execution engine. Select * from table name works fine. But select count(*) from table name fails with following error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask At times also got an error stating "failed to create spark client" I have also tried to modify the memort parameters but to no avail. Can you please tell me what should be the ideal memory setting? Below is the directory structure from hdfs drwxr-xr-x - admin admin 0 2017-07-28 16:36 /user/admin drwx------ - ec2-user supergroup 0 2017-07-28 17:50 /user/ec2-user drwxr-xr-x - hdfs hdfs 0 2017-07-28 11:37 /user/hdfs drwxrwxrwx - mapred hadoop 0 2017-07-16 06:03 /user/history drwxrwxr-t - hive hive 0 2017-07-16 06:04 /user/hive drwxrwxr-x - hue hue 0 2017-07-28 10:16 /user/hue drwxrwxr-x - impala impala 0 2017-07-16 07:13 /user/impala drwxrwxr-x - oozie oozie 0 2017-07-16 06:05 /user/oozie drwxr-x--x - spark spark 0 2017-07-28 17:17 /user/spark drwxrwxr-x - sqoop2 sqoop 0 2017-07-16 06:37 /user/sqoop2 the /user directory has owner as ec2-user and group as supergroup. I tried running the query from CLI: WARNING: Hive CLI is deprecated and migration to Beeline is recommended. hive> select count(*) from kaggle.test_house; Query ID = ec2-user_20170728174949_aa9d7be9-038c-44a0-a42b-1b210a37f4ec Total jobs = 1 Launching Job 1 out of 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)' FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
... View more
