Beeline can connect and read but can't insert

[#1]

First post, and I think this is in the right place. Please move it if I've dropped the ball.

I've built out a a small 3 node 5.7.1 cluster on Xen and one of my teams is running some tests using beeline and getting mixed results. This is an unsecured cluster (no kerberos).

We can connect and pull table and other info, but when it comes to inserts we see failures.

Able to connect;

[devuser@svqxbdcn6cdh57sparkn1 ~]$ beeline
16/06/16 11:07:01 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it.
Beeline version 1.1.0-cdh5.7.1 by Apache Hive
beeline> !connect jdbc:hive2://svqxbdcn6cdh57sparkn1:10000/default
scan complete in 6ms
Connecting to jdbc:hive2://svqxbdcn6cdh57sparkn1:10000/default
Enter username for jdbc:hive2://svqxbdcn6cdh57sparkn1:10000/default:
Enter password for jdbc:hive2://svqxbdcn6cdh57sparkn1:10000/default:
Connected to: Apache Hive (version 1.1.0-cdh5.7.1)
Driver: Hive JDBC (version 1.1.0-cdh5.7.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ

Reading data works fine.

0: jdbc:hive2://svqxbdcn6cdh57sparkn1:10000/d> show tables;
INFO : Compiling command(queryId=hive_20160616110707_4f06e4d5-b15b-4de1-b3b3-19cf85488c4c): show tables
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling command(queryId=hive_20160616110707_4f06e4d5-b15b-4de1-b3b3-19cf85488c4c); Time taken: 0.004 seconds
INFO : Executing command(queryId=hive_20160616110707_4f06e4d5-b15b-4de1-b3b3-19cf85488c4c): show tables
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=hive_20160616110707_4f06e4d5-b15b-4de1-b3b3-19cf85488c4c); Time taken: 0.012 seconds
INFO : OK
+--------------------------------+--+
| tab_name |
+--------------------------------+--+
| account |
| agg_c_10_sales_fact_1997 |
| agg_c_14_sales_fact_1997 |
| agg_c_special_sales_fact_1997 |
| agg_g_ms_pcat_sales_fact_1997 |
| agg_l_03_sales_fact_1997 |
| agg_l_04_sales_fact_1997 |
| agg_l_05_sales_fact_1997 |
| agg_lc_06_sales_fact_1997 |
| agg_lc_100_sales_fact_1997 |
| agg_ll_01_sales_fact_1997 |
| agg_pl_01_sales_fact_1997 |
| alena |
| category |
| currency |
| customer |
| days |
| department |
| employee |
| employee_closure |
| expense_fact |
| foo |
| inventory_fact_1997 |
| inventory_fact_1998 |
| new |
| position |
| product |
| product_class |
| promotion |
| region |
| reserve_employee |
| salary |
| sales_fact_1997 |
| sales_fact_1998 |
| sales_fact_dec_1998 |
| store |
| store_ragged |
| test |
| test123 |
| time_by_day |
| warehouse |
| warehouse_class |
| weblog_lilycount |
| weblogs |
| weblogs_count |
+--------------------------------+--+
45 rows selected (0.275 seconds)

Inserts do not work;

0: jdbc:hive2://svqxbdcn6cdh57sparkn1:10000/d> insert into table foo values ('a','b');
INFO : Compiling command(queryId=hive_20160616110707_0b83aaa1-15cd-44de-bb24-717101b3a22a): insert into table foo values ('a','b')
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:values__tmp__table__1.tmp_values_col1, type:string, comment:null), FieldSchema(name:values__tmp__table__1.tmp_values_col2, type:string, comment:null)], properties:null)
INFO : Completed compiling command(queryId=hive_20160616110707_0b83aaa1-15cd-44de-bb24-717101b3a22a); Time taken: 0.094 seconds
INFO : Executing command(queryId=hive_20160616110707_0b83aaa1-15cd-44de-bb24-717101b3a22a): insert into table foo values ('a','b')
INFO : Query ID = hive_20160616110707_0b83aaa1-15cd-44de-bb24-717101b3a22a
INFO : Total jobs = 3
INFO : Launching Job 1 out of 3
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=<number>
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=<number>
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=<number>
INFO : Starting Spark Job = a2bf34f5-21b1-438b-8648-15f682328743
INFO : Job hasn't been submitted after 61s. Aborting it.
ERROR : Status: SENT
ERROR : Failed to execute spark task, with exception 'java.lang.IllegalStateException(RPC channel is closed.)'
java.lang.IllegalStateException: RPC channel is closed.
at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at org.apache.hive.spark.client.rpc.Rpc.call(Rpc.java:276)
at org.apache.hive.spark.client.rpc.Rpc.call(Rpc.java:259)
at org.apache.hive.spark.client.SparkClientImpl$ClientProtocol.cancel(SparkClientImpl.java:532)
at org.apache.hive.spark.client.SparkClientImpl.cancel(SparkClientImpl.java:189)
at org.apache.hive.spark.client.JobHandleImpl.cancel(JobHandleImpl.java:62)
at org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.cancelJob(RemoteSparkJobRef.java:54)
at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:119)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1770)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1527)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1306)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1115)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:178)
at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:72)
at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:232)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:245)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR : Failed to execute spark task, with exception 'java.lang.IllegalStateException(RPC channel is closed.)'
java.lang.IllegalStateException: RPC channel is closed.
at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at org.apache.hive.spark.client.rpc.Rpc.call(Rpc.java:276)
at org.apache.hive.spark.client.rpc.Rpc.call(Rpc.java:259)
at org.apache.hive.spark.client.SparkClientImpl$ClientProtocol.cancel(SparkClientImpl.java:532)
at org.apache.hive.spark.client.SparkClientImpl.cancel(SparkClientImpl.java:189)
at org.apache.hive.spark.client.JobHandleImpl.cancel(JobHandleImpl.java:62)
at org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.cancelJob(RemoteSparkJobRef.java:54)
at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:119)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1770)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1527)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1306)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1115)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:178)
at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:72)
at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:232)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:245)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
INFO : Completed executing command(queryId=hive_20160616110707_0b83aaa1-15cd-44de-bb24-717101b3a22a); Time taken: 72.867 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask (state=08S01,code=1)
0: jdbc:hive2://svqxbdcn6cdh57sparkn1:10000/d>

The output from hadoop-cmf-hive-HIVESERVER2-svqxbdcn6cdh57sparkn1.pentahoqa.com.log.out has come data that I've been trying to parse (new to CDH/hadoop/hive/etc).

2016-06-16 11:07:35,490 INFO org.apache.hadoop.hive.ql.exec.Task: [HiveServer2-Background-Pool: Thread-480]: Starting Spark Job = a2bf34f5-21b1-438b-8648-15f682328743
2016-06-16 11:07:35,490 INFO org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-480]: <PERFLOG method=SparkRunJob from=org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor>
2016-06-16 11:07:35,490 INFO org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-480]: <PERFLOG method=SparkSubmitToRunning from=org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor>
2016-06-16 11:07:38,131 INFO org.apache.hive.spark.client.rpc.RpcDispatcher: [RPC-Handler-5]: [ClientProtocol] Closing channel due to exception in pipeline (Connection reset by peer).
2016-06-16 11:07:38,131 WARN org.apache.hive.spark.client.rpc.Rpc: [RPC-Handler-5]: Failed to send RPC, closing connection.
java.nio.channels.ClosedChannelException
2016-06-16 11:07:38,132 WARN org.apache.hive.spark.client.SparkClientImpl: [RPC-Handler-5]: Client RPC channel closed unexpectedly.
2016-06-16 11:08:31,786 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: 16/06/16 11:08:31 INFO yarn.Client: Application report for application_1465925473768_0011 (state: FINISHED)
2016-06-16 11:08:31,792 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: 16/06/16 11:08:31 INFO yarn.Client:
2016-06-16 11:08:31,792 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: client token: N/A
2016-06-16 11:08:31,792 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: diagnostics: Uncaught exception: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested virtual cores < 0, or requested virtual cores > max configured, requestedVirtualCores=4, maxVirtualCores=2
2016-06-16 11:08:31,792 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:256)
2016-06-16 11:08:31,792 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:224)
2016-06-16 11:08:31,792 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:231)
2016-06-16 11:08:31,792 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:96)
2016-06-16 11:08:31,792 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:504)
2016-06-16 11:08:31,792 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
2016-06-16 11:08:31,792 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
2016-06-16 11:08:31,792 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
2016-06-16 11:08:31,792 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
2016-06-16 11:08:31,792 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
2016-06-16 11:08:31,792 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
2016-06-16 11:08:31,792 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at java.security.AccessController.doPrivileged(Native Method)
2016-06-16 11:08:31,792 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at javax.security.auth.Subject.doAs(Subject.java:422)
2016-06-16 11:08:31,792 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
2016-06-16 11:08:31,792 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
2016-06-16 11:08:31,793 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]:
2016-06-16 11:08:31,793 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: ApplicationMaster host: 10.177.177.170
2016-06-16 11:08:31,793 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: ApplicationMaster RPC port: 0
2016-06-16 11:08:31,793 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: queue: root.anonymous
2016-06-16 11:08:31,793 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: start time: 1466089650311
2016-06-16 11:08:31,793 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: final status: FAILED
2016-06-16 11:08:31,793 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: tracking URL: http://svqxbdcn6cdh57sparkn1.pentahoqa.com:8088/proxy/application_1465925473768_0011/history/application_1465925473768_0011/1
2016-06-16 11:08:31,793 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: user: anonymous
2016-06-16 11:08:32,007 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: 16/06/16 11:08:32 WARN security.UserGroupInformation: PriviledgedActionException as:anonymous (auth:PROXY) via hive (auth:SIMPLE) cause:org.apache.spark.SparkException: Application application_1465925473768_0011 finished with failed status
2016-06-16 11:08:32,009 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
2016-06-16 11:08:32,009 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1706)
2016-06-16 11:08:32,009 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:161)
2016-06-16 11:08:32,009 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
2016-06-16 11:08:32,010 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
2016-06-16 11:08:32,010 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2016-06-16 11:08:32,010 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: Caused by: org.apache.spark.SparkException: Application application_1465925473768_0011 finished with failed status
2016-06-16 11:08:32,010 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.spark.deploy.yarn.Client.run(Client.scala:1035)
2016-06-16 11:08:32,011 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1082)
2016-06-16 11:08:32,011 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.spark.deploy.yarn.Client.main(Client.scala)
2016-06-16 11:08:32,011 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2016-06-16 11:08:32,011 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2016-06-16 11:08:32,012 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2016-06-16 11:08:32,012 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at java.lang.reflect.Method.invoke(Method.java:497)
2016-06-16 11:08:32,012 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
2016-06-16 11:08:32,012 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
2016-06-16 11:08:32,012 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:161)
2016-06-16 11:08:32,013 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at java.security.AccessController.doPrivileged(Native Method)
2016-06-16 11:08:32,013 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at javax.security.auth.Subject.doAs(Subject.java:422)
2016-06-16 11:08:32,013 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
2016-06-16 11:08:32,013 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: ... 4 more
2016-06-16 11:08:32,019 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: 16/06/16 11:08:32 INFO util.ShutdownHookManager: Shutdown hook called
2016-06-16 11:08:32,020 INFO org.apache.hive.spark.client.SparkClientImpl: [stderr-redir-1]: 16/06/16 11:08:32 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-b59204af-1ba8-42c9-88c9-121cde171966
2016-06-16 11:08:32,378 WARN org.apache.hive.spark.client.SparkClientImpl: [Driver]: Child process exited with code 1
2016-06-16 11:08:36,499 INFO org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor: [HiveServer2-Background-Pool: Thread-480]: Job hasn't been submitted after 61s. Aborting it.
2016-06-16 11:08:36,499 ERROR org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor: [HiveServer2-Background-Pool: Thread-480]: Status: SENT
2016-06-16 11:08:36,499 INFO org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-480]: </PERFLOG method=SparkRunJob start=1466089655490 end=1466089716499 duration=61009 from=org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor>
2016-06-16 11:08:36,500 ERROR org.apache.hadoop.hive.ql.exec.Task: [HiveServer2-Background-Pool: Thread-480]: Failed to execute spark task, with exception 'java.lang.IllegalStateException(RPC channel is closed.)'
java.lang.IllegalStateException: RPC channel is closed.
at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at org.apache.hive.spark.client.rpc.Rpc.call(Rpc.java:276)
at org.apache.hive.spark.client.rpc.Rpc.call(Rpc.java:259)
at org.apache.hive.spark.client.SparkClientImpl$ClientProtocol.cancel(SparkClientImpl.java:532)
at org.apache.hive.spark.client.SparkClientImpl.cancel(SparkClientImpl.java:189)
at org.apache.hive.spark.client.JobHandleImpl.cancel(JobHandleImpl.java:62)
at org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.cancelJob(RemoteSparkJobRef.java:54)
at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:119)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1770)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1527)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1306)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1115)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:178)
at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:72)
at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:232)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:245)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-06-16 11:08:36,500 ERROR org.apache.hadoop.hive.ql.exec.Task: [HiveServer2-Background-Pool: Thread-480]: Failed to execute spark task, with exception 'java.lang.IllegalStateException(RPC channel is closed.)'
java.lang.IllegalStateException: RPC channel is closed.
at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at org.apache.hive.spark.client.rpc.Rpc.call(Rpc.java:276)
at org.apache.hive.spark.client.rpc.Rpc.call(Rpc.java:259)
at org.apache.hive.spark.client.SparkClientImpl$ClientProtocol.cancel(SparkClientImpl.java:532)
at org.apache.hive.spark.client.SparkClientImpl.cancel(SparkClientImpl.java:189)
at org.apache.hive.spark.client.JobHandleImpl.cancel(JobHandleImpl.java:62)
at org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.cancelJob(RemoteSparkJobRef.java:54)
at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:119)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1770)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1527)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1306)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1115)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:178)
at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:72)
at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:232)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:245)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-06-16 11:08:36,507 ERROR org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-480]: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
2016-06-16 11:08:36,507 INFO org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-480]: </PERFLOG method=Driver.execute start=1466089643640 end=1466089716507 duration=72867 from=org.apache.hadoop.hive.ql.Driver>
2016-06-16 11:08:36,507 INFO org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-480]: Completed executing command(queryId=hive_20160616110707_0b83aaa1-15cd-44de-bb24-717101b3a22a); Time taken: 72.867 seconds
2016-06-16 11:08:36,507 INFO org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-480]: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
2016-06-16 11:08:36,960 INFO org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-480]: </PERFLOG method=releaseLocks start=1466089716507 end=1466089716960 duration=453 from=org.apache.hadoop.hive.ql.Driver>
2016-06-16 11:08:36,961 ERROR org.apache.hive.service.cli.operation.Operation: [HiveServer2-Background-Pool: Thread-480]: Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:374)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:180)
at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:72)
at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:232)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:245)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Can anyone see something that will help me figure out what's going on? Any help is appreciated.

R

[#2]

hello,

1. try select count(*) from any table -- this forces hive to start spark subsystem.

2. check yarn rm  if there is any error.

[#3]

I ended up doubting my data load and resorted to re-installing the cluster. Turns out it could have been the data load but I'm getting a batch of new errors. Has *anyone* ever managed to get Hive on Spark working with CDH 5.7?
The reason I ask is there seem to be a lot of posts around claiming that Hive/Spark compatability is version sensitive. 
This is what appears to be baked into CDH 5.7.1
Hive 1.1.0-cdh5.7.1
Spark version is 1.6
 

Running the suggested query I immediately am stymied by this;
 

ERROR : Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to creat                                                                       e spark client.)'

[#4]

The new cluster/data load actually worked. I was making mistakes with beeline authentication. This is solved.

[#5]

Hi @rdub, can you please post how did you solve it? Since I'm facing a set of similar issues, and I either end up with "Error, return code 1" or "Error, return code 2".

Thanks a lot,
M

[#6]

Sorry, that was several clusters ago. Initially I had some problems with my data load script. I rebuilt the cluster and reloaded the sample data we used. The second issue IIRC, I was using an improperly formed string when connecting.

[#7]

Thanks for the tip - I'll double-check my connection string.

[#8]

I have the same problem, have you resolved it?

[#9]

No, unfortunately not - still having it.

@invoker: Which versions do you have on your setup?

[#10]

The version of Hive: cdh5-1.1.0_5.7.1 and Spark: chd5-1.6.0_5.7.1 The state of SparkJob is always" SENT" and after 60 seconds the job failed.