Hi, I have a Hadoop cluster with 3 data nodes with the following versions. HDP - 2.5.3 Spark - 1.6.2 sbt version - 0.13.13 I am trying to write a data frame (a very basic data frame of 5 rows and 2 columns) into a file in hdfs using spark-submit. I get the below error (in Italics). I could only see _temporary folder at the destination path, containing partitions. It is not able to delete the contents of temporary folder and write the output permanently. However, when I try to write a RDD to text file in hdfs, it successfully writes the output. Error- attempt_xyz: not committed because the driver did not authorize commit. Task was denied committing. Task attemt_xyz aborted. Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times, most recent failure: Lost task 1.151 in stage 0.0 (TID 300, localhost): org.apache.spark.SparkException: Task failed while writing rows at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:269) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$anonfun$run$1$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:148) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$anonfun$run$1$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:148) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Failed to commit task at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.org$apache$spark$sql$execution$datasources$DefaultWriterContainer$commitTask$1(WriterContainer.scala:283) at org.apache.spark.sql.execution.datasources.DefaultWriterContainer$anonfun$writeRows$1.apply$mcV$sp(WriterContainer.scala:265) at org.apache.spark.sql.execution.datasources.DefaultWriterContainer$anonfun$writeRows$1.apply(WriterContainer.scala:260) at org.apache.spark.sql.execution.datasources.DefaultWriterContainer$anonfun$writeRows$1.apply(WriterContainer.scala:260) at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1277) at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:266) ... 8 more Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on ABC/day=2017-02-16/hour=15/_temporary/0/_temporary/attempt_201702161507_0000_m_000001_151/part-r-00001-3b30ab0b-dee9-429f-8a0a-f8e5704a6cc8 (inode 199900): File does not exist. Holder DFSClient_attempt_201702161507_0000_m_000001_151_2120850886_197 does not have any open files. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3521) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3611) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3578) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:905) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:544) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552) at org.apache.hadoop.ipc.Client.call(Client.java:1496) at org.apache.hadoop.ipc.Client.call(Client.java:1396) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at com.sun.proxy.$Proxy13.complete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:501) at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176) at com.sun.proxy.$Proxy14.complete(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2361) at org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2338) at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2303) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:111) at org.apache.spark.sql.execution.datasources.text.TextOutputWriter.close(DefaultSource.scala:168) at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.org$apache$spark$sql$execution$datasources$DefaultWriterContainer$commitTask$1(WriterContainer.scala:275) ... 13 more Kindly suggest the root cause of the issue with possible solution. ... View more

Hi, I have a Hadoop cluster with 3 data nodes with the following versions. HDP - 2.5.3 Spark - 1.6.2 sbt version - 0.13.12 I am trying to write a data frame (a very basic data frame of 5 rows and 2 columns) into a file in hdfs using spark-submit. I get the below error (in Italics). I could only see _temporary folder at the destination path, containing partitions. It is not able to delete the contents of temporary folder and write the output permanently. However, when I try to write a RDD to text file in hdfs, it successfully writes the output. Error- attempt_xyz: not committed because the driver did not authorize commit. Task was denied committing. Task attemt_xyz aborted. Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times, most recent failure: Lost task 1.151 in stage 0.0 (TID 300, localhost): org.apache.spark.SparkException: Task failed while writing rows at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:269) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$anonfun$run$1$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:148) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$anonfun$run$1$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:148) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Failed to commit task at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.org$apache$spark$sql$execution$datasources$DefaultWriterContainer$commitTask$1(WriterContainer.scala:283) at org.apache.spark.sql.execution.datasources.DefaultWriterContainer$anonfun$writeRows$1.apply$mcV$sp(WriterContainer.scala:265) at org.apache.spark.sql.execution.datasources.DefaultWriterContainer$anonfun$writeRows$1.apply(WriterContainer.scala:260) at org.apache.spark.sql.execution.datasources.DefaultWriterContainer$anonfun$writeRows$1.apply(WriterContainer.scala:260) at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1277) at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:266) ... 8 more Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on ABC/day=2017-02-16/hour=15/_temporary/0/_temporary/attempt_201702161507_0000_m_000001_151/part-r-00001-3b30ab0b-dee9-429f-8a0a-f8e5704a6cc8 (inode 199900): File does not exist. Holder DFSClient_attempt_201702161507_0000_m_000001_151_2120850886_197 does not have any open files. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3521) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3611) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3578) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:905) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:544) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552) at org.apache.hadoop.ipc.Client.call(Client.java:1496) at org.apache.hadoop.ipc.Client.call(Client.java:1396) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at com.sun.proxy.$Proxy13.complete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:501) at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176) at com.sun.proxy.$Proxy14.complete(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2361) at org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2338) at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2303) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:111) at org.apache.spark.sql.execution.datasources.text.TextOutputWriter.close(DefaultSource.scala:168) at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.org$apache$spark$sql$execution$datasources$DefaultWriterContainer$commitTask$1(WriterContainer.scala:275) ... 13 more Kindly suggest the root cause of the issue with possible solution. ... View more

hi, I'm trying to execute insert query on hive tables from query editor.I get below error in error log. Please help. Mentioned some settings and properties ,might be helpful for you to understand. I'm using HDP 2.4.2 with 4 datanodes (each with 5 TB storage and 64GB RAM) and 1 namenode. tez container size = 4096 MB number of containers held = 3 Hive.auto.convert.join.noconditionaltask.size=1512MB metastore heap size = 8192MB tez.am.resource.memory.mb = 8192MB tez.task.resource.memory.mb = 4096 MB hive.tez.java.opts = 3274 mapreduce.task.io.sort.mb =2047MB Error:: INFO : Session is already open INFO : Dag name: insert into evigil_firewall_details_um...10(Stage-1) INFO : Tez session was closed. Reopening... INFO : Session re-established. INFO : INFO : Status: Running (Executing on YARN cluster with App id application_1484654153148_0008) INFO : Map 1: -/-Reducer 2: 0/1Reducer 3: 0/24 INFO : Map 1: 0/1Reducer 2: 0/1Reducer 3: 0/24 INFO : Map 1: 0(+0,-1)/1Reducer 2: 0/1Reducer 3: 0/24 INFO : Map 1: 0(+0,-2)/1Reducer 2: 0/1Reducer 3: 0/24 INFO : Map 1: 0(+0,-3)/1Reducer 2: 0/1Reducer 3: 0/24 ERROR : Status: Failed ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1484654153148_0008_1_00, diagnostics=[Task failed, taskId=task_1484654153148_0008_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Container container_e23_1484654153148_0008_01_000002 finished with diagnostics set to [Container failed, exitCode=1. Exception from container-launch. Container id: container_e23_1484654153148_0008_01_000002 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 ]], TaskAttempt 1 failed, info=[Container container_e23_1484654153148_0008_01_000003 finished with diagnostics set to [Container failed, exitCode=1. Exception from container-launch. Container id: container_e23_1484654153148_0008_01_000003 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 ]], TaskAttempt 2 failed, info=[Container container_e23_1484654153148_0008_01_000004 finished with diagnostics set to [Container failed, exitCode=1. Exception from container-launch. Container id: container_e23_1484654153148_0008_01_000004 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 ]], TaskAttempt 3 failed, info=[Container container_e23_1484654153148_0008_01_000005 finished with diagnostics set to [Container failed, exitCode=1. Exception from container-launch. Container id: container_e23_1484654153148_0008_01_000005 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 ]]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1484654153148_0008_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE] ERROR : Vertex killed, vertexName=Reducer 3, vertexId=vertex_1484654153148_0008_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:24, Vertex vertex_1484654153148_0008_1_02 [Reducer 3] killed/failed due to:OTHER_VERTEX_FAILURE] ERROR : Vertex killed, vertexName=Reducer 2, vertexId=vertex_1484654153148_0008_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_1484654153148_0008_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE] ERROR : DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:2 ... View more

i'm trying to fetch the data from external table and insert it into internal table. This external table points to a folder having json files in it. i have written the query in file and used it in workflow. Both the tables have same columns and datatype.when i run my query in query editor it works.But when i run the same query by adding it to a workflow it gives me below error. I have set some workflow properties as shown in image. I have also added below jar file file in '/user/oozie/share/lib/lib_20160808184114/hive' path. I have hortanworks 2.4.2,oozie 4.2.0 ,Hive-Hcatalog 1.2.1000 , Hue 2.6.1-258 json-serde-1.3.6-jar-with-dependencies.jar , mysql-connector-java.jar , hive-serde-1.2.1000.2.4.2.0-258.jar ,hive-hcatalog-core-0.13.1.jar ERROR:: INFO [ATS Logger 0] impl.TimelineClientImpl (TimelineClientImpl.java:logException(273)) - Exception caught by TimelineClientConnectionRetry, will try 29 more time(s). Message: java.net.ConnectException: Connection refused 2017-01-13 13:45:08,209 INFO [ATS Logger 0] impl.TimelineClientImpl (TimelineClientImpl.java:logException(273)) - Exception caught by TimelineClientConnectionRetry, will try 4 more time(s). Message: java.net.ConnectException: Connection refused 2017-01-13 13:45:09,215 INFO [ATS Logger 0] impl.TimelineClientImpl (TimelineClientImpl.java:logException(273)) - Exception caught by TimelineClientConnectionRetry, will try 3 more time(s). Message: java.net.ConnectException: Connection refused 2017-01-13 13:45:09,618 INFO [main] SessionState (SessionState.java:printInfo(953)) - Map 1: 0(+0,-10)/3Reducer 2: 0/1Reducer 3: 0/1 2017-01-13 13:45:09,826 ERROR [main] SessionState (SessionState.java:printError(962)) - Status: Failed 2017-01-13 13:45:09,828 ERROR [main] SessionState (SessionState.java:printError(962)) - Vertex failed, vertexName=Map 1, vertexId=vertex_1483604431963_0816_1_00, diagnostics=[Task failed, taskId=task_1483604431963_0816_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator initialization failed 2017-01-13 13:45:08,209 INFO [ATS Logger 0] impl.TimelineClientImpl (TimelineClientImpl.java:logException(273)) - Exception caught by TimelineClientConnectionRetry, will try 4 more time(s). Message: java.net.ConnectException: Connection refused 2017-01-13 13:45:09,215 INFO [ATS Logger 0] impl.TimelineClientImpl (TimelineClientImpl.java:logException(273)) - Exception caught by TimelineClientConnectionRetry, will try 3 more time(s). Message: java.net.ConnectException: Connection refused 2017-01-13 13:45:09,618 INFO [main] SessionState (SessionState.java:printInfo(953)) - Map 1: 0(+0,-10)/3Reducer 2: 0/1Reducer 3: 0/1 2017-01-13 13:45:09,826 ERROR [main] SessionState (SessionState.java:printError(962)) - Status: Failed 2017-01-13 13:45:09,828 ERROR [main] SessionState (SessionState.java:printError(962)) - Vertex failed, vertexName=Map 1, vertexId=vertex_1483604431963_0816_1_00, diagnostics=[Task failed, taskId=task_1483604431963_0816_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:265) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.data.JsonSerDe not found at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:347) at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:382) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:227) ... 15 more Caused by: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.data.JsonSerDe not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:143) at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:313) ... 17 more ], TaskAttempt 1 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator initialization failed ... View more

I am using Hortonworks Hive ODBC Driver for Windows in C#.net code. Getting below error while inserting the data in Hive Table (used single Column Table) “ERROR [HY000] [Hortonworks][Hardy] (35) Error from server: error code: '0' error message: 'ExecuteStatement finished with operation state: ERROR_STATE'” Am able to select data and create table in HIVE with above ODBC driver. Code:- cmd.CommandText = "INSERT INTO TABLE SqlConTest VALUES('Mahendra')"; int noofrows=cmd.ExecuteNonQuery(); ... View more

I need to pull sql server data with the help of ODBC driver in Oozie (Sqoop) workflow. My sql server is running with integrated security. I have used Integrated Security=SSPI / Trusted_Connection = yes but getting error. ... View more