Member since
08-04-2016
19
Posts
1
Kudos Received
0
Solutions
04-30-2018
01:45 PM
Issue : Kafka Connect is not starting in STANDALONE mode in HDP 2.5.6 development cluster. Below are the steps I've followed. Description : Before that, I'd like to tell you that normal publishing/consuming are working with kafka topics. I've tested it fully. Now, lets begin. Main objective of this activity is to create a ETL data pipeline using Kafka Connect API like, we need to start Kafka Connect (in STANDALONE mode) to look a file, push new data into Kafka Topic and then sink into another file. As simple as that. Below are the steps I've followed. Step 1: Creating a properties file to configure source - a file. fsbappdev@DG1357> vi /home/fsbappdev/gnans/kafka_connect_source_file.properties
fsbappdev@DG1357> cat /home/fsbappdev/gnans/kafka_connect_source_file.properties
name=local-file-source
connector.class=FileStreamSource
tasks.max=1
file=/home/fsbappdev/gnans/kafka_connect_source_file.txt
topic=credit_risk_topic
Setup 2 : Creating a properties file to configure sink - a file fsbappdev@DG1357> vi /home/fsbappdev/gnans/kafka_connect_sink_file.properties
fsbappdev@DG1357> cat /home/fsbappdev/gnans/kafka_connect_sink_file.properties
name=local-file-sink
connector.class=FileStreamSink
tasks.max=1
file=/home/fsbappdev/gnans/kafka_connect_sink_file.txt
topics=credit_risk_topic
Step 3: Creating a properties file to start Kafka Connect in Standalone mode. fsbappdev@DG1357> cat /home/fsbappdev/gnans/kafka_connect_standalone.properties
bootstrap.servers=dg1358:6667
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
internal.key.converter=org.apache.kafka.connect.storage.StringConverter
internal.value.converter=org.apache.kafka.connect.storage.StringConverter
offset.storage.file.filename=/home/fsbappdev/gnans/kafka_connect.offsets
Step 4: Starting Kafka Connect (in Standalone Mode) using the list of properties file which I've created in above steps. sh connect-standalone.sh /home/fsbappdev/gnans/kafka_connect_standalone.properties /home/fsbappdev/gnans/kafka_connect_source_file.properties /home/fsbappdev/gnans/kafka_connect_sink_file.properties Error Stacktrace: Upon executing the above command, i encounter the below error: [2018-04-30 19:08:53,240] WARN FAILED o.e.j.s.ServletContextHandler@229f66ed{/,null,STARTING}: java.lang.NoSuchMethodError: javax.ws.rs.core.Application.getProperties()Ljava/util/Map; (org.eclipse.jetty.util.component.AbstractLifeCycle:212)
java.lang.NoSuchMethodError: javax.ws.rs.core.Application.getProperties()Ljava/util/Map;
at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:331)
at org.glassfish.jersey.servlet.WebComponent.<init>(WebComponent.java:392)
at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:177)
at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:369)
at javax.servlet.GenericServlet.init(GenericServlet.java:241)
at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:616)
at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:396)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:871)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:298)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.StatisticsHandler.doStart(StatisticsHandler.java:232)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.server.Server.start(Server.java:387)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:354)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.kafka.connect.runtime.rest.RestServer.start(RestServer.java:145)
at org.apache.kafka.connect.runtime.Connect.start(Connect.java:56)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:82)
Though it looks likes a WARN, its failure. I see last few lines of console as follows: [2018-04-30 19:08:53,253] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:68)
[2018-04-30 19:08:53,253] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:154)
[2018-04-30 19:08:53,257] INFO Stopped ServerConnector@ba2f4ec{HTTP/1.1}{0.0.0.0:8083} (org.eclipse.jetty.server.ServerConnector:306)
[2018-04-30 19:08:53,259] INFO Stopped o.e.j.s.ServletContextHandler@229f66ed{/,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:865)
[2018-04-30 19:08:53,259] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:165)
[2018-04-30 19:08:53,260] INFO Herder stopping (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:77)
[2018-04-30 19:08:53,260] INFO Worker stopping (org.apache.kafka.connect.runtime.Worker:128)
[2018-04-30 19:08:53,260] WARN Shutting down tasks [] uncleanly; herder should have shut down tasks before the Worker is stopped. (org.apache.kafka.connect.runtime.Worker:141)
[2018-04-30 19:08:53,260] INFO Stopped FileOffsetBackingStore (org.apache.kafka.connect.storage.FileOffsetBackingStore:68)
[2018-04-30 19:08:53,260] INFO Worker stopped (org.apache.kafka.connect.runtime.Worker:151)
[2018-04-30 19:08:56,964] INFO Reflections took 3837 ms to scan 109 urls, producing 6344 keys and 39164 values (org.reflections.Reflections:229)
[2018-04-30 19:08:56,974] INFO Herder stopped (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:91)
[2018-04-30 19:08:56,974] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:73)
fsbappdev@DG1357:/usr/hdp/current/kafka-broker/bin>
To me, it looks like it is due to jar files compatibility. But, I dont know which jar would suit well among them. My Hadoop/Kafka Cluster is being managed by another team and I need to give them some advice to fix it. Can anyone please help me on this? Note : I've attached ( kafka-connect-filesource-filesink-errorlog-2.txt ) full error stacktrace in this question to get help.
... View more
Labels:
08-29-2017
09:29 AM
Hi All, I have HDP 2.3.2 in production cluster and I have some INSERT-SELECT query to run to populate data from a set of tables to a table and since last week, I get a weird issue - java.lang.RuntimeException: Hive Runtime Error while closing operators: Unable to rename output from: hdfs:// How should I attack this problem to solve it permanently? Should I run this query with MR engine rather than Tez? Please throw me some light. Map 1: 2/2 Map 12: 2/2 Map 14: 2/2 Map 16: 2/2 Map 18: 3/3 Map 19: 2/2 Map 21: 3/3 Map 25: 2/2 Map 26: 2/2 Map 27: 3/3 Map 3: 2/2 M\
ap 5: 3/3 Reducer 10: 1/1 Reducer 11: 0(+1,-3)/111 Reducer 13: 1/1 Reducer 15: 1/1 Reducer 17: 1/1 Reducer 2: 1/1 Reducer 20: 1/1 R\
educer 22: 37/37 Reducer 23: 1/1 Reducer 24: 412/412 Reducer 4: 1/1 Reducer 7: 221/221 Reducer 9: 221/221
Status: Failed
Vertex failed, vertexName=Reducer 11, vertexId=vertex_1503954544552_1870_1_24, diagnostics=[Task failed, taskId=task_1503954544552_\
1870_1_24_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.\
RuntimeException: Hive Runtime Error while closing operators: Unable to rename output from: hdfs://STANCDR1TDH/apps/hive/warehouse/\
crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tmp.-ext-10002/_tmp.000\
000_0 to: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_79\
42865519677265277-1/_tmp.-ext-10002/000000_0
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators: Unable to rename output from: hdfs://STANCDR1TDH\
/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tm\
p.-ext-10002/_tmp.000000_0 to: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-\
08-29_14-34-17_200_7942865519677265277-1/_tmp.-ext-10002/000000_0
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:310)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162)
... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://STANCDR1TDH/apps/hive/warehouse/c\
rhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tmp.-ext-10002/_tmp.0000\
00_0 to: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_794\
2865519677265277-1/_tmp.-ext-10002/000000_0
at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:237)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$200(FileSinkOperator.java:143)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1051)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:631)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:287)
... 15 more
], TaskAttempt 1 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runti\
me Error while closing operators: Unable to rename output from: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_co\
mmentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tmp.-ext-10002/_tmp.000000_1 to: hdfs://STANCDR1TDH\
/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_tmp.-ex\
t-10002/000000_1
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators: Unable to rename output from: hdfs://STANCDR1TDH\
/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tm\
p.-ext-10002/_tmp.000000_1 to: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-\
08-29_14-34-17_200_7942865519677265277-1/_tmp.-ext-10002/000000_1
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:310)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162)
... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://STANCDR1TDH/apps/hive/warehouse/c\
rhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tmp.-ext-10002/_tmp.0000\
00_1 to: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_794\
2865519677265277-1/_tmp.-ext-10002/000000_1
at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:237)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$200(FileSinkOperator.java:143)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1051)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:631)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:287)
... 15 more
], TaskAttempt 2 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runti\
me Error while closing operators: Unable to rename output from: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_co\
mmentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tmp.-ext-10002/_tmp.000000_2 to: hdfs://STANCDR1TDH\
/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_tmp.-ex\
t-10002/000000_2
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators: Unable to rename output from: hdfs://STANCDR1TDH\
/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tm\
p.-ext-10002/_tmp.000000_2 to: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-\
08-29_14-34-17_200_7942865519677265277-1/_tmp.-ext-10002/000000_2
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:310)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162)
... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://STANCDR1TDH/apps/hive/warehouse/c\
rhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tmp.-ext-10002/_tmp.0000\
00_2 to: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_794\
2865519677265277-1/_tmp.-ext-10002/000000_2
at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:237)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$200(FileSinkOperator.java:143)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1051)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:631)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:287)
... 15 more
], TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runti\
me Error while closing operators: Unable to rename output from: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_co\
mmentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tmp.-ext-10002/_tmp.000000_3 to: hdfs://STANCDR1TDH\
/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_tmp.-ex\
t-10002/000000_3
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators: Unable to rename output from: hdfs://STANCDR1TDH\
/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tm\
p.-ext-10002/_tmp.000000_3 to: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-\
08-29_14-34-17_200_7942865519677265277-1/_tmp.-ext-10002/000000_3
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:310)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162)
... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://STANCDR1TDH/apps/hive/warehouse/c\
rhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tmp.-ext-10002/_tmp.0000\
00_3 to: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_794\
2865519677265277-1/_tmp.-ext-10002/000000_3
at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:237)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$200(FileSinkOperator.java:143)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1051)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:631)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:287)
... 15 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:110, Vertex vertex_1503954544552_1870_1_24 [Reducer 1\
1] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Reducer 11, verte\
xId=vertex_1503954544552_1870_1_24, diagnostics=[Task failed, taskId=task_1503954544552_1870_1_24_000000, diagnostics=[TaskAttempt \
0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while \
closing operators: Unable to rename output from: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-\
staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tmp.-ext-10002/_tmp.000000_0 to: hdfs://STANCDR1TDH/apps/hive/ware\
house/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_tmp.-ext-10002/000000_\
0
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators: Unable to rename output from: hdfs://STANCDR1TDH\
/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tm\
p.-ext-10002/_tmp.000000_0 to: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-\
08-29_14-34-17_200_7942865519677265277-1/_tmp.-ext-10002/000000_0
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:310)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162)
... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://STANCDR1TDH/apps/hive/warehouse/c\
rhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tmp.-ext-10002/_tmp.0000\
00_0 to: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_794\
2865519677265277-1/_tmp.-ext-10002/000000_0
at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:237)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$200(FileSinkOperator.java:143)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1051)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:631)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:287)
... 15 more
], TaskAttempt 1 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runti\
me Error while closing operators: Unable to rename output from: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_co\
mmentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tmp.-ext-10002/_tmp.000000_1 to: hdfs://STANCDR1TDH\
/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_tmp.-ex\
t-10002/000000_1
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators: Unable to rename output from: hdfs://STANCDR1TDH\
/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tm\
p.-ext-10002/_tmp.000000_1 to: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-\
08-29_14-34-17_200_7942865519677265277-1/_tmp.-ext-10002/000000_1
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:310)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162)
... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://STANCDR1TDH/apps/hive/warehouse/c\
rhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tmp.-ext-10002/_tmp.0000\
00_1 to: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_794\
2865519677265277-1/_tmp.-ext-10002/000000_1
at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:237)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$200(FileSinkOperator.java:143)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1051)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:631)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:287)
... 15 more
], TaskAttempt 2 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runti\
me Error while closing operators: Unable to rename output from: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_co\
mmentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tmp.-ext-10002/_tmp.000000_2 to: hdfs://STANCDR1TDH\
/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_tmp.-ex\
t-10002/000000_2
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators: Unable to rename output from: hdfs://STANCDR1TDH\
/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tm\
p.-ext-10002/_tmp.000000_2 to: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-\
08-29_14-34-17_200_7942865519677265277-1/_tmp.-ext-10002/000000_2
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:310)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162)
... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://STANCDR1TDH/apps/hive/warehouse/c\
rhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tmp.-ext-10002/_tmp.0000\
00_2 to: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_794\
2865519677265277-1/_tmp.-ext-10002/000000_2
at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:237)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$200(FileSinkOperator.java:143)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1051)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:631)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:287)
... 15 more
], TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runti\
me Error while closing operators: Unable to rename output from: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_co\
mmentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tmp.-ext-10002/_tmp.000000_3 to: hdfs://STANCDR1TDH\
/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_tmp.-ex\
t-10002/000000_3
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators: Unable to rename output from: hdfs://STANCDR1TDH\
/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tm\
p.-ext-10002/_tmp.000000_3 to: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-\
08-29_14-34-17_200_7942865519677265277-1/_tmp.-ext-10002/000000_3
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:310)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162)
... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://STANCDR1TDH/apps/hive/warehouse/c\
rhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_7942865519677265277-1/_task_tmp.-ext-10002/_tmp.0000\
00_3 to: hdfs://STANCDR1TDH/apps/hive/warehouse/crhsctl.db/crhs_sci_break_commentary/.hive-staging_hive_2017-08-29_14-34-17_200_794\
2865519677265277-1/_tmp.-ext-10002/000000_3
at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:237)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$200(FileSinkOperator.java:143)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1051)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:631)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:287)
... 15 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:110, Vertex vertex_1503954544552_1870_1_24 [Reducer 1\
1] killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Tez
06-15-2017
10:41 AM
thank you @Frank Welsch. I'll check
... View more
06-15-2017
10:40 AM
thank you @Dennis Connolly. I'll check
... View more
06-12-2017
12:04 PM
Hi Friends, We are facing a
problem in monitoring the Hive queries execution. Basically, we need to have a tool or method to check the list of hive queries we're executing and performance of each query which ran in
past. Cluster Admin is having tools to
monitor the currently running queries only. Do you happen to know
what tools can help here?
... View more
Labels:
- Labels:
-
Apache Hive
04-02-2017
03:58 PM
now, i've solved this issue by setting below property. Now, all mapper/reducer's output will not be stored in memory. But, i need to revisit my table data and predicates(where clause) once again to check if any unnecessary data is fetched. set hive.auto.convert.join=false;
... View more
04-02-2017
03:55 PM
Now, i've solved this issue by setting below property. Now, all mapper/reducer's output will not be stored in memory. But, i need to revisit my table data and predicates(where clause) once again to check if any unnecessary data is fetched. set hive.auto.convert.join=false;
... View more
04-02-2017
05:45 AM
Hi @Alind Billore How much memory you increased for this property? I too face this issue with below settings. set hive.auto.convert.join.noconditionaltask.size=3300000000;
... View more
04-01-2017
01:14 PM
Hi All, I too face this issue in production and here is my error and production hive settings. Execution log at: /tmp/crhscrvs/crhscrvs_20170401171447_7fa9db9e-7265-4844-a325-0e11b8e2e2c5.log 2017-04-01
17:18:50 Starting to launch local task to process map
join; maximum memory = 2130968576 2017-04-01
17:18:53 Processing
rows: 200000 Hashtable size:
199999 Memory usage:
180192576
percentage: 0.085 2017-04-01
17:18:53 Processing
rows: 300000 Hashtable size:
299999 Memory usage:
203985896
percentage: 0.096 2017-04-01
17:18:54 Processing
rows: 400000 Hashtable size:
399999 Memory usage:
247108088
percentage: 0.116 2017-04-01
17:18:55 Processing
rows: 500000 Hashtable size:
499999 Memory usage:
329110392
percentage: 0.154 2017-04-01
17:18:55 Processing
rows: 600000 Hashtable size:
599999 Memory usage:
347313416
percentage: 0.163 2017-04-01
17:18:55 Processing
rows: 700000 Hashtable size:
699999 Memory usage:
410839712
percentage: 0.193 2017-04-01
17:18:55 Processing rows:
800000 Hashtable size: 799999 Memory usage:
453803856
percentage: 0.213 2017-04-01
17:18:56 Processing
rows: 900000 Hashtable size:
899999 Memory usage:
528026968
percentage: 0.248 2017-04-01
17:18:56 Processing
rows: 1000000 Hashtable size:
999999 Memory usage:
564196224
percentage: 0.265 2017-04-01
17:18:56 Processing
rows: 1100000 Hashtable size: 1099999
Memory usage: 592163176
percentage: 0.278 2017-04-01 17:18:57
Processing rows: 1200000 Hashtable
size: 1199999 Memory usage: 658466272
percentage: 0.309 2017-04-01
17:18:57 Processing
rows: 1300000 Hashtable size: 1299999
Memory usage: 699296984
percentage: 0.328 2017-04-01
17:18:57 Processing
rows: 1400000 Hashtable size: 1399999
Memory usage: 759936160
percentage: 0.357 2017-04-01
17:18:58 Processing
rows: 1500000 Hashtable size: 1499999
Memory usage: 846875144
percentage: 0.397 2017-04-01
17:18:59 Processing
rows: 1600000 Hashtable size: 1599999
Memory usage: 863823240
percentage: 0.405 2017-04-01
17:18:59 Processing
rows: 1700000 Hashtable size: 1699999
Memory usage: 923698304
percentage: 0.433 2017-04-01
17:19:00 Processing
rows: 1800000 Hashtable size: 1799999
Memory usage: 998273304
percentage: 0.468 2017-04-01
17:19:00 Processing
rows: 1900000 Hashtable size: 1899999
Memory usage: 1009902104
percentage: 0.474 2017-04-01
17:19:00 Processing
rows: 2000000 Hashtable size: 1999999
Memory usage: 1080755328
percentage: 0.507 2017-04-01
17:19:01 Processing
rows: 2100000 Hashtable size: 2099999
Memory usage: 1118238920
percentage: 0.525 2017-04-01
17:19:01 Processing
rows: 2200000 Hashtable size: 2199999
Memory usage: 1147275760
percentage: 0.538 2017-04-01
17:19:01 Processing
rows: 2300000 Hashtable size: 2299999
Memory usage: 1214495864
percentage: 0.57 Execution failed with exit
status: 3 Obtaining error information Task failed! Task ID: Stage-55 Logs: /tmp/crhscrvs/hive.log Production Hive Settings: hive> set hive.cbo.enable; hive.cbo.enable=true hive> set
hive.stats.autogather; hive.stats.autogather=true hive> set
hive.stats.fetch.column.stats; hive.stats.fetch.column.stats=false hive> set
hive.stats.fetch.partition.stats; hive.stats.fetch.partition.stats=true hive> set hive.tez.java.opts; hive.tez.java.opts=-server
-Xmx3072m -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA
-XX:+UseParallelGC hive> set hive.auto.convert.join.noconditionaltask; hive.auto.convert.join.noconditionaltask=true hive> set
hive.auto.convert.join.noconditionaltask.size; hive.auto.convert.join.noconditionaltask.size=1561644237 hive> set
hive.exec.reducers.bytes.per.reducer; hive.exec.reducers.bytes.per.reducer=269798605 hive> set
hive.cli.print.header=true; I'm running Hive on Tez in HDP cluster - 2.3.2.0 For two months, it woked good and due to sudden data growth, i'm facing this memory issue. Exactly, am getting this error stacktrace: hive> set tez.task.resource.memory.mb=16384;
hive> set tez.am.resource.memory.mb=16384;
hive> set hive.tez.container.size=16384;
hive> insert overwrite table
crhs_fmtrade_break_latest_user_commentary partition(source_system)
> select break_id,
reporting_date, original_reporting_date, investigationstatus,
investigationcloseddate, userdefinedclassification,
> freeformatcomments, systeminvestigationstatus,
commentaryuploaddatetime, comment_id, commentarysourcesystem from
v_fmtrade_unique_latest_commentary_view;
Query ID =
crhscrvs_20170401164631_46f52aa6-7548-4625-9bcd-6c27f97ac207
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id
application_1490695811857_8269)
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED
RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map
1
KILLED
-1
0 0
-1 0 0
Map
11
KILLED
-1
0 0
-1 0 0
Map
13
KILLED
-1
0
0 -1
0 0
Map 16 .........
SUCCEEDED
2
2
0
0 0 0
Map
17
KILLED
-1
0 0
-1 0 0
Map 18
KILLED
-1
0
0 -1
0 0
Map
20
FAILED
-1
0 0
-1 0 0
Map 25 .........
SUCCEEDED
2
2
0
0 0 0
Map
26
KILLED
-1
0 0
-1 0 0
Map
27
FAILED
-1
0 0
-1 0 0
Map 4 ..........
SUCCEEDED
2
2
0
0 0 1
Map
5
KILLED -1
0
0 -1
0 0
Reducer
10
KILLED
1
0
0
1 0 0
Reducer
12
KILLED
1
0
0
1 0 0
Reducer
14
KILLED
2
0
0
2 0 0
Reducer
15
KILLED
1
0
0
1 0 0
Reducer
19
KILLED
1
0
0
1 0 0
Reducer
2
KILLED
1
0 0
1 0 0
Reducer 21
KILLED 1009
0 0
1009 0
0
Reducer
22
KILLED
185
0 0
185 0 0
Reducer
23
KILLED
1
0
0
1 0 0
Reducer
24 KILLED
1009
0 0
1009 0
0
Reducer
3
KILLED
1
0 0
1 0 0
Reducer
7
KILLED 1009
0 0
1009 0
0
Reducer 9
KILLED
1009
0 0
1009 0
0
--------------------------------------------------------------------------------
VERTICES: 03/25
[>>--------------------------] 0% ELAPSED TIME: 395.80
s
--------------------------------------------------------------------------------
Status: Failed
Vertex failed, vertexName=Map 20,
vertexId=vertex_1490695811857_8269_1_11, diagnostics=[Vertex
vertex_1490695811857_8269_1_11 [Map 20] killed/failed due
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: f1 initializer failed,
vertex=vertex_1490695811857_8269_1_11 [Map 20], java.lang.RuntimeException:
serious problem
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1025)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1052)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:305)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:407)
at
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:155)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:246)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:240)
at
java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:240)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:227)
at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException:
java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
at
java.util.concurrent.FutureTask.report(FutureTask.java:122)
at
java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1002)
... 15 more
Caused by: java.io.IOException: Couldn't create proxy
provider class
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
at
org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:515)
at
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:170)
at
org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
at
org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
at
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
at
org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at
org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:354)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:638)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:624)
... 4 more
Caused by: java.lang.reflect.InvocationTargetException
at
sun.reflect.GeneratedConstructorAccessor23.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:498)
... 14 more
Caused by: java.lang.OutOfMemoryError: GC overhead
limit exceeded
at
java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1019)
at
java.util.concurrent.ConcurrentHashMap.putAll(ConcurrentHashMap.java:1084)
at
java.util.concurrent.ConcurrentHashMap.<init>(ConcurrentHashMap.java:852)
at
org.apache.hadoop.conf.Configuration.<init>(Configuration.java:713)
at
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.<init>(ConfiguredFailoverProxyProvider.java:70)
... 18 more
]
Vertex failed, vertexName=Map 27,
vertexId=vertex_1490695811857_8269_1_17, diagnostics=[Vertex vertex_1490695811857_8269_1_17
[Map 27] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: r31
initializer failed, vertex=vertex_1490695811857_8269_1_17 [Map 27],
java.lang.RuntimeException: serious problem
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1025)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1052)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:305)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:407)
at
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:155)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:246)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:240)
at
java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:240)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:227)
at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException:
java.io.IOException: Couldn't create proxy provider class
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
at
java.util.concurrent.FutureTask.report(FutureTask.java:122)
at
java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1002)
... 15 more
Caused by: java.io.IOException: Couldn't create proxy
provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
at
org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:515)
at
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:170)
at
org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
at
org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
at
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
at
org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at
org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:354)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:638)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:624)
... 4 more
Caused by: java.lang.reflect.InvocationTargetException
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:498)
... 14 more
Caused by: java.lang.OutOfMemoryError: GC overhead
limit exceeded
at
java.util.Hashtable$Entry.clone(Hashtable.java:1250)
at
java.util.Hashtable.clone(Hashtable.java:550)
at
org.apache.hadoop.conf.Configuration.<init>(Configuration.java:706)
at
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.<init>(ConfiguredFailoverProxyProvider.java:70)
... 19 more
]
... View more
03-21-2017
10:53 AM
thank you Deepesh. It works for me.
... View more