Hi,   Recently we started seeing Unable to rename output files and its causing job failures and also found a bug not sure if its related   https://issues.apache.org/jira/browse/HIVE-4605   We are on Hive1.2.1000.2.5.3.0-37 HDP: 2.5.3.0-37   Unable to rename output from: /apps/hive/warehouse/<testxyx.db/.hive-staging_hive_2019-08-28_07-37-56_908_2490136324495723304-3/_task_tmp.-ext-10002/_tmp.000079_0 to: /apps/hive/warehouse/testxyx.db.db/.hive-staging_hive_2019-08-28_07-37-56_908_2490136324495723304-3/_tmp.-ext-10002/000079_0  Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: /apps/hive/warehouse/testxyz.db/.hive-staging_hive_2019-08-28_06-38-58_174_8774387112611734538-10/_task_tmp.-ext-10002/_tmp.000048_3 to: /apps/hive/warehouse/testxyz.db/.hive-staging_hive_2019-08-28_06-38-58_174_8774387112611734538-10/_tmp.-ext-10002/000048_3 at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:241) at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$200(FileSinkOperator.java:145) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1062) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:631) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:631) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:290) ... 15 more ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:504, Vertex vertex_1563822885587_4235051_1_66 [Reducer 15] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Map 65, vertexId=vertex_1563822885587_4235051_1_15, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_1563822885587_4235051_1_15 [Map 65] killed/failed due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 18, vertexId=vertex_1563822885587_4235051_1_10, ... View more

Hello Everyone, The Oozie coordinator 0047782-180612065206168-oozie-oozi-C is showing Status: DONEWITHERROR and no new instances are getting created. This is a INTRADAY coordinator and run 4 times a day. Our jobs are scheduled 360 mins daily at below GMT timings. 07:00 13:00 19:00 01:00 Because of this issue our latest 13:00 is not yet created Background on this :The coordinator should be in running state if the workflow is KILLED or FAILED. We have so many coordinators running in which some of the workflow will fail or killed but it has never put Oozie coordinator job with DONEWITHERROR. HDP version: 2.5.3.0-37 ... View more

Hive is fetching different results, if we query on a specific partition its giving correct results, but when we do the query on a specific range its giving incorrect results. when we query on specific date range its fetching correct results date_dt (DATE'2018-03-31',DATE'2018-05-19',DATE'2018-04-28',DATE'2018-03-17') but when we do the query on specific range, its yielding incorrect results date_dt BETWEEN DATE'2017-07-08' AND DATE'2018-07-07' ... View more

Hello Everyone, Is there a way we can identify resources wasted or used by Tez container, for example if we set "hive.tez.continer.size=8192" how do we identify how much of memory being used and wasted. In our environment we have Dr.Elephant but shows resources usage/wastage only for MR jobs as we don't have ATS enabled, is there a way or work-around to monitor resource usage and wastage from Yarn logs or any other way. ... View more

screen-shot-2018-06-19-at-104245-pm.png Count distinct doesn't always give me the right answer. I've attached two different queries that should both result in 7 unique items purchased. If I don't do an operation on mdse_item_i like cast it to a bigint, it doesn't always count them correctly. to explain it simply, when i do cast on "mdse_itcountsdistinct-1.txtdoesnotcountdistinct.txtem_i" it gives unique results as 7, but when i don't do casting it gives unique results as 10 which is not correct. hive> select * from dfr_distinct; OK 100000000938 5 7 12.33 2 2.75 4.27 8.060 2 8 0 Time taken: 0.479 seconds, Fetched: 1 row(s) hive> select * from dfr_distinctnot; OK 100000000938 5 10 12.33 2 2.75 4.27 8.06 0 2 8 0 Time taken: 0.932 seconds, Fetched: 1 row(s) tried running the query in both MR and Tez modes still giving same results when i don't do casting. ... View more

Hello HCC, We are reading data/messages from Kafka and writing into HBase, while writing data into HBase we are seeing below exceptions in the logs. As mentioned in one of the HCC article from HortonWorks engineers we ran HBase Hbck and repair holes command but none of them helped we are still seeing exceptions. The namespace got over 396 online regions. It was working fine for last couple of months, but recently we started to see this exceptions. we are HDP:2.5.3.0-37 HBase:Version 1.1.2.2.5.3.0-37 is it something related to https://issues.apache.org/jira/browse/HBASE-17276 ? any help will be greatly appreciated @Kuldeep Kulkarni @Rohan Pednekar @Geoffrey Shelton Okot @Aditya Sirna @Jay Kumar SenSharma @kkanchu @Artem Ervits @Enis @Joe Young @Orlando Teixeira org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: org.apache.hadoop.hbase.exceptions.FailedSanityCheckException: Requested row out of range for doMiniBatchMutation on HRegion prodnamespace:contactdetails-v2,69498218,1518378381699.8b7a1df21ed54a47facb99845845305a., startKey='69498218', getEndKey()='697452052100488', row='535021032017430' at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:729) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:679) at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2056) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2127) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at java.lang.Thread.run(Thread.java:745) : 1 time, servers with issues: xyz.testing.com,16020,1518641778110, at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:260) at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$2400(AsyncProcess.java:240) at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1886) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:247) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:197) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1461) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1029) at com.tgt.move.WlmInvData$$anonfun$main$1$$anonfun$apply$1.apply(WlmInvData.scala:129) at com.tgt.move.WlmInvData$$anonfun$main$1$$anonfun$apply$1.apply(WlmInvData.scala:70) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2069) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2069) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:108) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ... View more

Hello HCC, We have 16 node cluster running on HDP 2.5.3.0, Ambari 2.5.1.0, on this cluster we allocated resource manager heap to 4G and when there is no single job running in the cluster for almost 5-6hrs, but still resource manager heap usage is going to 80-85% always. We have another big production cluster with 800 data nodes in that cluster RM usage will be under 25-30% always.On what basis heap usage is utilized on the cluster ? and how can we control heap usage when cluster is idle. i also checked this property "yarn.resourcemanager.max-completed-applications" which is hard coded to "${yarn.resourcemanager.max-completed-applications}" but this hard coded property not defined anywhere in Ambari. Thanks in advance HCC. ... View more

Hello HCC, We recently upgraded our prod and all dev cluster from HDP 2.5.3.0 to HDP 2.6.1.0, as we are observing weird behavior in HDP 2.6.1.0 some of the nodes are getting very high allocation of containers causing very high avg load on the server and that is causing nodes to go in heart beat lost state. When the nodes got very high avg load NN making those nodes as Dead nodes where as RM still keep on assigning containers ( we know that both RM and NN work independently) on that node and all those containers are causing jobs to go in failed state. Every time when we having this issue we are asking our SA team to reboot those servers to alleviate the issue, we didn't had this behavior with HDP 2.5.3.0. Please find the screenshot for reference where nodes got very high no.of containers and load avg Present versions : HDP 2.6.1.0 and Ambari 2.5.2.0 @Kuldeep Kulkarni @Jay SenSharma @Artem Ervits @ssathish ss-1.pngss-2.pngss-3.pngss-4.png ... View more