Member since
01-11-2017
65
Posts
1
Kudos Received
0
Solutions
05-21-2019
09:26 PM
@JT Ng Oozie spark action is not supported instead use spark oozie shell action.
... View more
12-26-2018
06:09 PM
@Saurav Ranjit please collect the yarn logs and attach them here to debug further, use below command to collect logs yarn logs -applicationId applicatioon_1545806970486_0001
... View more
09-10-2018
02:43 PM
@kgautam we already tried running MSCK repair and analyze table before posting this issue here, thanks.
... View more
09-08-2018
05:11 PM
Hive is fetching different results, if we query on a specific partition its giving correct results, but when we do the query on a specific range its giving incorrect results. when we query on specific date range its fetching correct results date_dt (DATE'2018-03-31',DATE'2018-05-19',DATE'2018-04-28',DATE'2018-03-17') but when we do the query on specific range, its yielding incorrect results date_dt BETWEEN DATE'2017-07-08' AND DATE'2018-07-07'
... View more
Labels:
- Labels:
-
Apache Hive
09-06-2018
02:43 PM
@Jay Kumar SenSharma is there way we can get "Average percentage of local files to Region server in the cluster" for entire HBase cluster instead of querying by each region server , thanks.
... View more
09-05-2018
04:02 AM
@Jay Kumar SenSharma thanks for your response, i'm looking for "Average percentage of local files to Region server in the cluster". Is there any other way apart from Ambari and Grafana to pull directly from HBase because we don't use Ambari or Grafana, thanks.
... View more
09-05-2018
02:53 AM
Hello Everyone, Is there any API/CURL command to pull HBase data locality ? we can monitor or view data locality from Ambari without that is there any easy way to find data locality in HBase. HBase version: 1.1.2.2.5.3.0-37 HDP version:2.5.3 Thanks.
... View more
Labels:
06-26-2018
02:50 PM
@Vinicius Higa Murakami Thanks for your reply. if we enable vectorization it corrupts timestamp values every time or it's intermittent ? because with vectorization enabled it was working fine, we are started to see this happening recently.
... View more
06-25-2018
04:58 PM
hive-logs.txtHello Everyone, Hive Insert query to table(have partitions) failing with vectorization enabled . It was working fine till last week.However it is running fine with vectorization disabled. Can someone help in identifying the issue. please find the logs with vectorization enabled and disabled/turned off.
... View more
Labels:
06-20-2018
05:56 PM
screen-shot-2018-06-20-at-125515-pm.png@anarasimham thanks for your reply. mdse_item_i data type is int `mdse_item_i` int and the output attached as screenshot
... View more
06-20-2018
03:41 AM
screen-shot-2018-06-19-at-104245-pm.png Count distinct doesn't always give me the right answer. I've attached two different queries that should both result in 7 unique items purchased. If I don't do an operation on mdse_item_i like cast it to a bigint, it doesn't always count them correctly. to explain it simply, when i do cast on "mdse_itcountsdistinct-1.txtdoesnotcountdistinct.txtem_i" it gives unique results as 7, but when i don't do casting it gives unique results as 10 which is not correct. hive> select * from dfr_distinct;
OK
100000000938 5 7 12.33 2 2.75 4.27 8.060 2 8 0
Time taken: 0.479 seconds, Fetched: 1 row(s)
hive> select * from dfr_distinctnot;
OK
100000000938 5 10 12.33 2 2.75 4.27 8.06 0 2 8 0
Time taken: 0.932 seconds, Fetched: 1 row(s) tried running the query in both MR and Tez modes still giving same results when i don't do casting.
... View more
Labels:
06-18-2018
03:56 AM
@Tejaswi Chandra check your queue resource utilization from RM UI
... View more
06-18-2018
03:53 AM
@Prakash Punj cleanup if there are any stale pid files for node manager, if the same server hosting region server then try stopping RS and first start NM and then RS. also look for any possible zombie process.
... View more
03-07-2018
11:21 PM
Hello HCC, We have a production cluster with 800+ data nodes and hive-clients installed one more than 200+ nodes, i can see that from Ambari is there a simple way or CURL command to find all hive-clients installed in the cluster which i can see on Cli/Linux box. HDP:2.6.1 Ambari 2.5.2.0 Thanks,Jalender.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hive
12-14-2017
08:36 PM
Hello HCC, We have 16 node cluster running on HDP 2.5.3.0, Ambari 2.5.1.0, on this cluster we allocated resource manager heap to 4G and when there is no single job running in the cluster for almost 5-6hrs, but still resource manager heap usage is going to 80-85% always. We have another big production cluster with 800 data nodes in that cluster RM usage will be under 25-30% always.On what basis heap usage is utilized on the cluster ? and how can we control heap usage when cluster is idle. i also checked this property "yarn.resourcemanager.max-completed-applications" which is hard coded to "${yarn.resourcemanager.max-completed-applications}" but this hard coded property not defined anywhere in Ambari. Thanks in advance HCC.
... View more
Labels:
- Labels:
-
Apache YARN
-
Cloudera Manager
11-21-2017
10:19 PM
Hello HCC, Can someone please help on how to reset password for SmartSense--->Acitivty explorer-->Zeppelin, we lost the password for Zeppelin login page for Activity explorer. Thanks in Advance.
... View more
Labels:
11-09-2017
09:04 PM
Hello HCC, is there a way we can check user permissions for a user called 'xyz' in HBase shell and also how to check permissions granted for a HBase table. Thanks in advance.
... View more
Labels:
11-03-2017
01:19 AM
Please try installing with Ambari rest API, like below curl --user <ambari-admin-user>:<ambari-admin-password> -i -H 'X-Requested-By: ambari' -X POST http://AMBARI_SERVER_HOST:8080/api/v1/clusters/CLUSTER_NAME/hosts/NEW_HOST_ADDED/host_components/SPARK_CLIENT
... View more
10-28-2017
01:22 AM
Log messages from one of the server where we are observing this behavior 2017-10-27 21:20:43,991 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 10059 for container-id container_e746_1508665985104_313505_01_002805: -1B of 4 GB physical memory used; -1B of 8.4 GB virtual memory used
2017-10-27 21:20:44,049 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 108356 for container-id container_e746_1508665985104_313505_01_002168: 1.2 MB of 4 GB physical memory used; 103.6 MB of 8.4 GB virtual memory used
2017-10-27 21:20:44,105 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 13033 for container-id container_e746_1508665985104_304789_01_002499: -1B of 4 GB physical memory used; -1B of 8.4 GB virtual memory used
~
... View more
10-28-2017
01:16 AM
Hello HCC, We recently upgraded our prod and all dev cluster from HDP 2.5.3.0 to HDP 2.6.1.0, as we are observing weird behavior in HDP 2.6.1.0 some of the nodes are getting very high allocation of containers causing very high avg load on the server and that is causing nodes to go in heart beat lost state. When the nodes got very high avg load NN making those nodes as Dead nodes where as RM still keep on assigning containers ( we know that both RM and NN work independently) on that node and all those containers are causing jobs to go in failed state. Every time when we having this issue we are asking our SA team to reboot those servers to alleviate the issue, we didn't had this behavior with HDP 2.5.3.0. Please find the screenshot for reference where nodes got very high no.of containers and load avg Present versions : HDP 2.6.1.0 and Ambari 2.5.2.0 @Kuldeep Kulkarni @Jay SenSharma @Artem Ervits @ssathish ss-1.pngss-2.pngss-3.pngss-4.png
... View more
Labels:
10-27-2017
06:25 PM
Hi @Jay SenSharma We are already on HDP 2.6.1.0 and Ambari 2.5.2.0, as you mentioned it was fixed in "And the Fix Version/s are : HDP-2.6.0.0 , Ambari-2.5.0" not sure why we hit the issue on HDP 2.6.1 Thanks, Jalender.
... View more
10-26-2017
03:10 AM
Miguel Marquez Please try installing with Ambari rest API, like below curl --user <ambari-admin-user>:<ambari-admin-password> -i -H 'X-Requested-By: ambari' -X POST http://AMBARI_SERVER_HOST:8080/api/v1/clusters/CLUSTER_NAME/hosts/NEW_HOST_ADDED/host_components/SPARK2_CLIENT
... View more
10-26-2017
01:28 AM
Hello HCC, We recently upgraded our cluster from HDP 2.5.3.0 to HDP 2.6.1.0 and Ambari from 2.4.2 to 2.5.2 and we observed in pig version changes after HDP upgrade pig default execution engine changed from MR mode to TEZ execution engine, is it a bug or expected behavior, i gone through HDP 2.6.1.0 release notes but no where its mentioned about this change. Can someone please clarify about this change 🙂 Thanks, Jalender. @Orlando Teixeira @Kuldeep Kulkarni @kkanchu
... View more
Labels:
06-19-2017
04:06 AM
This
is a known issue issue (regression bug of Hive UNION logic in HDP 2.5.x)
https://issues.apache.org/jira/browse/HIVE-16050 The behaviour was introduced/changed by https://issues.apache.org/jira/browse/HIVE-14251,
which was part of the HDP 2.5.x release Workaround is to use 'cast' while running the query, for example: select * from (select case when browser_type is not null THEN 'OPEN' ELSE NULL end as action_code from mcom.esp_responsys_opened_stg union all select cast(null as string) as action_code from mcom.esp_responsys_complaint_stg_tmp) a limit 10;
... View more
06-05-2017
02:23 AM
Issue resolved by setting below property set tez.grouping.max-size = 268,435,456;
... View more
05-19-2017
05:17 PM
our jobs are failing with below exception we are using HDP 2.5.3.0 , Tez 0.7.0.2.5 , Hive 1.2.1.2.5 and Ambari 2.4.0 failing with below exception Status: Failed
Vertex failed, vertexName=Reducer 2, vertexId=vertex_**, diagnostics=[Exception in VertexManager, vertex:vertex_ [Reducer 2],org.apache.tez.dag.api.TezUncheckedException: Atleast 1 bipartite source should exist
at org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.onVertexStarted(ShuffleVertexManager.java:456)
at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEventOnVertexStarted.invoke(VertexManager.java:587)
at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:658)
at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:653)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:653)
at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:642)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
, Vertex did not succeed due to AM_USERCODE_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_** [Reducer 2] killed/failed due to:AM_USERCODE_FAILURE]
Vertex did not succeed due to AM_USERCODE_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_ [Reducer 2] killed/failed due to:AM_USERCODE_FAILURE]Vertex killed, vertexName=Reducer 3, vertexId=vertex_, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_ [Reducer 3] killed/failed due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Map 4, vertexId=vertex_, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_ [Map 4] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:2
After Staging execution
is it something related to https://issues.apache.org/jira/browse/TEZ-3239 https://issues.apache.org/jira/browse/TEZ-2107 Thanks.
... View more
Labels:
03-06-2017
11:20 PM
Vertex killed, vertexName=scope-492, vertexId=vertex_1486843207585_727267_1_09, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1486843207585_727267_1_09 [scope-492] killed/failed due to:OTHER_VERTEX_FAILURE]
Vertex killed, vertexName=scope-479, vertexId=vertex_1486843207585_727267_1_07, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:999, Vertex vertex_1486843207585_727267_1_07 [scope-479] killed/failed due to:OTHER_VERTEX_FAILURE]
Vertex killed, vertexName=scope-468, vertexId=vertex_1486843207585_727267_1_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:22, Vertex vertex_1486843207585_727267_1_00 [scope-468] killed/failed due to:OTHER_VERTEX_FAILURE]
Vertex killed, vertexName=scope-478, vertexId=vertex_1486843207585_727267_1_06, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:322, Vertex vertex_1486843207585_727267_1_06 [scope-478] killed/failed due to:OTHER_VERTEX_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:20
... View more