Member since
10-26-2016
98
Posts
11
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1431 | 09-16-2019 04:00 AM | |
831 | 04-21-2017 01:43 PM | |
637 | 03-30-2017 11:35 AM | |
3256 | 12-26-2016 10:19 AM |
02-28-2020
02:47 AM
@jsensharma Thank you for your response. please check if you have enabled the LLAP interactive query in order to get the statistics for the partition table : I have configured HiveServer2 Interactive but how can I verify whether it is being enabled to get the statistics for the partition table. "Interactive Mode" was set to "false" and I changed it to "true" and did hard refresh of the browser but still the same issue.
... View more
02-28-2020
01:16 AM
Hi Team, We are getting below error when clicking on Statistics tab in Hive View for partitioned table. I found one bug which seems to be relevant but this is already resolved in 2.5.0. Ambari version which I am using is 2.6.2.2. Hive view 2.0. https://issues.apache.org/jira/browse/AMBARI-20113 Any help on this issue. Regards, Shyam
... View more
Labels:
01-06-2020
11:03 PM
@Shelton Sorry for late response. I followed the thread and was able to configure Fair Scheduler successfully. Thank you for your help. Cheers.
... View more
01-02-2020
09:30 PM
Hi @Shelton Thank you for the update. It seems there are some configuration needs to be added/changed in YARN. I will perform the changes and update you in case of any failures.
... View more
01-02-2020
08:54 AM
Hi Team, I am trying to use FairScheduler in HDP 3.0.1. As per https://stackoverflow.com/questions/38208845/switching-from-capacity-scheduler-to-fair-scheduler-on-hortonworks-data-platform it seems to be possible in HDP 3.0. I have changed the schedule claa and set yarn.scheduler.fair.allocation.file in custom yarn-site in ambari (screenshot attached). But when I restart ResourceManager it is failing with below error. 2020-01-02 16:21:54,134 INFO fair.AllocationFileLoaderService (AllocationFileLoaderService.java:reloadAllocations(230)) - Loading allocation file file:/etc/hadoop/conf/fair-scheduler.xml
2020-01-02 16:21:54,221 INFO monitor.SchedulingMonitor (SchedulingMonitor.java:serviceInit(60)) - Initializing SchedulingMonitor=SchedulingMonitor (ProportionalCapacityPreemptionPolicy)
2020-01-02 16:21:54,221 INFO capacity.ProportionalCapacityPreemptionPolicy (ProportionalCapacityPreemptionPolicy.java:init(160)) - Preemption monitor:org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy
2020-01-02 16:21:54,222 INFO service.AbstractService (AbstractService.java:noteFailure(267)) - Service SchedulingMonitor (ProportionalCapacityPreemptionPolicy) failed in state INITED
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Class org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler not instance of org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
at org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.init(ProportionalCapacityPreemptionPolicy.java:165)
at org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitor.serviceInit(SchedulingMonitor.java:61)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitorManager.updateSchedulingMonitors(SchedulingMonitorManager.java:93)
at org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitorManager.initialize(SchedulingMonitorManager.java:123)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1422)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1230)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1512) Any help on this issue.
... View more
Labels:
12-10-2019
12:35 AM
Couldn't find a better way than running an ansible command to find out stale llap daemons and kill the process. ansible -m shell -a 'ps -ef | grep -i llap' -i hosts.ini workr_nodes Note : No restart required for HiveServer2 Interactive.
... View more
12-09-2019
09:03 AM
HiveServer2 Interactive UI is showing Launching Instances. This seems to be stale llap deamons. How can we find these deamon process running on which server as we have around 300 nodes?
... View more
- Tags:
- Hive
Labels:
- Labels:
-
Apache Hive
12-05-2019
12:43 AM
@prabhat_ratnala you can also try to disable cbo and then run the query. This worked for me. set hive.cbo.enable=false;
... View more
11-18-2019
03:50 AM
@sagarshimpi Thank you.
... View more
11-18-2019
02:39 AM
@sagarshimpi Thank you for the link. Is it okay to remove the PIG clients if there no such use cases. Will this impact any of the service in the cluster.
... View more
11-18-2019
12:45 AM
Hi Team,
Could you please help me to understand why installing Hive Service requires PIG clients to be installed. Also, is it okay to remove PIG clients after installing hive.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hive
10-15-2019
06:59 AM
Hive's execution engine supports MapReduce & Tez. Does it supports Spark in HDP 3.x.
https://issues.apache.org/jira/browse/HIVE-7292
... View more
Labels:
09-16-2019
04:00 AM
I was able to figure out with the help of below community article. https://community.cloudera.com/t5/Community-Articles/How-to-install-all-clients-on-all-nodes-in-your- cluster/ta-p/246821 script.sh
---------
curl -s -u admin:admin http://ambari:8080/api/v1/hosts|grep host_name|sed -n 's/.*"host_name" : "\([^\"]*\)".*/\1/p'>hostcluster.txt
while read line ;
do
j=$line
mkdir -p $j
curl -s -u admin:admin -H "X-Requested-By:ambari" -i -X PUT -d '{"RequestInfo":{"context":"Restart HDFS Client","operation_level":{"level":"HOST_COMPONENT","cluster_name":"rest","host_name":"$j","service_name":"HDFS"}},"Body":{"HostRoles":{"state":"INSTALLED"}}}' http://ambari:8080/api/v1/clusters/rest/hosts/$j/host_components/HDFS_CLIENT
done < hostcluster.txt
... View more
09-16-2019
01:40 AM
Below article shows how to restart clients on Specific host. https://cwiki.apache.org/confluence/display/AMBARI/Restarting+host+components+via+the+API Is there any way we can perform the same on all the hosts in the cluster at one time. Thank you in advance.
... View more
Labels:
- Labels:
-
Ambari Blueprints
-
Apache Ambari
09-11-2019
12:29 AM
Hi @jsensharma Thank you for your response. I checked the Grafana Admin and the version is v2.6.0. In Dependencies, Prometheus 1.0.0 is also mentioned as DataSource. Do we need to configure other DataSource in newer version of Grafana for this feature or can we use AmbariMetrics as DataSource.
... View more
09-10-2019
09:45 AM
Hi Team, I am trying to create a custom metric and import NFS metrics into Grafana but it doesn't seems to be working. Followed, https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html and now metrics is available at http://<hostname>:50079/jmx as a JSON document. <property>
<name>nfs.metrics.percentiles.intervals</name>
<value>100</value>
</property> Also, tried to import json as mentioned in below link but no luck. https://grafana.com/grafana/dashboards/6257 HDP - 2.6.5 Ambari - 2.6.2.2 Any help on this.
... View more
Labels:
- Labels:
-
Apache Ambari
05-22-2019
07:12 AM
This seems to be related to https://issues.apache.org/jira/browse/YARN-3432
... View more
05-17-2019
08:04 AM
Hi All, We have observed that whenever VCores Total equals to VCores Used, total memory in RM UI shows incorrect memory. We have checked and all the NodeManagers are up. Is this a expected behavior/Bug? Regards, Shyam
... View more
Labels:
- Labels:
-
Apache YARN
-
Cloudera Manager
01-07-2019
11:12 AM
@siva s please refer - https://community.hortonworks.com/questions/91199/yarn-web-interface-reporting-0-for-many-metrics-fo.html
... View more
05-09-2018
08:18 AM
Hi All, We are trying to find out a way from where we can get the information of applications ran on YARN over a period of time. Let's say in a week. We tried with YARN/APP Timeline Server rest api's but that didn't helped much. We have AMS configured. In YARN - HOME, under Apps we can view the completed application on a day to day basis. We tried accessing hbase using phoenix client to capture the same but with no luck. Any help on this.
... View more
Labels:
- Labels:
-
Apache YARN
03-22-2018
07:04 AM
Hi @apatel can you validate the custom yarn-site properties once again. yarn.timeline-service.http-authentication.proxyuser.<AMBARI_SERVER_PRINCIPAL_USER>.groups=* yarn.timeline-service.http-authentication.proxyuser.<AMBARI_SERVER_PRINCIPAL_USER>.hosts=* yarn.timeline-service.http-authentication.proxyuser.<AMBARI_SERVER_PRINCIPAL_USER>.users=*
... View more
02-15-2018
07:22 AM
@Sanjeev Rao check this - https://issues.apache.org/jira/browse/KNOX-728. This could be related.
... View more
01-16-2018
12:01 PM
Hi, @Pranav Lal We have changed the owner for hive.service.keytab (under /etc/security/keytabs) from root:root to hive:hadoop. Generated ticket using kinit and tried the beeline command which worked fine from terminal. As discussed, the issue still persists when the user is trying to connect to hive using beeline connection string using %%bash function (in JupyterHub notebook).
... View more
01-10-2018
11:02 AM
@Aditya Sirna Thanks for the information.
... View more
01-10-2018
10:46 AM
@Aditya Sirna Thanks for the response. Could there be any downside of skipping the service check during rolling upgrade.
... View more
01-10-2018
08:07 AM
We are now sure that we can't remove the clients from Datanodes as our applications will start failing. So, is there any way we can skip the service checks during the rolling upgrade process.
... View more
01-09-2018
05:57 PM
Hello All, We have a cluster with 160+ nodes. We are planning for an rolling upgrade from HDP 2.5 to 2.6 using ambari. We have edgenodes where all the clients has been installed. In addition, we also have clients installed on all the datanodes (150+ nodes) like HDFS client, Zookeeper client etc. If you review the below link, we could see that ambari does validate the services multiple times by running service checks during upgrade process. https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-upgrade/content/upgrading_HDP_perform_rolling_upgrade.html This will consume a lot of time during the upgrade. We can minimize the time by limiting the clients service checks. Do we really need all the clients to be installed on the datanodes? If we remove the clients from datanodes, will there be any impact on the cluster in terms of functionality. If there is no impact then what all clients can be removed . Appreciate your input!!
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
11-10-2017
01:16 PM
Hi, @Jay Kumar SenSharma Surprisingly, now I am able to connect to Hiveserver2 using Database Type as Apache Hive2 also. I believe it is picking up the correct driver from /opt/yellowfin/appserver/webapps/ROOT/WEB-INF/lib directory.
... View more
11-10-2017
12:58 PM
Hi, @Jay Kumar SenSharma I was able to fix the issue. Now we are able to connect to Hiveserver2. -> Used hive-jdbc-1.2.1000.2.6.2.0-205-standalone.jar instead hive-jdbc-1.2.1000.2.6.2.0-205.jar -> Selected Database, Generic JDBC instead Apache Hive2 as Apache Hive2 was using com.hof.mi.jdbc.driver.HiveJDBC as default driver and it was not allowing me to change driver to org.apache.hive.jdbc.HiveDriver -> Connection String : jdbc:hive2://<Hiveserver2 hostname>:10000/default -> Username/Password : blank -> Test Connection (It worked this time), Create View & Analyze.
... View more