Member since
09-20-2017
35
Posts
6
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2367 | 01-18-2022 08:23 AM |
09-05-2024
11:15 AM
Issue: During or Post upgrade to CDP 7.1.9x ( CM 7.11.X ), In some cases the Solr will inaccessible and solr instances will be unusable with the solr log containing log similar to below 2024-09-04 11:32:22.535 WARN (main) [ ] o.e.j.w.WebAppContext Failed startup of context o.e.j.w.WebAppContext@7ec3394b{/solr
,file:///opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p1000.55406660/lib/solr/server/solr-webapp/webapp/,STOPPED}{/opt/cloudera/pa
rcels/CDH-7.1.9-1.cdh7.1.9.p1000.55406660/lib/solr/server/solr-webapp/webapp} => java.io.FileNotFoundException: /opt/cloudera/p
arcels/CDH-7.1.9-1.cdh7.1.9.p1000.55406660/lib/solr/server/solr-webapp/webapp/WEB-INF/lib/ozone-filesystem-hadoop3-1.4.0.7.1.9.
1000-103.jar (No such file or directory)
at java.util.zip.ZipFile.open(Native Method)
java.io.FileNotFoundException: /opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p1000.55406660/lib/solr/server/solr-webapp/webapp/WEB
-INF/lib/ozone-filesystem-hadoop3-1.4.0.7.1.9.1000-103.jar (No such file or directory) You might also able see the alternative --list broken for "ozone-filesystem-hadoop3.jar " and will have error related to this. Cause: The issue looks same as a bug OPSAPS-67607, which is documented in known issue but we didn't had failures but just solr impacted, so we didn't try the workaround provided "as is". https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/runtime-release-notes/topics/rt-pvc-known-issues-ozone.html Workaround: On the issue solr hosts we did the following create /var/lib/hadoop-hdfs set the permission and ownership as 755 and hdfs:hadoop run the below command to fix the symlink sudo alternatives --install /var/lib/hadoop-hdfs/ozone-filesystem-hadoop3.jar ozone-filesystem-hadoop3.jar /opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p1000.55406660/lib/hadoop-ozone/ozone-filesystem-hadoop3.jar 10000 restart Solr instances with issue which will fix the issue
... View more
Labels:
01-18-2022
08:23 AM
@Claro2020 pls check the attachment
... View more
01-18-2022
06:52 AM
@Claro2020 which interpreter you are using?
... View more
01-18-2022
05:53 AM
@Claro2020 you can have the interpreter setting as "Per Note" "isolated" and can run a query in two different notebook at a same time
... View more
01-18-2021
07:50 AM
@MikeZ Haven't checked the feature in CDP yet. but as per document with pre-existing child queue this wont work. you could try creating new parent queue and edit it's child queue options. "The managed parent queue does not support other pre-configured queues to co-exist along with dynamically-created queues" REF: Link
... View more
01-07-2020
10:29 PM
@nlewis135031 We remove livy.spark.master in zeppelin-0.7. Because we suggest user to use livy 0.3 in zeppelin-0.7. And livy 0.3 don't allow to specify livy.spark.master by default, it enforce yarn-cluster mode refer here. As you aware always to have yarn-cluster mode is better after all using the cluster resource is wise. yarn-cluster mode --> you can't directly mention yarn-cluster as master option ( in HDP 2.6.x i guess - i tested in HDP 2.6.3) instead you can use the workaround below to use the yarn cluster mode. this option is available from 0.8.0 jira. master yarn spark.submit.deployMode cluster Hope this helps!
... View more
12-05-2019
04:49 AM
@GerardReverte thanks fort the inputs, yes as of now it's best to go for " Dedicated Instance with dedicated storage and configuration:" or "Dedicated Instance:" mentioned in the article for extensive or heavy production use. Thanks for sharing your experience, much appreciated.
... View more
10-07-2019
11:12 PM
1 Kudo
Motivation: As zeppelin is used for Data Ingestion/Discovery/Analytics/Visualization, the muti-purpose notebook's usage is increasing tremendously pushing users to find a way to scale out. Intention of the article is to give a fair idea on how zeppelin can be scaled out with the current available options in 2.6.x Disclaimer: This article is based on my personal experience and knowledge. Don't take it as a standard guidelines, understand the concept and modify it for your environmental best practices and use case. Beefing up the Zeppelin/interpreter Memory, tweaking the performance affecting factors will help to a certain extend but when the usage of notebook increase it is wise to scale out. In this article we are going discuss on various considerations and best practices while scaling out. Note: Please refer Key factors that affects Zeppelin's Performance before deciding to scaling out, It will be also having the best practices to maintain zeppelin Environment Benefits of multiple instances: Load sharing and performance Zeppelin currently don't have HA, while adding more instance with loadbalancer(not part of HDP) we can achieve it. You can create custom configuration groups with different configurations for each or set of zeppelin instances by using Manage Config Groups in Ambari. (Ex. you can have different shiro authentication altogether) Refer: https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-operations/content/using_host_config_groups.html Adding multiple instances of zeppelin is not exposed in UI because fo few limitations. (No HA support without a external tool, notebook created in one instance of zeppelin will not be visible until you manually re-load the notebooks or restart the instance are the few limitations I am aware off) Two things needs to be considered according to your use case when setting up multiple instances: 1) Storage 2) Accessibility Storage: Shared: From HDP 2.6.3 onwards we have HDFS as storage, so all the zeppelin instances can share same storage and access the notebooks and configuration hassle free. If you are using older HDP version refer the "Multi - Tenancy & HA" section of article Zeppelin Best Practices Dedicated: For complete isolation dedicated storage can be given for each instances (Discussed in detail below) 2) Accessibility Dedicated Instance: We can dedicatedly give users access to particular instance. Benefits: No external tool required, easy to implement and can be considered as partial HSA, since the storage is common even with one node down business critical operation can be continued Disadvantages: Ineffective use of resources. (Ex. when 10 users are using in the first instance, the second and third might be used only by 1 or two users hence the load will be hig on one instance whereas the resource in others are available. Dedicated Instance with dedicated storage and configuration: We can dedicatedly give users access to particular instance and each instance with separate HDFS or local storage space using config groups Benefits: Complete Isolation of configuration for different set of users Disadvantages: Ineffective use of resources. Using Load balancer: Having a loadbalancer with Round-robin load balancing is one of the simplest solution. Benefits: Effective use of resource, can consider as full HA Disadvantages: Need a external Tool and need proper maintenance Below is the process to install multiple zeppelin instance 1. Command: curl -u $AMBARI_USER:$AMBARI_PASSWORD -H 'X-Requested-By: ambari' -i -X POST -d '{"host_components" : [{"HostRoles":{"component_name":"ZEPPELIN_MASTER"}}] }' http://$AMBARI_HOSTNAME:8080/api/v1/clusters/$CLUSTER_NAME/hosts?Hosts/host_name=$NEW_HOST Note: please replace $AMBARI_USER, $AMBARI_PASSWORD, $AMBARI_HOSTNAME, $CLUSTER_NAME, $NEW_HOST 2. In Ambari navigate to the host you specified in the above command and click 'Re-Install' for the Zeppelin Server component.
... View more
Labels:
03-22-2019
07:32 AM
@Jay Kumar SenSharma, It resolves the issue
... View more
03-21-2019
02:46 PM
https://community.hortonworks.com/articles/38149/how-to-create-and-register-custom-ambari-alerts.html doesn't seems to be working with ambari 2.7.1, stuck at step 3) No exception in ambari server or agent logs @Jay Kumar SenSharma
... View more
Labels:
- Labels:
-
Apache Ambari