Motivation: As zeppelin is used for Data Ingestion/Discovery/Analytics/Visualization, the muti-purpose notebook's usage is increasing tremendously pushing users to find a way to scale out. Intention of the article is to give a fair idea on how zeppelin can be scaled out with the current available options in 2.6.x Disclaimer: This article is based on my personal experience and knowledge. Don't take it as a standard guidelines, understand the concept and modify it for your environmental best practices and use case. Beefing up the Zeppelin/interpreter Memory, tweaking the performance affecting factors will help to a certain extend but when the usage of notebook increase it is wise to scale out. In this article we are going discuss on various considerations and best practices while scaling out. Note: Please refer Key factors that affects Zeppelin's Performance before deciding to scaling out, It will be also having the best practices to maintain zeppelin Environment Benefits of multiple instances: Load sharing and performance Zeppelin currently don't have HA, while adding more instance with loadbalancer(not part of HDP) we can achieve it. You can create custom configuration groups with different configurations for each or set of zeppelin instances by using Manage Config Groups in Ambari. (Ex. you can have different shiro authentication altogether) Refer: https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-operations/content/using_host_config_groups.html Adding multiple instances of zeppelin is not exposed in UI because fo few limitations. (No HA support without a external tool, notebook created in one instance of zeppelin will not be visible until you manually re-load the notebooks or restart the instance are the few limitations I am aware off) Two things needs to be considered according to your use case when setting up multiple instances:  1) Storage 2) Accessibility Storage: Shared: From HDP 2.6.3 onwards we have HDFS as storage, so all the zeppelin instances can share same storage and access the notebooks and configuration hassle free. If you are using older HDP version refer the "Multi - Tenancy & HA" section of article Zeppelin Best Practices Dedicated: For complete isolation dedicated storage can be given for each instances (Discussed in detail below) 2) Accessibility Dedicated Instance: We can dedicatedly give users access to particular instance.  Benefits: No external tool required, easy to implement and can be considered as partial HSA, since the storage is common even with one node down business critical operation can be continued  Disadvantages: Ineffective use of resources. (Ex. when 10 users are using in the first instance, the second and third might be used only by 1 or two users hence the load will be hig on one instance whereas the resource in others are available.  Dedicated Instance with dedicated storage and configuration: We can dedicatedly give users access to particular instance and each instance with separate HDFS or local storage space using config groups  Benefits: Complete Isolation of configuration for different set of users  Disadvantages: Ineffective use of resources. Using Load balancer: Having a loadbalancer with Round-robin load balancing is one of the simplest solution.  Benefits: Effective use of resource, can consider as full HA  Disadvantages: Need a external Tool and need proper maintenance  Below is the process to install multiple zeppelin instance 1. Command:     curl -u $AMBARI_USER:$AMBARI_PASSWORD -H 'X-Requested-By: ambari' -i -X POST -d '{"host_components" : [{"HostRoles":{"component_name":"ZEPPELIN_MASTER"}}] }' http://$AMBARI_HOSTNAME:8080/api/v1/clusters/$CLUSTER_NAME/hosts?Hosts/host_name=$NEW_HOST     Note: please replace $AMBARI_USER, $AMBARI_PASSWORD, $AMBARI_HOSTNAME, $CLUSTER_NAME, $NEW_HOST 2. In Ambari navigate to the host you specified in the above command and click 'Re-Install' for the Zeppelin Server component. ... View more

Target Audience: It is recommended for those who have some operational experience in Zeppelin. Intention of the article is to give a fair idea on key factors which plays significant role in zeppelin's stability and performance. Disclaimer: This article is based on my personal experience and knowledge. Don't take it as a standard guidelines, understand the concept and modify it for your environmental best practices and use case. A common question everyone has, How many users can use a single zeppelin instance? It may vary from 10 to 50 users, we can't provide a number as an answer since zeppelin is a Multi-purpose Notebook and used for variety of use case with different interpreters and interpreter modes. Below are the factors that needs to be considered while deciding how many users can share/use a single instance in an environment. Often, tweaking these factors can give lot of performance boost, try doing the below checks before considering to scaling out for more instances. For scaling out please refer Zeppelin Multiple Instances Zeppelin Memory: To get better performance, Increase your Zeppelin Memory if you have enough resource in the system and when number of users are more( by default its 1GB, Option ZEPPELIN_MEM can be tuned in zeppelin_env_content) Often this could be the cause for slowness in UI responses you might need to increase Zeppelin Memory. use "jmap -heap <zeppelin PID>" as zeppelin user to analyse the heap usage at peak utilisation times of zeppelin.     Livy: Prefer Livy over spark/spark2. Follow the link Advantages of Livy to see the advantages of Livy Interpreter binding mode: Available modes are shared, scoped and isolated' In 'shared' mode, every notebook bound to the Interpreter Setting will share the single Interpreter instance (less memory consumption). Typically if your primarily used interpreter is in shared mode you should increase the ZEPPELIN_INTP_MEM in zeppelin_env_content to have better performance. In 'scoped' mode, each notebook will create new Interpreter instance in the same interpreter process In 'isolated' mode, each notebook will create new Interpreter process. ( Memory consumption is more but every user gets their their own interpreter process, interpreter restart will not affect other users) Important: Before choosing Isolated check your system has enough resources, Ex if 30 users Interpreter memory is 1GB then you will need 30 GB of RAM available in the zeppelin node. Note: Users should restart their own interpreter session from the notebook page button instead of the interpreter page which would restart sessions for all users Interpreter Memory: Logging: If you have debug log enabled, you will have considerable slowness in UI response especially while saving the configuration in interpreter while creating new notebook. ( only enable DEBUG when u troubleshoot and if its absolutely necessary) Execution Mode: If your spark code is running locally ( interpreter setting's "master" property is set to "local[*]" then your zeppelin node should have enough resources for it. always use distributed mode on the interpreter wherever available). Note: Most of the interpreter's first run will be slower since it has to create context and do create Application master in the first run. Storage: The disk used for storaged shouldn't be a bottleneck when multiple I/O operation performed, I recommend HDFS storage if you are using 2.6.3+ (Refer Enabling HDFS Storage in HDP-2.6.3+) Multiple Hung process: Sometimes unhandled exceptiona may leave stale process' which will severely affect the performance, you might see content unavailable and even it will not fix after a clean restart. Check if any old process(process' will have timestamp before the zeppelin restart) runs as zeppelin users and kill them ps -ef |grep ^zeppelin. Also you can use ps -L -u zeppelin |wc -l to see how much thread zeppelin is currently spawned and analyse the trend. you should see only 30-60 after a restart Browser: The browser plays a vital role as visualisation is one of the key objective of zeppelin, browser has to load the notebook ( say your Notebook is 10MB big ) and display its content and output. in Older version of IE you could even see that you won't be able to sign out properly. I personally recommend to use latest chrome. Clearing the notebook/pragraph's output: Clearing the output of the notebook or paragraph which has large result(output)as a practice because in the note.json the last result will be stored, where many paragraph with huge results will increase the note.json size which will be hard for the browser and JVM to hold and will slow down the UI response. ... View more