Re: Services configuration base on hardware - is t...

shahrukh_khan · ‎12-19-2018

@venu devu Execute the below command on Ambari Metrics monitor node cd /usr/lib/python2.6/site-packages/resource_monitoring/psutilmake install ambari-metrics-monitor restart Then Perform a restart of Metrics Monitor

shahrukh_khan · ‎10-16-2018

Dr. Elephant Dr. Elephant is a performance monitoring and tuning tool for Hadoop and Spark. It automatically gathers a job's metrics, analyzes them, and presents them in a simple way for easy consumption. Its goal is to improve developer productivity and increase cluster efficiency by making it easier to tune the jobs. It analyzes the Hadoop and Spark jobs using a set of pluggable, configurable, rule-based heuristics that provide insights on how a job performed and then uses the results to make suggestions about how to tune the job to make it perform more efficiently. It also computes a number of metrics for a job which provides valuable information about the job performance on the cluster. Build Steps cd ~; sudo yum update -y; sudo yum upgrade -y; sudo curl --silent --location https://rpm.nodesource.com/setup_8.x | sudo bash - sudo yum install -y nodejs wget unzip git zip gcc-c++ make java-1.8.0-openjdk-devel; wget https://github.com/linkedin/dr-elephant/archive/v2.1.7.tar.gz; wget https://downloads.typesafe.com/typesafe-activator/1.3.12/typesafe-activator-1.3.12.zip; tar -xvzf v2.1.7.tar.gz; unzip typesafe-activator-1.3.12.zip export ACTIVATOR_HOME=~/activator-dist-1.3.12/; export PATH=$ACTIVATOR_HOME/bin:$PATH; sudo npm config set strict-ssl false; npm config set strict-ssl false; sudo npm install ember-tooltips; sudo npm install -g bower; # Run using non root user cd dr-elephant-2.1.7/; cd web; bower install; cd ..; ./compile.sh ;

shahrukh_khan · ‎02-21-2018

Installing Hue can be tricky as the latest version Hue 4.1.0 is available only as source code and Compiling it requires a lot of dependencies and development libraries for which there is very little information available on the Internet The best document that I could find as reference was Hue 4.1.0 Installation Guide, though there are a few packages missing in the doc After series of trial and error came up with the below steps to Compile / Install Hue successfully Install JDK, Hue dependencies, and development Libraries $ sudo yum -y update $ sudo yum install -y git gcc libxml2-devel libxslt-devel cyrus-sasl-devel cyrus-sasl-gssapi mysql-devel python-devel python-setuptools sqlite-devel ant gcc-c++ *sasl* krb5-devel libtidy openldap-devel wget libffi-devel gmp-devel java-1.8.0-openjdk Install Apache Maven from public Fedora repository $ sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo $ sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo $ sudo yum install -y apache-maven Clone Hue Git Repository git clone https://github.com/cloudera/hue.git Build Hue from Source # Configure $PREFIX with the path where you want to install Hue $ cd hue $ PREFIX=/usr/share make install The above commands were executed and tested on OS : CentOS release 6.9 (Final) Hue : Version 4.1 Date : 21/Feb/2018

shahrukh_khan · ‎09-27-2017

@Ajit Sonawane Great!! Good to hear that the documentation helped in solving the Issue. For the HBase one please create a seperate question

shahrukh_khan · ‎09-23-2017

Can you ensure if you have Successfully completed all the Steps for Securing the Cluster using Kerberos. I guess you must have missed configuring Storm and Kafka for Kerberos. Please refer to the following URL to secure the Services.

shahrukh_khan · ‎09-23-2017

Your User Limit factor is 1, which allows only a single user to use the queues full capacity and Minimum user Limit is 100%, which means that only one user gets 100% Capacity. This blocks other users from launching jobs in parallel, which causes the later jobs to starve till first one's complete as you Ordering Policy is "fifo". Also check the Maximum Application configuration of your base queue "root" as the configuration is Inherited.

shahrukh_khan · ‎09-23-2017

Ambari usually takes care of Configuring the Cluster, but sometimes manual Intervention is needed to Tune to Cluster for Performance as per the UseCase. In relation with your question for a script you can use the YARN Utility Script for Determining HDP Memory Configuration Settings - https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_command-line-installation/content/determine-hdp-memory-config.html Hortonworks support customers can utilize the SmartSense(R) tool, which gives configuration tuning recommendations - http://hortonworks.com/blog/introducing-hortonworks-smartsense/

shahrukh_khan · ‎09-23-2017

Ambari usually takes care of Configuring the Cluster, but sometimes manual Intervention is needed to Tune to Cluster for Performance as per the UseCase. In relation with your question for a script you can use the YARN Utility Script for Determining HDP Memory Configuration Settings - https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_command-line-installation/content/determine-hdp-memory-config.html Hortonworks support customers can utilize the SmartSense(R) tool, which gives configuration tuning recommendations - http://hortonworks.com/blog/introducing-hortonworks-smartsense/

shahrukh_khan · ‎09-23-2017

@Sarnath K, Do you mean Authentication or Authorization for Spark Thrift Server. For Authentication you can Enable ACL's - http://spark.apache.org/docs/latest/security.html Authorization for Spark can be done using HDFS ACLs, which can also be managed using Ranger. LLAP enabled Spark for Coloumn Security, in which reads from HDFS go directly through LLAP. You can refer to the following KB Article - SPARKSQL, RANGER, AND LLAP VIA SPARK THRIFT SERVER FOR BI SCENARIOS TO PROVIDE ROW, COLUMN LEVEL SECURITY, AND MASKING

shahrukh_khan · ‎09-23-2017

@mohammed fazil, What do you mean by Integrating MongoDB with Hadoop Components. We have a MongoDB Connector for Hadoop that provides the ability to use MongoDB as an input source and/or an output destination - https://docs.mongodb.com/ecosystem/tools/hadoop/. Ambari provides the feature of creating Custom Stacks and Services to manage Components not available by default in Ambari -https://cwiki.apache.org/confluence/display/AMBARI/How-To+Define+Stacks+and+Services. For using Ambari to install, configure and manage MongoDB you refer to following Public GitHub Repository - https://github.com/nikunjness/mongo-ambari

shahrukh_khan · ‎09-23-2017

You can temporarily resolve the Alert Issue by putting Grafana to Maintainence Mode in Ambari Web UI or disable the Ambari Alert for that particular Service. To understand why Grafana is Listening on ipv6 will have to troubleshoot the issue by analyzing logs, configuration, etc. Try getting it resolved if you have Hortonworks Vendor Support.

shahrukh_khan · ‎09-23-2017

@vsubramanian Yes you can. Please refer to Apache Documentation - https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#SnapshotOperations

shahrukh_khan · ‎09-23-2017

@Anurag Mishra Did you Enable Kerberos for Ambari Server? If its enabled, did you Enable Browser Access to a SPNEGO-enabled Web UI Please complete all the steps mentioned in the following KB Article- Provide restricted access to resource manager UI. You can also setup Perimeter Security for your Cluster Using Knox for YARN.

shahrukh_khan · ‎09-15-2017

@Junichi Oda You cannot manage access_log using log4j, as the configuration AccessLogValve is hardcoded in the code. Following logs can be managed using log4j by leveraging maxBackupIndex UserSync TagSync XA Portal Below logs cannot be managed using log4j, hence will have to leverage logrotate [ a standard tool for log rotation in linux ] - Manage Ranger Admin access_log log file growth Access Log Or else as mentioned by @Neeraj Sabharwal you can use cron script with the find command find /var/log/ranger -mtime +30| xargs --no-run-if-empty rm

shahrukh_khan · ‎06-28-2017

This is due to Snappy version mismatch between Hadoop and Pig. You can resolve this by executing the below command before loading the Grunt Shell export HADOOP_USER_CLASSPATH_FIRST=true To avoid executing the above command everytime before loading a pig grunt shell, you can streamline the above process by adding the above line of configuration in pig-env.sh and deploy the configuration file to the nodes.

Online	Offline
Last Visited	‎03-01-2020 12:01 AM

Date Registered	‎05-28-2019 01:05 AM
Last Visited	‎03-01-2020 12:01 AM
Total Messages Posted	16
Total Kudos Received	3

Re: Services configuration base on hardware - is t...

Re: Internal error. org.xerial.snappy.SnappyNative...

Re: from psutil import _common ImportError: cannot...

Dr Elephant - Build / Compile Source Code

Compile / Install Hue from Source Code on CentOS6

Re: Storm, Kafka and ambari-metrics-collector not ...

Re: Storm, Kafka and ambari-metrics-collector not ...

Re: Why does YARN Capacity Scheduling queue not re...

Re: Services configuration base on hardware - is t...

Re: Services configuration base on hardware - is t...

Re: How do I authorize users for access of Spark S...

Re: Whats are the prerequisites to install and in...

Re: Ambari Grafana - Tunnel Connection Failed on A...

Re: Is there any way to get snapshot status of a d...

Re: Restricting resource manager ui to all users ...

Re: How do I control the access log of the ranger-...

Re: Internal error. org.xerial.snappy.SnappyNative...