Member since
12-02-2015
42
Posts
28
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
263 | 04-03-2018 03:30 AM | |
230 | 04-25-2017 09:27 PM | |
1496 | 03-22-2017 04:45 PM |
11-27-2020
06:52 AM
I did everything as indicated here, but nothing is written to the file /var/log/atlas/atlas_perf.log (the file exists but is empty)
... View more
11-16-2020
10:33 AM
I use this tool to get a set hdfs permission https://github.com/hadoopol/hdfs_copy_permission
... View more
08-19-2020
07:42 AM
If you use the $ZK_HOST defined in infra-solr-env.sh you should not need to include the /infra-solr prefix when getting the solrconfig.xml: source /etc/ambari-infra-solr/conf/infra-solr-env.sh /usr/lib/ambari-infra-solr/server/scripts/cloud-scripts/zkcli.sh -z $ZK_HOST \ -cmd getfile /configs/ranger_audits/solrconfig.xml solrconfig.xml The same applies when uploading the edited config.
... View more
06-27-2018
11:54 PM
1 Kudo
Current version of Ambari doesn't monitor Number of Hiveserver2 connections. We often see Hiveserver2 slowness due to heavy load as part increase in number connections to Hiveserver2. Setting up a alert for Hiveserver2 established connections will help us to take required actions like adding additional Hiveserver2 services, proper load balancing or scheduling the jobs. NOTE : Please go through this article https://github.com/apache/ambari/blob/2.6.2-maint/ambari-server/docs/api/v1/alert-definitions.md to understand Ambari Alert Definition Please find the python script and .json file used below in the attachments.
alert_hiveserver_num_connection.py - Is the python script that finds the current established connection for each Hiveserver2 and based on number of connection it returns 'CRITICAL', 'WARN', 'OK’ alerths.json - Is the Ambari Alert definition Below are the steps to setup the Ambari Alert on Hiveserver2 Established connections. Step 1 - Place the file “alert_hiveserver_num_connection.py" in the following path on the ambari-server : "/var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/alerts/ " [root@vb-atlas-ambari tmp]# cp alert_hiveserver_num_connection.py /var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/alerts/ Step 2 - Restart Ambari Server, to force Ambari agents to pull alert_hiveserver_num_connection.py python script to every host. ambari-server restart Once Ambari Server is restarted , we can verify if alert_hiveserver_num_connection.py is available in " /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/ " location on Hiveserver2 host.
Note : Some time it takes longer for Ambari agent to pull the script from Ambari server. [root@vb-atlas-node1 ~]# ll /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/
total 116
-rw-r--r--. 1 root root 9740 Jun 27 17:01 alert_hive_interactive_thrift_port.py
-rw-r--r--. 1 root root 7893 Jun 27 17:01 alert_hive_interactive_thrift_port.pyo
-rw-r--r--. 1 root root 9988 Jun 27 17:01 alert_hive_metastore.py
-rw-r--r--. 1 root root 9069 Jun 27 17:01 alert_hive_metastore.pyo
-rw-r--r--. 1 root root 1888 Jun 27 17:01 alert_hiveserver_num_connection.py
-rw-r--r--. 1 root root 11459 Jun 27 17:01 alert_hive_thrift_port.py
-rw-r--r--. 1 root root 9362 Jun 27 17:01 alert_hive_thrift_port.pyo
-rw-r--r--. 1 root root 11946 Jun 27 17:01 alert_llap_app_status.py
-rw-r--r--. 1 root root 9339 Jun 27 17:01 alert_llap_app_status.pyo
-rw-r--r--. 1 root root 8886 Jun 27 17:01 alert_webhcat_server.py
-rw-r--r--. 1 root root 6563 Jun 27 17:01 alert_webhcat_server.pyo
Step 3 - Post the Alert Definition (alerths.json) to Ambari using curl curl -u <Ambari_admin_username>:<Amabri_admin_password> -i -h 'X-Requested-By:ambari' -X POST -d @alerths.json http://<AMBARI_HOST>:<AMBARI_PORT>/api/v1/clusters/<CLUSTER_NAME>/alert_definitions Example : [root@vb-atlas-ambari tmp]# curl -u admin:admin -i -H 'X-Requested-By:ambari' -X POST -d @alerths.json http://172.26.108.142:8080/api/v1/clusters/vinod/alert_definitions
HTTP/1.1 201 Created
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Cache-Control: no-store
Pragma: no-cache
Set-Cookie: AMBARISESSIONID=10f33laf224yy1834ygq9cekbo;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
User: admin
Content-Type: text/plain
Content-Length: 0
We should be able to see the Alert in Ambari -> Alerts ( HiveServer2 Established Connections) Alternative we can also see the "HiveServer2 Established Connections" listed in Alert definitions “ http://<AMBARI_HOST>:<AMBARI_PORT>/api/v1/clusters/<CLUSTER_NAME>/alert_definitions “ Step 4 - As per Alert Definition (alerths.json) CRITICAL alert is set to 50 and WARNING is set to 30 connections by default. You can update the values directly from Ambari by editing the values.
... View more
- Find more articles tagged with:
- ambari-alerts
- ambari-server
- hiveserver2
- How-ToTutorial
- monitoring
- solutions
Labels:
06-14-2018
05:35 PM
2 Kudos
In DPS-1.1.0 We can't remove cluster from DPS UI. We can use the curl to remove the cluster. Note : User with Dataplane Admin Role can perform the below. screen-shot-2018-06-14-at-102253-am.png To delete smayani-hdp cluster Step1:- Find the cluster ID, which you want to remove. You can use the developers tools from the browser to find the cluster ID. screen-shot-2018-06-14-at-101629-am.png From above example smayani-hdp cluster ID is: 3 ( https://172.26.125.109/api/lakes/3/servicesDetails ) Step 2 :- From the console use the below curl to remove the cluster. curl -k -u <username>:<Password> -X DELETE https://<DPS_HOST>/api/lakes/<cluster_ID>; Example : curl -k -u admin:kjncsadasdcsdc -X DELETE https://172.26.125.109/api/lakes/3 Once the above is executed you should no longer see the cluster in UI. screen-shot-2018-06-14-at-102509-am.png Alternative You can also use rm_dp_cluster.sh in /usr/dp/current/core/bin on DPS installed server. Usage: ./rm_dp_cluster.sh DP_JWT HADOOP_JWT DP_HOST_NAME CLUSTER_NAME DATA_CENTER_NAME DP_JWT: Value of dp_jwt cookie from a valid user's browser session HADOOP_JWT: Value of hadoop-jwt cookie from a valid user's browser session
DP_HOST_NAME: Hostname or IP address of the DataPlane server CLUSTER_NAME: Name of the cluster to delete
DATA_CENTER_NAME: Name of the datacenter cluster belongs to You can use developers tool to use find the cookies ( DP_JWT, HADOOP_JWT)
... View more
- Find more articles tagged with:
- dps
- Governance & Lifecycle
- hdp-2.3.4
- How-ToTutorial
Labels:
06-06-2018
09:35 PM
OBJECTIVE: Updating the log configs of DPS App. Example default log file is set to logs/application.log which can be changed or Updating the log level to DEBUG for troubleshooting. Since DP App will be running in docker we can use docker commands to update them. STEPS: 1. Find the docker container running DP App on the host running DPS. Use "docker ps" [root@dps-node ~]#docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
abd412417907 hortonworks/dlm-app:1.1.0.0-41 "runsvdir /etc/sv" 28 hours ago Up 2 hours 9011/tcp dlm-app
62620e578e31 hortonworks/dp-app:1.1.0.0-390 "/bootstrap.sh" 2 days ago Up 16 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 9000/tcp dp-app
38dda17dfdf4 hortonworks/dp-cluster-service:1.1.0.0-390 "./docker_service_st…" 2 days ago Up 2 days 9009-9010/tcp
Copy the container ID, from above example it is "62620e578e31" 2. Get the current logback.xml file [root@dps-node ~]# docker exec -it 62620e578e31 /bin/cat /usr/dp-app/conf/logback.xml > logback.xml 3. Update the configs in local logback.xml which we redirected in above command. In below I have updated the location from default logs/application.logs to /usr/dp-app/logs/. <configuration>
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/usr/dp-app/logs/application.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!-- Daily rollover with compression -->
.
.
<appender name="AKKA" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/usr/dp-app/logs/akka.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
.
.
.
</encoder>
</appender>
<appender name="ACCESS_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/usr/dp-app/logs/access.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
.
.
We can also update the log level . <root level="DEBUG">
<appender-ref ref="FILE"/>
</root>
4. If needed make a backup of the original logback.xml file and cp the updated logback.xml [root@dps-node ~]#docker exec -it 62620e578e31 /bin/cp /usr/dp-app/conf/logback.xml /usr/dp-app/conf/logback.xml.bck
[root@dps-node ~]# docker exec -i 62620e578e31 tee /usr/dp-app/conf/logback.xml < logback.xml 5. Restart the docker container is required to make changes effective. [root@dps-node ~]# docker restart 62620e578e31 6. You can verify if the changes have updated. [root@dps-node ~]# docker exec -it 62620e578e31 /bin/ls -lrt /usr/dp-app/logs
total 64
-rw-r--r-- 1 root root 0 Jun 6 20:50 access.log
-rw-r--r-- 1 root root 62790 Jun 6 21:27 application.log
... View more
- Find more articles tagged with:
- dps
- How-ToTutorial
- logs
- Sandbox & Learning
- setup
06-04-2018
05:52 PM
1 Kudo
Short Description: Describes ways to manually regenerate keytabs for services through Ambari REST API Article Make sure KDC credentials are added to Ambari credentials store. You can follow this Article to perform. Once KDC credentials are added. You can use the below Ambari's REST API to regenerate keytabs. curl -H "X-Requested-By:ambari" -u <Ambari_Admin_username>:<Amabri_Admin_password> -X PUT -d '{ "Clusters": { "security_type" : "KERBEROS" } }' http://<Ambari_HOST>:8080/api/v1/clusters/<Cluster_Name>/?regenerate_keytabs=all Example : curl -H "X-Requested-By:ambari" -u admin:admin -X PUT -d '{ "Clusters": { "security_type" : "KERBEROS" } }' http://172.26.108.142:8080/api/v1/clusters/vinod/?regenerate_keytabs=all&ignore_config_updates=true Once the Keytabs are regenerated it requires Service restart to use the newly generated keytabs.
... View more
- Find more articles tagged with:
- ambari-api
- ambari-server
- How-ToTutorial
- Kerberos
- Security
Labels:
04-23-2018
10:19 PM
Problem Description Atlas uses Solr to store Lineage meta information and uses Zookeeper for co-ordination and store/maintain configuration. Due to heavy load on Zookeeper on larger cluster we need to increase timeout for ZK session for some services from default. One such config is for Ambari infra (Solr) zookeeper time on Atlas side. ERROR: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper vb-hortonwork.com:2181/infra-solr within 15000 ms java.util.concurrent.TimeoutException: Could not connect to ZooKeeper vb-hortonwork.com:2181/infra-solr within 15000 ms RESOLUTION: We can increase the session timeout from default 15000 by adding below properties in custom application-properties in atlas -> config atlas.graph.index.search.solr.zookeeper-connect-timeout=60000 atlas.graph.index.search.solr.zookeeper-session-timeout=60000
... View more
- Find more articles tagged with:
- Atlas
- Governance & Lifecycle
- Issue Resolution
- Zookeeper
- zookeeper-slow
Labels:
07-18-2019
04:36 PM
I did the following. I created a symlink for hbase-site.xml on each edge node as follows. ln -s /etc/hbase/conf/hbase-site.xml /etc/spark2/conf/hbase-site.xml I started spark shell with the following params. spark-shell --driver-memory 5g --jars /usr/hdp/2.6.5.0-292/shc/shc-core-1.1.0.2.6.5.0-292.jar,/usr/hdp/current/hbase-client/lib/hbase-common.jar,/usr/hdp/current/hbase-client/lib/hbase-client.jar,/usr/hdp/current/hbase-client/lib/hbase-server.jar,/usr/hdp/current/hbase-client/lib/hbase-protocol.jar,/usr/hdp/current/hbase-client/lib/guava-12.0.1.jar,/usr/hdp/current/hbase-client/lib/htrace-core-3.1.0-incubating.jar
... View more
07-19-2018
10:22 PM
This I believe, is in correlation to the following bug Link which was corrected in specific versions of Tez. If you are running Interactive query HDInsight in Azure, they are still using Tez .7.0 (not fixed). I did however, find this blog post that explains how to at least stop the queries from running. As far as I can tell, Tez will still show them as running, but it does free up your resources again. Edit: the follow up link is probably helpful 🙂 Fix
... View more
05-05-2017
03:30 PM
@Saba Baig are you using apache hadoop ? or HDP ? To get the whole hadoop classpath, you just need to run the following command on your machine. You can copy that pass it to HADOOP_CLASSPATH hadoop classpath Example :[root@vb-atlas-node2 ~]# hadoop classpath
/usr/hdp/2.6.0.3-8/hadoop/conf:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop/.//*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/./:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/.//*:/usr/hdp/2.6.0.3-8/hadoop-yarn/lib/*:/usr/hdp/2.6.0.3-8/hadoop-yarn/.//*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/.//*::mysql-connector-java-5.1.17.jar:mysql-connector-java.jar:/usr/hdp/2.6.0.3-8/tez/*:/usr/hdp/2.6.0.3-8/tez/lib/*:/usr/hdp/2.6.0.3-8/tez/conf
To install hadoop client and hive client you can go to Ambari and select hosts -> select the atlas hostname -> components -> Add ( hive client, hadoop client )
... View more
03-23-2017
04:44 PM
awesome @Ken Jiiii hive-site.xml should be available across the cluster in /etc/spark/conf ( where /usr/hdp/current/spark-client/conf will be symlink to) and spark client need to be installed across the cluster worker nodes for your yarn-cluster mode to run as your spark driver can run on any worker node and should be having client installed with spark/conf. If you are using Ambari it will taking care of hive-site.xml available in /spark-client/conf/
... View more
03-12-2017
01:51 AM
Is this needed even after the HDP 2.5 native Oozie Spark action?
... View more
01-01-2017
12:15 AM
Setting up Log Search SSL and HTTPS A keystore and a truststore are required for this setup. These instructions assume that you have already created .jks for the keystore and truststore 1. Create keystore location a. Keystore Setup Place the keystore in /etc/security/certs/ and else by using a symlink you can point to another location of your keystore.jks b. Ensure log search user can read the keystore chown logsearch:hadoop *.keyStore.jks 2. Create a truststore for logsearch: a. Cert signed by CA: i. Copy the keystore into <host>.trustStore.jks ii. Create a symlink to this similar to the keystore /etc/security/certs/truststore.jks -> /etc/security/certs/<host>.trustStore.jks b. Ensure log search user can read the trust store chown logsearch:hadoop *.trustStore.jks 3. Update Ambari configuration a. Update logsearch UI Protocol to https b. Update Trust store location ( logsearch_truststore_location ) and password c. Update Keystore location ( logsearch_keystore_location ) and password 4. Restart log search server UPDATE Logsearch Alert in Ambari Once the Log Search is configured to be accessed using SSL, the following steps are to be performed to update the Alert Definition of "Log Search Web UI" to check https URL. Note: Please replace the variables with appropriate values for your cluster ( Admin credentials, Ambari host and Cluster name ) 1. GET Alert Definition ID. Execute the below command, by replacing the variables with appropriate values. and search for logsearch_ui section curl -s -k -u $AMB_USER:$AMB_PASS -H 'X-Requested-By: ambari' -X GET http://<Ambari_HOST>:8443/api/v1/clusters/<CLUSTER_NAME>/alert_definitions Sample output for logsearch_ui section: { "href" : "http://sandbox.hortonworks.com:8443/api/v1/clusters/sandbox/alert_definitions/451", "AlertDefinition" : { "cluster_name" : “sandbox", "id" : 451, "label" : "Log Search Web UI", "name" : "logsearch_ui" } }, 2. GET the Alert Definition. Use the href value from the above step's sample output to get the Alert Definition of "Log Search Web UI" by executing the below command. curl -s -k -u $AMB_USER:$AMB_PASS -H 'X-Requested-By: ambari' -X GET http://sandbox.hortonworks.com:8443/api/v1/clusters/sandbox/alert_definitions/451 Sample Output: { "href" : "http://sandbox.hortonworks.com.com:8443/api/v1/clusters/sandbox/alert_definitions/451", "AlertDefinition" : { "cluster_name" : “sandbox", "component_name" : "LOGSEARCH_SERVER", "description" : "This host-level alert is triggered if the Log Search UI is unreachable.", "enabled" : true, "help_url" : null, "id" : 451, "ignore_host" : false, "interval" : 1, "label" : "Log Search Web UI", "name" : "logsearch_ui", "repeat_tolerance" : 1, "repeat_tolerance_enabled" : false, "scope" : "ANY", "service_name" : "LOGSEARCH", "source" : { "reporting" : { "critical" : { "text" : "Connection failed to {1} ({3})" }, "ok" : { "text" : "HTTP {0} response in {2:.3f}s" }, "warning" : { "text" : "HTTP {0} response from {1} in {2:.3f}s ({3})" } }, "type" : "WEB", "uri" : { "http": "{{logsearch-env/logsearch_ui_port}}", "https": "{{logsearch-env/logsearch_ui_port}}", "default_port": 61888, "connection_timeout": 5 } } } } 3. Create a temp file with new variables. Create a temp file (in this example: logsearch_uri) with below contents to update the URI sections to include https_property and https_property_value variables and values. logsearch_uri file contents: { "AlertDefinition": { "source": { "reporting": { "ok": { "text": "HTTP {0} response in {2:.3f}s" }, "warning": { "text": "HTTP {0} response from {1} in {2:.3f}s ({3})" }, "critical": { "text": "Connection failed to {1} ({3})" } }, "type": "WEB", "uri": { "http": "{{logsearch-env/logsearch_ui_port}}", "https": "{{logsearch-env/logsearch_ui_port}}", "https_property": "{{logsearch-env/logsearch_ui_protocol}}", "https_property_value": "https", "default_port": 61888, "connection_timeout": 5 } } } } 4. PUT the updated Alert Definition. Execute the below command to update the Alert Definition using logsearch_uri file created in the previous step. There will be no output to displayed after the execution of this command. curl -s -k -u $AMB_USER:$AMB_PASS -H 'X-Requested-By: ambari' -X PUT -d @logsearch_uri http://sandbox.hortonworks.com:8443/api/v1/clusters/sandbox/alert_definitions/451 5. Validate the update Execute again the get Alert Definition command (as below) and verify the https_property and https_propert_value are now part of uri section. curl -s -k -u $AMB_USER:$AMB_PASS -H 'X-Requested-By: ambari' -X GET http://sandbox.hortonworks.com:8443/api/v1/clusters/sandbox/alert_definitions/451 Sample Output: { "href" : "http://sandbox.hortonworks.com:8443/api/v1/clusters/sandbox/alert_definitions/451", "AlertDefinition" : { "cluster_name" : “sandbox", "component_name" : "LOGSEARCH_SERVER", "description" : "This host-level alert is triggered if the Log Search UI is unreachable.", "enabled" : true, "help_url" : null, "id" : 451, "ignore_host" : false, "interval" : 1, "label" : "Log Search Web UI", "name" : "logsearch_ui", "repeat_tolerance" : 1, "repeat_tolerance_enabled" : false, "scope" : "ANY", "service_name" : "LOGSEARCH", "source" : { "reporting" : { "critical" : { "text" : "Connection failed to {1} ({3})" }, "ok" : { "text" : "HTTP {0} response in {2:.3f}s" }, "warning" : { "text" : "HTTP {0} response from {1} in {2:.3f}s ({3})" } }, "type" : "WEB", "uri" : { "http": "{{logsearch-env/logsearch_ui_port}}", "https": "{{logsearch-env/logsearch_ui_port}}", "https_property": "{{logsearch-env/logsearch_ui_protocol}}", "https_property_value": "https", "default_port": 61888, "connection_timeout": 5 } } } } NOTE: In the first if you had disabled Alert Definition for "Log Search Web UI" in Ambari, then Enable it again, else wait for the time interval for alert check to execute.
... View more
- Find more articles tagged with:
- ambari-alerts
- Atlas
- How-ToTutorial
- Sandbox & Learning
Labels:
08-05-2016
11:31 AM
No, I've no idea what Nifi is. I will check it, thanks for the response 🙂
... View more