Note : This feature is available from HDP 3.0 (Ambari 2.7) Ambari 2.7 has a cool new feature where it is integrated with Swagger and you can try and explore all the REST APIs. Steps to use Swagger Login to Ambari Hit this url ( http://{ambari-host}:8080/api-docs) This page takes you to the API explorer where you can try different APIs. Here are some of the screenshots. You can get all the supported endpoints from http://{ambari-host}:8080/api-docs/swagger.json) . Hope this helps 🙂 ... View more

In this article we will see how to produce messages using a simple python script and consume messages using ConsumeMQTT processor and put them in HDFS using PutHDFS Note: I'm using CentOS 7 and HDP 2.6.3 for this article . 1) Install MQTT sudo yum -y install epel-release sudo yum -y install mosquitto . 2) Start MQTT sudo systemctl start mosquitto sudo systemctl enable mosquitto . 3) Install paho-mqtt python library yum install python-pip pip install paho-mqtt . 4) Configure MQTT password for the user. I have created a sample user 'aditya' and set the password to 'test' [root@test-instance-4 ~]# useradd aditya [root@test-instance-4 ~]# sudo mosquitto_passwd -c /etc/mosquitto/passwd aditya Password: Reenter password: . 5) Disable anonymous login to MQTT Open the file (/etc/mosquitto/mosquitto.conf ) and add the below entries and restart mosquitto allow_anonymous false password_file /etc/mosquitto/passwd sudo systemctl restart mosquitto . 6) Design the NiFi flow to consume messages and put into hdfs Configure MQTT processor: Right Click on ConsumeMQTT -> Configure -> Properties. Set Broker URI, Client Id, username, password, Topic filter and Max Queue Size Configure PutHDFS processor: Set Hadoop Configuration resources and Directory( to store messages) . 7) Create a sample python script to publish messages. Use mqttpublish.txt attached and rename it to MQTTPublish.py to publish messages . 😎 Run the Nifi flow. . 9) Run the python script attached. python MQTTPublish.py . 10) Check the directory to check if the messages are put in HDFS hdfs dfs -ls /user/aditya/ hdfs dfs -cat /user/aditya/* . Hope this helps 🙂 mqttpublish.txt ... View more

Issue: When running hive shell in a docker you will be getting a message "mbind: Operation not permitted" printed on the console but the operations will pass. . Root Cause: mbind syscall is used for NUMA (non-uniform memory access) operations which is blocked by docker by default. But in hive opts there is an option which specifies '+UseNUMA'. . Resolution: Go to Ambari -> Hive -> Configs -> Advanced 1) Remove '-XX:+UseNUMA' from 'hive.tez.java.opts'. 2) Remove '-XX:+UseNUMA' from hive-env template. Hope this helps 🙂 ... View more

1) Download the Ambari Integration module and HDFS transparency connector. You can get it here. 2) Stop all the services from Ambari. Ambari -> Actions -> Stop All 3) After successfully stopping all the services, Go to Spectrum Scale service and Unintegrate transparency. SpectrumScale -> Service Actions -> Unintegrate Transparency This step will replace the HDFS transparency modules with native HDFS and add back Secondary Namenode. 4) Delete the Spectrum Scale service. Type "delete" to confirm deletion. SpectrumScale -> Service Actions -> Delete Service 5) Extract the tar file downloaded in Step 1 in your Ambari Server node and run the mPack uninstaller. ./SpectrumScaleMPackUninstaller.py The uninstaller prompts for few values such as Ambari IP, username etc. Enter them. 6) Performing above steps will delete the service from Ambari. Restart Ambari server and Start all services. Ambari -> Actions -> Start All If you were using a existing ESS clusters, the nodes would be still part of the ESS cluster. Remove the nodes from ESS cluster as well. To remove the nodes, perform below steps. 1) On any of the ESS cluster node, run mmlscluster [root@xxx-ems1 ~]# mmlscluster GPFS cluster information ======================== GPFS cluster name: xxx.yyy.net GPFS cluster id: 123123123123132123123 GPFS UID domain: xxx.yyy.net Remote shell command: /usr/bin/ssh Remote file copy command: /usr/bin/scp Repository type: CCR Node Daemon node name IP address Admin node name Designation ------------------------------------------------------------------------------------------------------------------------------------------ 1 xxx-yyy1-hs.gpfs.net 10.10.99.11 xxx-yyy1-hs.gpfs.net quorum-manager-perfmon 2 xxx-yyy2-hs.gpfs.net 10.10.99.12 xxx-yyy2-hs.gpfs.net quorum-manager-perfmon 3 xxx-ems1-hs.gpfs.net 10.10.99.13 xxx-ems1-hs.gpfs.net quorum-perfmon 62 hdp-gpds-node-6.openstacklocal 10.101.76.62 hdp-gpds-node-6.openstacklocal 63 hdp-gpds-node-5.openstacklocal 10.101.76.40 hdp-gpds-node-5.openstacklocal 64 hdp-gpds-node-4.openstacklocal 10.101.76.60 hdp-gpds-node-4.openstacklocal This is the sample response for mmcluster commands. In the above output, Node 1,2 and 3 are the ESS nodes and Node 62,63 and 64 are the HDP cluster nodes which were enrolled into the ESS cluster. We will go ahead and delete the HDP nodes. 2) Delete the node by running the below command mmdelnode -N hdp-gpds-node-6.openstacklocal Run this command for every HDP node. After deleting all nodes you can confirm if the nodes are really deleted by running "mmlscluster" again. Hope this helps 🙂 ... View more