Resources

Problem Statement:When you try to execute GET call using ambari api to list/GET services, it usually gives error as shown below -# curl -u admin:admin -H "X-Requested-By: ambari" -X GET http://<AMBARI_SERVER_HOST>:8080/api/v1/clusters/<cluster_name>/services/curl: (1) Protocol http not supported or disabled in libcurlOR# curl -u admin:admin -H "X-Requested-By: ambari" -X GET “http://node1.example.com:8080/api/v1/clusters/HDP_TEST/services/“curl: (1) Protocol “http not supported or disabled in libcurlSolution:It’s a tipical curl_php error, but, the error response is not very, ehmmmm easy to deduce. It’s simple, surely there is an extra space before ‘http’, so check the CURLOPT_URL declaration, and search for this space, and then, delete it!!!! Make sure the syntax of double quotes is correct. This also leads to the error.

Zeppelin interpreters allow Zeppelin to submit requests to multiple languages or data processing engines (like Scala, PySpark, Shell, Hive, HBase etc). More details available at https://zeppelin.incubator.apache.org/docs/0.5.6-incubating/manual/interpreters.htmlHowever, before you can use interpreters, in most cases you need to configure them (e.g. point hive interpreter to server/port where HiveServer2 is running on your cluster). You can do this manually via the Zeppelin UI by clicking on the Interpreter tab.However, in some cases you will want to automate/script out this step. Below is some sample code on how to do this. To run, on the host where Zeppelin is running, copy/paste the code into a file called zeppelin.py using your favourite editor, and then run it by typing:python zeppelin.pySample code:#!/usr/local/bin/pythondef post_request(url, body): import json, urllib2 encoded_body = json.dumps(body) req = urllib2.Request(str(url), encoded_body) req.get_method = lambda: 'PUT'

Twitter has opened source another real-time, distributed, fault-tolerant stream processing engine called Heron. They see as the successor for Storm. It is backwards compatible with Storm's topology API.First I followed the getting started guide. Downloading and installing on MacOsx.Downloads ./heron-client-install-0.14.0-darwin.sh --userHeron client installer----------------------Uncompressing......Heron is now installed!Make sure you have "/usr/local/bin" in your path.See http://heronstreaming.io/docs/getting-started.html for how to use Heron.heron.build.version : 0.14.0heron.build.time : Tue May 24 22:44:01 PDT 2016heron.build.timestamp : 1464155053000heron.build.host : tw-mbp-kramasamyheron.build.user : kramasamyheron.build.git.revision : be87b09f348e0ed05f45503340a2245a4ef68a35heron.build.git.status : Clean➜ Downloads export PATH=$PATH::/usr/local/bin➜ Downloads ./heron-tools-install-0.14.0-darwin.sh --userHeron tools installer---------------------Uncompressing......Heron Tool

Hdfs can be accessed in R by specifying namenode host (hdfs://<hostname>:/user/test), but in case of namenode failover it won't work. So we should set hdfs config directory in R so it recognizes the hadoop namenode and failover configurations..Set Hadoop and Spark home and config directories into R environment as below:# set up the SPARK_HOMESys.setenv (SPARK_HOME="/usr/hdp/current/spark-client")#set up the HADOOP config dirSys.setenv (YARN_CONF_DIR="/usr/hdp/current/hadoop-client/conf")Sys.setenv (HADOOP_CONF_DIR="/usr/hdp/current/hadoop-client/conf")# read the data file sc = sparkR.init(sparkPackages="com.databricks:spark-csv_2.11:1.0.3")sqlContext = sparkRSQL.init(sc)people = read.df(sqlContext, "hdfs://HDFS-HA/users/people.json", "json")head(people)

The goal of this article is to provide you step by step instruction to install jVisualVM to monitor JVMs inside your hadoop environment. Consult your operations team prior to making any production changesGo here to download jVisualVM - tool to visually monitor JVM in hadoop. HBase is a great example where you want to visually analyze JVM health. You may already have this tool on your workstation. Simple go to command line and type jVisualVM. If it comes up your in business. else download it:Once you have application up and running install the Visual GC plugin. Tools ->available plugins->Visual GCLets go to the node you want to monitor. For jVisualVM to work it needs jstatd to run on the node. Once jstatd is running it automatically sends updates of remote applications running on the node to jVisualVM. Run jstatd from command line. If you get the following error:We need to perform an additional simple configuration. Do the following stepsRun which jstatd then cd into th

Adding service through ambari gives error as shown below - [root@sandbox ~]# curl -u admin:admin -i -X POST -d '{"ServiceInfo":{"service_name":"STORM"}}' http://xxx.xxx.xxx.xxx:8080/api/v1/clusters/Sandbox/servicesHTTP/1.1 400 Bad RequestSet-Cookie: AMBARISESSIONID=qraouzksi4vktobhob5heqml;Path=/Expires: Thu, 01 Jan 1970 00:00:00 GMTContent-Type: text/plainContent-Length: 107Server: Jetty(7.6.7.v20120910){ "status" : 400, "message" : "CSRF protection is turned on. X-Requested-By HTTP header is required." You need to disable CSRF protection as mentioned below - 1.Login to ambari server using cli [superuser credentials] vi /etc/ambari-server/conf/ambari.properties 2. Add below line at the bottom of the file api.csrfPrevention.enabled=false 3. Restart ambari server #ambari-server restart 4. Try executing POST command again to add service and it should work [root@sandbox ~]# curl -u admin:admin -i -X POST -d '{"ServiceInfo":{"service_name":"STORM"}}' http://xxx.xxx.xxx.xxx:8080/api/v1/cl

Recently I decided it was time to give my lawn a makeover. Years of brutally hot Atlanta summers have taken their toll on my grass and its well … dead. I chose the do it yourself route and as usual went over budget and invested far more time than I had planned. None the less I now have a decent looking backyard. Given my investment and how much I travel I decided it would be worth the extra money to install an automated sprinkler system. There are several of these that exist in the market so I set out doing my research. I chose Rachio (http://rachio.com) for its ability to control watering based on weather and other conditions which is awesome! While Rachio is a great product with several features it had a few shortcomings that I was hoping to supplement with my existing home setup (powered by Apache NiFi of course). The idea was the use all of the existing features that Rachio offered and then use data from my local home automation setup to further supplement the watering system. Ther

Create a Hive Table as ORC File through Spark SQL in Zeppelin.%sqlcreate table default.logs_orc_table (clientIp STRING, clientIdentity STRING, user STRING, dateTime STRING, request STRING, statusCode INT, bytesSent FLOAT, referer STRING, userAgent STRING) stored as orcLoad data from a DataFrame into this table:%sqlinsert into table default.logs_orc_table select t.* from accessLogsDF tI can create a table in the Hive View from Ambari.CREATE TABLE IF NOT EXISTS survey ( firstName STRING, lastName STRING, gender STRING, phone STRING, email STRING, address STRING, city STRING, postalcode STRING, surveyanswer STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' STORED AS TEXTFILE;Then really easy to load some data from a CSV file.LOAD DATA INPATH '/demo/survey1.csv' OVERWRITE INTO TABLE survey;I can create an ORC based table in Hive from Hive View in Ambari, Spark / Spark SQL or Hive areas in Zeppelin:create table survey_orc(firstName varchar(255),lastName

I have written ambari-admin utility to simplify efforts required to find and trigger API curl call. I'm planning to add many more features than ambari-shell.Currently below features are supported.Demo on multinode cluster:1. Clone our github repo to your local machine or any of the node in your cluster. [root@sme-ambari-server ~]# git clone https://github.com/crazyadmins/useful-scripts.gitInitialized empty Git repository in /root/useful-scripts/.git/remote: Counting objects: 106, done.remote: Total 106 (delta 0), reused 0 (delta 0), pack-reused 106Receiving objects: 100% (106/106), 16.89 KiB, done.Resolving deltas: 100% (37/37), done.2. Goto useful-scripts/ambari/ [root@sme-ambari-server ~]# cd useful-scripts/ambari/3. Edit ambari.props and modify value of below parameters as per your cluster environment [root@sme-ambari-server ambari]# cat ambari.propsCLUSTER_NAME=smeAMBARI_ADMIN_USER=adminAMBARI_ADMIN_PASSWORD=adminAMBARI_HOST=sme-ambari-server.hwxblr.comKDC_HOST=sme-ambari-server.hw

When installing HDB/HAWQ on Sandbox, it is necessary to relocate the default Ambari postgres database to a postgres instance running on a different port. The following script performs the move in a mostly automated fashion. When prompted by ambari-server setup, select option 4 for the database configuration and fill in the details. Note that this is only intended for Sandbox. Please do not use in production.#!/usr/bin/env bash## Change as needed#PGPORT=12346PGDATA=/var/lib/pgsql/ambariAMBARI_WEB_USER=adminAMBARI_WEB_PW=adminAMBARI_DB_NAME=ambariAMBARI_DB_USER=ambariAMBARI_DB_PW=bigdata## Variables#PG_INIT_PATH=/etc/init.d/postgresqlDB_BKUP_DIR=/tmp/ambari-db-backupAMBARI_PROPS=/etc/ambari-server/conf/ambari.properties## Main#echo -e "\n#### Stopping ambari-server"ambari-server stopecho -e "\n#### Creating the pgpass file"echo "*:*:*:$AMBARI_DB_USER:$AMBARI_DB_PW" >> $HOME/.pgpasschmod 600 $HOME/.pgpassecho -e "\n#### Creating database backup directory"if [ -d $DB_BKUP_DIR ]; then

No Valid Credentials Provided ErrorUpon Enabling Kerberos in Ambari, some components started (Name Node), but other components that are are failing, such as MapReduce and Hive. Here is an example of the error output when we try to start these services. Fail: Execution of 'hadoop fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` /app-logs /mapred /mapred/system /mr-history/tmp /mr-history/done && hadoop fs -chmod -R 777 /app-logs && hadoop fs -chmod 777 /mr-history/tmp && hadoop fs -chmod 1777 /mr-history/done && hadoop fs -chown mapred /mapred && hadoop fs -chown hdfs /mapred/system && hadoop fs -chownyarn:hadoop/app-logs && hadoop fs -chownmapred:hadoop/mr-history/tmp /mr-history/done' returned 1. mesg: ttyname: Invalid argument 15/04/28 16:12:33 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials pr

It is very easy to connect various tools to Hive.Apache Zeppelin is my favorite tool to query Hive via HiveQL, but you have a ton of options.First, DBVisualizer, I am using the Free Edition for MacOSx.For both tools, you will need to copy (or acquire) a copy of two Hive JARS:hadoop-common.jarhive-jdbc-1.2.1000.2.4.0.0-169-standalone.jarOn HDP 2.4:/usr/hdp/current/hive-client/lib/hive-jdbc-1.2.1000.2.4.0.0-169-standalone.jar/usr/hdp/current/hadoop-client/hadoop-common.jarAnother tool, DBeaver which has a unique twist (it finds and downloads the JARS itself via Maven, though it takes a bit for the initial driver install). It also supports Phoenix, Drill and GemfireXD.Also I did SquirrelSQL. For SquirrelSQL setup details, check out the official Hive Docs.jdbc:hive2://localhost:10000/defaultorg.apache.hive.jdbc.HiveDriverFor Hive URL, see the Hive Docs.I also connected from SQLWorkBench/J as well. This is a free SQL Tool.

Grafana is a popular, open-source graph and dashboard builder ( licensed under Apache License, Version 2.0).Grafana is feature rich; not only does it let users create customizable and shareable dashboards, it also offers templated/scripted dashboards, LDAP integration, multiple data sources, and more. Out of the box, Grafana supports data sources such as Graphite, OpenTSDB, ElasticSearch, InfluxDB, CloudWatch, and etc. Luckily, the data source part is pluggable; it was not too difficult to create an Ambari data source for it: https://github.com/u39kun/ambari-grafanaIt works for all service metrics currently exposed via Ambari: HDFS, YARN, HBase, Storm, Kafka, Flume, Accumulo, and Ambari Metrics.It supports typeahead of component names and their associated metrics, as well as aggregation (“avg”, “sum”, “min”, and “max” of metrics across all instances of the specified component).Host Metrics are not supported yet, but it shouldn't be too difficult to add.Please visit https://github.com/

Setup your environment GistDemoSource

Hive: Apache Hive is a data warehouse infrastructure built on top of Hadoop for providing data summarization, query, and analysis. HBase: Apache HBase™ is the Hadoop database, a distributed, scalable, big data store Hawq: http://hawq.incubator.apache.org/ PXF: PXF is an extensible framework that allows HAWQ to query external system data Let's learn Query federation This topic describes how to access Hive data using PXF. Link Previously, in order to query Hive tables using HAWQ and PXF, you needed to create an external table in PXF that described the target table's Hive metadata. Since HAWQ is now integrated with HCatalog, HAWQ can use metadata stored in HCatalog instead of external tables created for PXF. HCatalog is built on top of the Hive metastore and incorporates Hive's DDL. This provides several advantages: You do not need to know the table schema of your Hive tables You do not need to manually enter information about Hive table location or format If Hive table metadata

Deploy HDB (HAWQ) on HDP 2.4.2 and use Zeppelin for VisualizationGoals:Install a 4 node cluster running HDP 2.4.2 using Ambari 2.2.2.0 (including Zeppelin and HDB) using Ambari bootstrap via blueprints or Ambari install wizardConfigure HAWQ for ZeppelinConfigure Zeppelin for HAWQRun HAWQ queries via ZeppelinAccess Hive tables from HAWQNotes:HDB managed via Ambari is only supported from Ambri 2.2.2.0 onwards. Do not attempt using older versions of AmbariSingle node setup: The same steps can be used to install a single node setup - just make sure to change the HAWQ master port from default (5432) - for example: to 10432. Sample script to automate bringing up a single node HDP cluster with HDB/Zeppelin installed is available herePrebuilt single node VM for VMWare built with HDP 2.4.2, Ambari 2.2.2.0, HDB 2.0.0.0 and Zeppelin is available for download here (size: 4.84 GB). It was setup using below so once services are up, you can access Zeppelin (on port 9995) and use the existing 'HAWQ'

Problem Statement: Ranger HDFS repository test connection getting failed while the policies are working fine. When tried enabling debug more for ranger admin service found below error in log -(RangerAuthenticationProvider.java:335) - Unix Authentication Failed:org.springframework.security.authentication.AuthenticationServiceException: FAILED: unable to authenticate to AuthenticationService: node.example.com:5151 at org.springframework.security.authentication.jaas.DefaultLoginExceptionResolver.resolveException(DefaultLoginExceptionResolver.java:33) at org.springframework.security.authentication.jaas.AbstractJaasAuthenticationProvider.authenticate(AbstractJaasAuthenticationProvider.java:181) at org.apache.ranger.security.handler.RangerAuthenticationProvider.getUnixAuthentication(RangerAuthenticationProvider.java:327) at org.apache.ranger.security.handler.RangerAuthenticationProvider.authenticate(RangerAuthenticationProvider.java:114) at org.springframew

Having run a bunch of Spark jobs locally, in Spark Standalone clusters and in HDP Yarn Clusters; I have found a few JVM settings that helped with debugging non-production jobs and assist with better Garbage Collection. This is important even with off-heap storage and bare metal optimizations.spark-submit --driver-java-options "-XX:+PrintGCDetails -XX:+UseG1GC -XX:MaxGCPauseMillis=400" You can also set options extra options in the runtime environment (see Spark Documentation).For HDP / Spark, you can add this from Ambari.In your Scala Spark Program:sparkConf.set("spark.cores.max", "4")sparkConf.set("spark.serializer", classOf[KryoSerializer].getName)sparkConf.set("spark.sql.tungsten.enabled", "true")sparkConf.set("spark.eventLog.enabled", "true")sparkConf.set("spark.app.id", "MyAppIWantToFind")sparkConf.set("spark.io.compression.codec", "snappy")sparkConf.set("spark.rdd.compress", "false")sparkConf.set("spark.suffle.compress", "true")Make sure you have Tungsten on, the KryoSerializer

After a rolling or express upgrade of HDP from 2.2.x to 2.4 (and I'm told also to 2.3.2 and 2.3.4) you may face some issues with your Kafka as I did. In my case HDP was upgraded from HDP-2.2.6 (Kafka-0.8.1) to HDP-2.4 (Kafka-0.9.0) using Ambari-2.2.1.1. Here are 4 items to check and fix if needed. After the upgrade and Kafka restart, brokers IDs are set to 1001, 1002, ... However, topics created before the upgrade point to brokers numbered 0, 1, ... For example (k81c was created before the upgrade, k9a after):$ ./kafka-topics.sh --zookeeper zk1.example.com:2181 --describe --topic k81c Topic:k81c PartitionCount:2 ReplicationFactor:2 Configs: Topic: k81c Partition: 0 Leader: 0 Replicas: 0,1 Isr: 0 Topic: k81c Partition: 1 Leader: 0 Replicas: 1,0 Isr: 0$ ./kafka-topics.sh --zookeeper zk1.example.com:2181 --describe --topic k9aTopic:k9a PartitionCount:2 ReplicationFactor:1 Configs: Topic: k9a

SmartSense is an excellent tool for keeping your cluster running at optimal efficiency while maintaining operational best practices. We’ve combined knowledge from the greatest minds in the industry, and use it to analyze metadata about your cluster from the bundles you submit.Have you ever wondered exactly what data you’re sending to SmartSense? The SmartSense Admin Guide contains a high-level description (see What’s Included in a Bundle), but for the greatest understanding you should extract a bundle and explore it with your own eyes!Obtain a BundleThere are two types of bundles...Analysis Bundle: configs and metrics for all services on all hostsTroubleshooting Bundle: Analysis Bundle + logs for selected service(s)To begin, let’s capture an Analysis Bundle:...and download an unencrypted copy to our local machine:The bundle is a gzipped tar file that contains a gzipped tar file from each host running the HST Agent. In the following examples, notice the bundle variable excludes the .tgz

Chronos is a replacement for cron. A fault tolerant job scheduler for Mesos which handles dependencies and ISO8601 based schedules Marathon is a framework for Mesos that is designed to launch long-running applications, and, in Mesosphere, serves as a replacement for a traditional system In Mesosphere, Chronos compliments Marathon as it provides another way to run applications, according to a schedule or other conditions, such as the completion of another job. It is also capable of scheduling jobs on multiple Mesos slave nodes, and provides statistics about job failures and successes. Source Install https://mesos.github.io/chronos/docs/ and gist

This article is a continuation of Monitoring Kafka with Burrow - Part 1.Before diving into evaluation rules, HTTP endpoint API and notifiers, I would like to point-out a few other tools that are utilizing Burrow. Burrower (http://github.com/splee/burrower), a tool for gathering consumer lag information from Burrow and sending it into influxDBansible-burrow (https://github.com/slb350/ansible-burrow) provides an Ansible role for installing Burrow.Consumer Lag EvaluationStatusThe status of a consumer group in Burrow is determined based on several rules evaluated against the offsets for each partition the group consumes. Thus, there is no need for setting a discrete threshold for the number of messages a consumer is allowed to be behind before alerts go off. By evaluating against every partition the group consumes, the entire consumer group health status is evaluated, and not just the topics that are being monitored. This is very important for wildcard consumers, such as Kafka Mirror Maker

How to Index PDF File with Flume and MorphlineSolrSink The flow is as follows: Spooling Directory Source > File Channel > MorphlineSolrSinkThe reason I wanted to complete this exercise was to provide a less complex solution; that is, fewer moving parts, less configuration, and no coding compared to kafka / storm or spark. Also, the example is easy to setup and demonstrate quickly.Flume compared to Kafka/Storm is limited by its declarative nature, but that is what makes it easy to use. However, the morphline does even provide a java command (with some potential performance side effects), so you can get pretty explicit.I’ve read that Flume can handle at 50,000 events per second on a single server, so while the pipe may not be as fat as a Kafka/Storm pipe, it may be well suited for many use cases. Step-by-step guide1. Take care of dependencies. I am running HDP 2.2.4 Sandbox and the Solr that came with it. To get started, you will need to add a lot of dependencies to

QUESTION:How are the Ambari-2.2.1.1 local accounts protected, such as "admin"? What about all the various component's configuration data managed by Ambari-2.2.1.1?ANSWER:Ambari local account credentialsThese are stored in the Ambari database as the SHA256 hash of the (randomly salted) password.Service configuration password propertiesThese are stored in the Ambari database in blobs of JSON-formatted data in plaintext.When returned via API calls, the properties marked as passwords are masked and not displayed as plaintext.When sent to the agents, they are stored in plaintext in the command.json files stored in /var/lib/ambari-agent/data (readable only by root and the user that executes ambari-agent).Ambari-specific database and ldap credentialsThese are stored in plaintext in the ambari.properities file by default but can be encrypted via ambari-server setup-security.If encrypted, they are stored in a Java Keystore implementation (JCEKS) which uses 3DES in CBC mode with PKCS #5 padding

Virtual Memory swapping can have a large impact on the performance of a Hadoop system. Because of the memory requirements of YARN containers and processes running on the nodes in a cluster, swapping process out of memory to disk can cause serious performance limitations. As such, the historical recommendations for setting the swappiness, or propensity to swap out a process, on a Hadoop system has been to disable swap altogether. With newer versions of the Linux kernel, Out Of Memory (OOM) situations can be more likely to indiscriminately kill important processes to reclaim valuable physical memory on the system with a swappiness of 0.In order to prevent the system from swapping processes too frequently, but still allow for emergency swapping (instead of killing processes), the recommendation is now to set swappiness to 1 on Linux systems. This will still allow swapping, but with the least possible aggressiveness (for comparison, the default value for swappiness is 60).To change the sw