Community Articles

Platform: HDP 2.4 (Sandbox) Hadoop version: 2.7 OS: CentOS 6.8 For every data node that Spark will run on, Python 2.7 and all dependent libraries that your code uses must be installed on each data node - e.g.Pandas, Matplotlib, Scipy Below are the steps I performed to get iPython up and running. My notebook jobs are executed via YARN. I did this install as ‘root’ user. STEP 1 Install/configure the following 5 iPython dependencies by typing in the following commands: yum install nano centos-release-SCL zlib-devel yum install bzip2-devel openssl-devel ncurses-devel yum install sqlite-devel readline-devel tk-devel yum install gdbm-devel db4-devel libpcap-devel xz-devel yum install libpng-devel libjpg-devel atlas-devel STEP 2 iPython has a requirement for Python 2.7 or higher. Check which version of Python you are using:In my case, I had an older version of Python so I then had to install a new version. Install the “Development tools” dependency for Python 2.7Command to run: yum gro

Most enterprises require data transport between applications to be via RESTful interfaces in order to facilitate cross application and organization communication and to reduce the degree of data integration coupling between applications. The summary table below identifies out the major HDP services ability to support RESTful communication over HTTP. HTTP Operations Supported HDP Service Read from HTTP Write to HTTP Retrieve Meta information Execute Queries Flume X X Sqoop X X X X Hive X X X X HBase X X X X Storm Java programming Java programming Java programming X Spark Spark Programming Spark Programming Spark Programming Spark Programming Kafka X (requires plugin) X (requires plugin) X (requires plugin) X (requires plugin)

LDAP Usage Hadoop may be configured to use LDAP as the source for resolving an authenticated user's list of group memberships. A common example where Hadoop needs to resolve group memberships is the permission checks performed by HDFS at the NameNode. The Apache documentation's HDFS Permissions Guide contains further discussion of how the group mapping works: the NameNode calls a configurable plugin to get the user's group memberships before checking permissions.Despite that document's focus on group resolution at the NameNode, many other Hadoop processes also call the group mapping. The information in this document applies to the entire ecosystem of Hadoop-related components.As described in that document, the exact implementation of the group mapping is configurable. Here is the documentation of the configuration property from core-default.xml and its default value. <property> <name>hadoop.security.group.mapping</name> <value>org.apache.hadoop.security

https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/host_scripts/alert_disk_space.pyvi /var/lib/ambari-server/resources/host_scripts/alert_disk_space.py# defaults in case no script parameters are passedMIN_FREE_SPACE_DEFAULT = 5000000000L #5GBPERCENT_USED_WARNING_DEFAULT = 50PERCENT_USED_CRITICAL_DEFAULT = 80You can change the above parameter to avoid alerting at 80% threshold reached in case it needs to be aletered to 85 or 90

SYMPTOM: Specifying a valid group in Ranger Knox policy results in a 403 authorization error.ERROR: The result is a 403 forbidden error. ROOT CAUSE: Most likely the cause of this issue is that the topology is not setup in Knox for ldap groups to be passed to Ranger from the Knox plugin. RESOLUTION: Make sure the following values are present and correct in the topology:<!-- changes needed for group sync--><param> <name>main.ldapRealm.authorizationEnabled</name> <value>true</value></param><param> <name>main.ldapRealm.groupSearchBase</name> <value>OU=MyUsers,DC=AD-HDP,DC=COM</value></param><param> <name>main.ldapRealm.groupObjectClass</name> <value>group</value></param><param> <name>main.ldapRealm.groupIdAttribute</name> <value>cn</value></param> ALTERNATIVE SOLUTION:Instead of getting the above LDAP group settings w

I've received a couple of "how to" questions after folks successfully deploy the Hortonworks Data Platform Standard on Microsoft's Azure. I've collected my responses here as a reference to others:What is Hortonworks Data Platform (HDP) Standard?It is a muti-node HDP 2.4/Ambari 2.2.1 cluster on Microsoft’s Azure Cloud launched in a few mouse clicks. Hortonworks aims to match this service with the latest version of HDP. You provide: your nameemailpasswords or ssh key the number of nodes the VM types for your masters and workers should your cluster be HA or notas shown in this screenshot:Where's Ambari?Once the cluster is successfully deployed, the Azure Dashboard will go from something like:tothe Ambari service is located on the first master server. In the Azure portal goto Resource group, <your resource group name which was selected at first "Basics" step>, master1, settings and look for Public IP address. Use a web browser to access Ambari with:<master1 Public IP address>:

When adding a net new data source to Metron, the first step is to decide how to push the events from the new telemetry data source into Metron. You can use a number of data collection tools and that decision is decoupled from Metron. However, we recommend evaluating Apache Nifi as it is an excellent tool to do just that (this article uses Nifi to push data into Metron). The second step is to configure Metron to parse the telemetry data source so that downstream processing can be done on it. In this article we will walk you through how to perform both of these steps. In the previous article of this blog series, we described the following set of requirements for Customer Foo who wanted to add the Squid telemetry data source Into Metron. The proxy events from Squid logs need to be ingested in real-time. The proxy logs must be parsed into a standardized JSON structure that Metron can understand. In real-time, the squid proxy event must be enriched so that the domain names are enriched wit

The documentation is straight forward BUT based on my experience, we always end up some kind of LDAP error messages.This article is walking through the Hive and LDAP integration.HDP 2.3.4Ambari 2.2Security workshop was used to setup the openLDPA Most important: 2016-02-10 08:02:50,705 ERROR [HiveServer2-Handler-Pool: Thread-39]: transport.TSaslTransport (TSaslTransport.java:open(315)) - SASL negotiation failurejavax.security.sasl.SaslException: Error validating the login [Caused by javax.security.sasl.AuthenticationException: Error validating LDAP user [Caused by javax.naming.AuthenticationException: [LDAP: error code 32 - No Such Object]]]at org.apache.hive.service.auth.PlainSaslServer.evaluateResponse(PlainSaslServer.java:109)I kept hitting the above error because of wrong baseDNCorrect setting:Test:

SYMPTOM: KMS gets 500 error when decrypting files when being accessed from another one way trust realm. ERRORS: Command line error: [root@support ~]$ hdfs dfs -cat /zone_encr3/abc1.txt cat: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 500, message: Internal Server Error KMS Stack Trace error: 2016-05-04 15:44:21,677 ERROR [webservices-driver] - Servlet.service() for servlet [webservices-driver] in context with path [/kms] threw exceptionorg.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to user06@HDP.COM at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:377) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:347) at java.secu

1. Make sure webhbase is running:[root@ambari-server]# curl http://localhost:60080/versionrest 0.0.3 [JVM: Oracle Corporation 1.8.0_60-25.60-b23] [OS: Linux 2.6.32-504.el6.x86_64 amd64] [Server: jetty/6.1.26.hwx] [Jersey: 1.9] 2. If the webhdfs REST daemon is not running then you will need to start it. [root@ambari-server]# /usr/hdp/current/hbase-client/bin/hbase-daemon.sh start rest -p 60080starting rest, logging to /var/log/hbase/hbase-root-rest-ambari-server.support.com.out [root@ambari-server security]# ps -ef | grep 60080root 19147 1 0 20:17 pts/1 00:00:00 bash /usr/hdp/current/hbase-client/bin/hbase-daemon.sh --config /usr/hdp/current/hbase-client/bin/../conf foreground_start rest -p 60080root 19161 19147 28 20:17 pts/1 00:00:03 /usr/jdk64/jdk1.8.0_60/bin/java -Dproc_rest -XX:OnOutOfMemoryError=kill -9 %p -Dhdp.version=2.3.4.0-3485 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log -Djava.security.auth.login.config=/usr/hdp/current/hbase-re

1. Username and password input fields in ranger should be the ldap user and password authenticating through knox. 2. Configure the admin topology to meet your configuration needs by default it points to the demo ldap server on knox server. 3. To test manually, login to the URL directly using ldap credentials from the ranger server. https://<knoxhost>:8443/gateway/admin/api/v1/topologies 4. Make sure the ranger server trusts the knox certificate being presented; if it doesn't; you will need to add the knox gateway trust certificate to the ranger/java truststore. 5. Currently knox service lookups don't work in ranger and a bug has been filed to fix this. Currently, as of HDP 2.4, only topology lookup in ranger works.

Predicting stock portfolio losses using Monte Carlo simulation in Spark Summary Have you ever asked yourself: what is the most money my stock holdings could lose in a single day? If you own stock through a 401k, a personal trading account, or employer provided stock options then you should absolutely ask yourself this question. Now think about how to answer it. Your first guess maybe to pick a random number, say 20%, and assume that is the worst case scenario. While simple, this is likely to be wildly inaccurate and certainly doesn’t take into account the positive impacts of a diversified portfolio. Surprisingly, a good estimate is hard to calculate. Luckily, financial institutions have to do this for their stock portfolios (called Value at Risk (VaR)), and we can apply their methods to individual portfolios. In this article we will run a Monte Carlo simulation using real trading data to try to quantify what can happen to your portfolio. You should now go to your broker website

Apache Storm is a distributed real-time computation system for processing large volumes of high-velocity data. Debugging a distributed system is inherently hard since there are many moving parts spread across a large number of hosts in a cluster. Tracing failures to a particular component or a host in the system is hard and requires one to collect and analyze a lot of logs and debug/trace processes running in the distributed cluster.In the past the support for debugging topologies in an Apache storm cluster was very minimal. Users would have to first turn on debug logs, restart the workers, redeploy the topologies and then collect logs from different worker nodes and analyze them. If they wanted to take a jstack dump or profile the workers, they would have to login to the worker hosts and run the commands manually which was painful. If one wanted to inspect the tuples flowing through the topology pipeline, they would have to add special “debug” bolts to log the tuples to some log and t

Modern CPU and NUMAProcessor clock speed has increased dramatically in the recent years, however, the CPU needs to be supplied with a large amount of memory bandwidth to use its processing power effectively. Even a single CPU running a memory-intensive workload, such as a scientific computing application, can be constrained by memory bandwidth. This problem is amplified on symmetric multiprocessing (SMP) systems, where many processors must compete for bandwidth on the same system bus.Non-uniform memory access (NUMA) is a computer memory design used in multiprocessing, where the memory access time depends on the memory location relative to the processor. NUMA systems have more than one system bus and they can harness large numbers of processors in a single system, linking several small, cost-effective nodes using a high-performance connection.. Each node contains processors and memory, much like a small SMP system. However, an advanced memory controller allows a node to use memory on

When using smartsense 1.2 or below in conjunction with OpenJDK, you get the following error upon startup. It's a none issue which will be resolved in the next smart sense version. Traceback (most recent call last): File "/usr/sbin/hst-agent.py", line 420, in <module> main(sys.argv) File "/usr/sbin/hst-agent.py", line 397, in main setup(options) File "/usr/sbin/hst-agent.py", line 323, in setup server_hostname = get_server_hostname(server, tries, try_sleep, options.quiet) File "/usr/sbin/hst-agent.py", line 107, in get_server_hostname hostname = validate_server_hostname(default_hostname, tries, try_sleep) File "/usr/sbin/hst-agent.py", line 125, in validate_server_hostname elif not register_agent(server_hostname): File "/usr/sbin/hst-agent.py", line 143, in register_agent if not server_api.register_agent(agent_version): File "/usr/hdp/share/hst/hst-agent/lib/hst_agent/ServerAPI.py", line 104, in register_agent content = self.call(request) File "/usr/hdp/share/hst/hst-agent

Problem Statement:When you try to execute GET call using ambari api to list/GET services, it usually gives error as shown below -# curl -u admin:admin -H "X-Requested-By: ambari" -X GET http://<AMBARI_SERVER_HOST>:8080/api/v1/clusters/<cluster_name>/services/curl: (1) Protocol http not supported or disabled in libcurlOR# curl -u admin:admin -H "X-Requested-By: ambari" -X GET “http://node1.example.com:8080/api/v1/clusters/HDP_TEST/services/“curl: (1) Protocol “http not supported or disabled in libcurlSolution:It’s a tipical curl_php error, but, the error response is not very, ehmmmm easy to deduce. It’s simple, surely there is an extra space before ‘http’, so check the CURLOPT_URL declaration, and search for this space, and then, delete it!!!! Make sure the syntax of double quotes is correct. This also leads to the error.

Zeppelin interpreters allow Zeppelin to submit requests to multiple languages or data processing engines (like Scala, PySpark, Shell, Hive, HBase etc). More details available at https://zeppelin.incubator.apache.org/docs/0.5.6-incubating/manual/interpreters.htmlHowever, before you can use interpreters, in most cases you need to configure them (e.g. point hive interpreter to server/port where HiveServer2 is running on your cluster). You can do this manually via the Zeppelin UI by clicking on the Interpreter tab.However, in some cases you will want to automate/script out this step. Below is some sample code on how to do this. To run, on the host where Zeppelin is running, copy/paste the code into a file called zeppelin.py using your favourite editor, and then run it by typing:python zeppelin.pySample code:#!/usr/local/bin/pythondef post_request(url, body): import json, urllib2 encoded_body = json.dumps(body) req = urllib2.Request(str(url), encoded_body) req.get_method = lambda: 'PUT'

Twitter has opened source another real-time, distributed, fault-tolerant stream processing engine called Heron. They see as the successor for Storm. It is backwards compatible with Storm's topology API.First I followed the getting started guide. Downloading and installing on MacOsx.Downloads ./heron-client-install-0.14.0-darwin.sh --userHeron client installer----------------------Uncompressing......Heron is now installed!Make sure you have "/usr/local/bin" in your path.See http://heronstreaming.io/docs/getting-started.html for how to use Heron.heron.build.version : 0.14.0heron.build.time : Tue May 24 22:44:01 PDT 2016heron.build.timestamp : 1464155053000heron.build.host : tw-mbp-kramasamyheron.build.user : kramasamyheron.build.git.revision : be87b09f348e0ed05f45503340a2245a4ef68a35heron.build.git.status : Clean➜ Downloads export PATH=$PATH::/usr/local/bin➜ Downloads ./heron-tools-install-0.14.0-darwin.sh --userHeron tools installer---------------------Uncompressing......Heron Tool

Hdfs can be accessed in R by specifying namenode host (hdfs://<hostname>:/user/test), but in case of namenode failover it won't work. So we should set hdfs config directory in R so it recognizes the hadoop namenode and failover configurations..Set Hadoop and Spark home and config directories into R environment as below:# set up the SPARK_HOMESys.setenv (SPARK_HOME="/usr/hdp/current/spark-client")#set up the HADOOP config dirSys.setenv (YARN_CONF_DIR="/usr/hdp/current/hadoop-client/conf")Sys.setenv (HADOOP_CONF_DIR="/usr/hdp/current/hadoop-client/conf")# read the data file sc = sparkR.init(sparkPackages="com.databricks:spark-csv_2.11:1.0.3")sqlContext = sparkRSQL.init(sc)people = read.df(sqlContext, "hdfs://HDFS-HA/users/people.json", "json")head(people)

The goal of this article is to provide you step by step instruction to install jVisualVM to monitor JVMs inside your hadoop environment. Consult your operations team prior to making any production changesGo here to download jVisualVM - tool to visually monitor JVM in hadoop. HBase is a great example where you want to visually analyze JVM health. You may already have this tool on your workstation. Simple go to command line and type jVisualVM. If it comes up your in business. else download it:Once you have application up and running install the Visual GC plugin. Tools ->available plugins->Visual GCLets go to the node you want to monitor. For jVisualVM to work it needs jstatd to run on the node. Once jstatd is running it automatically sends updates of remote applications running on the node to jVisualVM. Run jstatd from command line. If you get the following error:We need to perform an additional simple configuration. Do the following stepsRun which jstatd then cd into th

Adding service through ambari gives error as shown below - [root@sandbox ~]# curl -u admin:admin -i -X POST -d '{"ServiceInfo":{"service_name":"STORM"}}' http://xxx.xxx.xxx.xxx:8080/api/v1/clusters/Sandbox/servicesHTTP/1.1 400 Bad RequestSet-Cookie: AMBARISESSIONID=qraouzksi4vktobhob5heqml;Path=/Expires: Thu, 01 Jan 1970 00:00:00 GMTContent-Type: text/plainContent-Length: 107Server: Jetty(7.6.7.v20120910){ "status" : 400, "message" : "CSRF protection is turned on. X-Requested-By HTTP header is required." You need to disable CSRF protection as mentioned below - 1.Login to ambari server using cli [superuser credentials] vi /etc/ambari-server/conf/ambari.properties 2. Add below line at the bottom of the file api.csrfPrevention.enabled=false 3. Restart ambari server #ambari-server restart 4. Try executing POST command again to add service and it should work [root@sandbox ~]# curl -u admin:admin -i -X POST -d '{"ServiceInfo":{"service_name":"STORM"}}' http://xxx.xxx.xxx.xxx:8080/api/v1/cl

Recently I decided it was time to give my lawn a makeover. Years of brutally hot Atlanta summers have taken their toll on my grass and its well … dead. I chose the do it yourself route and as usual went over budget and invested far more time than I had planned. None the less I now have a decent looking backyard. Given my investment and how much I travel I decided it would be worth the extra money to install an automated sprinkler system. There are several of these that exist in the market so I set out doing my research. I chose Rachio (http://rachio.com) for its ability to control watering based on weather and other conditions which is awesome! While Rachio is a great product with several features it had a few shortcomings that I was hoping to supplement with my existing home setup (powered by Apache NiFi of course). The idea was the use all of the existing features that Rachio offered and then use data from my local home automation setup to further supplement the watering system. Ther

Create a Hive Table as ORC File through Spark SQL in Zeppelin.%sqlcreate table default.logs_orc_table (clientIp STRING, clientIdentity STRING, user STRING, dateTime STRING, request STRING, statusCode INT, bytesSent FLOAT, referer STRING, userAgent STRING) stored as orcLoad data from a DataFrame into this table:%sqlinsert into table default.logs_orc_table select t.* from accessLogsDF tI can create a table in the Hive View from Ambari.CREATE TABLE IF NOT EXISTS survey ( firstName STRING, lastName STRING, gender STRING, phone STRING, email STRING, address STRING, city STRING, postalcode STRING, surveyanswer STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' STORED AS TEXTFILE;Then really easy to load some data from a CSV file.LOAD DATA INPATH '/demo/survey1.csv' OVERWRITE INTO TABLE survey;I can create an ORC based table in Hive from Hive View in Ambari, Spark / Spark SQL or Hive areas in Zeppelin:create table survey_orc(firstName varchar(255),lastName

I have written ambari-admin utility to simplify efforts required to find and trigger API curl call. I'm planning to add many more features than ambari-shell.Currently below features are supported.Demo on multinode cluster:1. Clone our github repo to your local machine or any of the node in your cluster. [root@sme-ambari-server ~]# git clone https://github.com/crazyadmins/useful-scripts.gitInitialized empty Git repository in /root/useful-scripts/.git/remote: Counting objects: 106, done.remote: Total 106 (delta 0), reused 0 (delta 0), pack-reused 106Receiving objects: 100% (106/106), 16.89 KiB, done.Resolving deltas: 100% (37/37), done.2. Goto useful-scripts/ambari/ [root@sme-ambari-server ~]# cd useful-scripts/ambari/3. Edit ambari.props and modify value of below parameters as per your cluster environment [root@sme-ambari-server ambari]# cat ambari.propsCLUSTER_NAME=smeAMBARI_ADMIN_USER=adminAMBARI_ADMIN_PASSWORD=adminAMBARI_HOST=sme-ambari-server.hwxblr.comKDC_HOST=sme-ambari-server.hw

When installing HDB/HAWQ on Sandbox, it is necessary to relocate the default Ambari postgres database to a postgres instance running on a different port. The following script performs the move in a mostly automated fashion. When prompted by ambari-server setup, select option 4 for the database configuration and fill in the details. Note that this is only intended for Sandbox. Please do not use in production.#!/usr/bin/env bash## Change as needed#PGPORT=12346PGDATA=/var/lib/pgsql/ambariAMBARI_WEB_USER=adminAMBARI_WEB_PW=adminAMBARI_DB_NAME=ambariAMBARI_DB_USER=ambariAMBARI_DB_PW=bigdata## Variables#PG_INIT_PATH=/etc/init.d/postgresqlDB_BKUP_DIR=/tmp/ambari-db-backupAMBARI_PROPS=/etc/ambari-server/conf/ambari.properties## Main#echo -e "\n#### Stopping ambari-server"ambari-server stopecho -e "\n#### Creating the pgpass file"echo "*:*:*:$AMBARI_DB_USER:$AMBARI_DB_PW" >> $HOME/.pgpasschmod 600 $HOME/.pgpassecho -e "\n#### Creating database backup directory"if [ -d $DB_BKUP_DIR ]; then