Community Articles

How to setup a multi user (Active Directory backed) zeppelin integrated with ldap and using Livy Rest server .Pre-requisites:Setup the LDAP/AD integration for ambari using this lab (Enable Active Directory Authentication for Ambari):https://github.com/HortonworksUniversity/Security_Labs#lab-1If you are using self-signed certificate, Download the SSL certificate to where zeppelin is running<code>mkdir -p /etc/security/certificatesstore the certificate in this directoryImport certificate for zeppelin to work with the self signed certificate.<code>cd /etc/security/certificates keytool -import -alias sampledcfieldcloud -file ad01.your.domain.name.cer -keystore /usr/jdk64/jdk1.8.0_77/jre/lib/security/cacertskeytool -list -v -keystore /usr/jdk64/jdk1.8.0_77/jre/lib/security/cacerts | grep sampledcfieldcloudCreate home directory in hdfs for the user that you will login:<code>hdfs dfs -mkdir /user/hadoopadminhdfs dfs -chown hadoopadmin:hdfs /user/hadoopadminEnable multi-user

PROBLEM: Ambari Server won't be able to start because of DB inconsistencies.Sample Error:2017-02-06 05:08:43,975 ERROR - You have non selected configs: zeppelin-ambari-config for service ZEPPELIN from cluster XXXX! 2017-02-06 05:08:43,976 INFO - ******************************* Check database completed ******************************* 2017-02-06 05:10:12,834 INFO - Checking DB store version 2017-02-06 05:10:14,094 INFO - DB store version is compatible 2017-02-07 13:50:31,769 INFO - ******************************* Check database started ******************************* 2017-02-07 13:50:41,247 INFO - Checking for configs not mapped to any cluster 2017-02-07 13:50:41,322 INFO - Checking for configs selected more than once 2017-02-07 13:50:41,326 INFO - Checking for hosts without state 2017-02-07 13:50:41,330 INFO - Checking host component states count equals host component desired states count 2017-02-07 13:50:41,334 INFO - Checking services and their configs 2017-02-07 13:50:45,793 INFO - P

ENVIRONMENT and SETUPI did test the below solution withHDP 2.5.0.0-1245 and Ambari 2.4.0.1HDP 2.5.3.0-37 and Ambari 2.4.2.0Shiro.ini[users]# List of users with their password allowed to access Zeppelin.# To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISectionsadmin = password, adminmaria_dev = password, adminuser1 = password, role1user2 = password, role2user3 = password, admin[main]#activeDirectoryRealm = org.apache.zeppelin.server.ActiveDirectoryGroupRealm#activeDirectoryRealm.systemUsername = CN=Administrator,CN=Users,DC=HW,DC=EXAMPLE,DC=COM#activeDirectoryRealm.systemPassword = Password1!#activeDirectoryRealm.hadoopSecurityCredentialPath = jceks://user/zeppelin/zeppelin.jceks#activeDirectoryRealm.searchBase = CN=Users,DC=HW,DC=TEST,DC=COM#activeDirectoryRealm.url = ldap://ad-nano.test.example.com:389#activeDirectoryRealm.groupRolesMap = ""#activeDirectoryRealm.authorizationCachingEnabled = true#ld

- Ambari manages it's Http Session information's using "AMBARISESSIONID". So if we want to logout the logged in user then we will need to find out it's "AMBARISESSIONID" cookie value. We can get it using browser debug tools like for Google chrome we can use Menu -> More Tools --> Developer ToolsThe we can go to the "Network" tab of the developer tool and then on the right hand side we should see the Cookies and other request headers.Here i have found that the value for the "AMBARISESSIONID" Http Session Cookie is "1t4o6q0ph8j23etnl3e6ig5b3" hence now we can simply make a curl call as following in order to get this user logged out:$ curl -u admin:admin -i -H 'X-Requested-By:ambari' -H "Cookie: AMBARISESSIONID=1t4o6q0ph8j23etnl3e6ig5b3" -X GET http://c6401.ambari.apache.org:8080/api/v1/logoutHTTP/1.1 200 OKX-Frame-Options: DENYX-XSS-Protection: 1; mode=blockUser: adminContent-Type: text/plainContent-Length: 0Server: Jetty(8.1.19.v20160209)- As soon as the above curl command is ex

IntroductionHortonworks Dataflow (HDF) powered by Apache NiFi, kafka and Storm, collects, curates, analyzes and delivers real-time data from the IoAT to data stores both on-premises and in the cloud. Apache NiFi automates and manages the flow of information between systems. NiFi data flows are made of series of processors each with specific task. NiFi provides hundreds of general purpose processors. NiFi can pull data from various sources. This document mainly discusses in detail the integration with AWS S3 data source.Amazon S3 is cloud storage for the Internet. To upload your data (photos, videos, documents etc.), you first create a bucket in one of the AWS regions. You can then upload any number of objects to the bucket. S3 buckets and objects are resources, and Amazon S3 provides APIs for you to manage them.With many existing AWS customers there is need to integrate with S3 to process data across multiple applications. NiFi provides many processors to manage and process S3 objects

Overview Atlas provides powerful Tagging capabilities which Data Analysts to identify all data sets containing specific types of data. The Atlas UI itself provides a powerful Tag based search capability which require no REST API interaction. However, for those of you out there who need to integrate Tag based search with some of their data discovery and governance activities, this posting is for you. Within this posting are some instructions regarding how you can use the Atlas REST API to retrieve entity data based on a TAG name. Before getting too deep into the Atlas Tag search examples it is important to recognize that Atlas Tags are basically a form of an Atlas type. If you invoke the REST API command “/api/atlas/types”, in the summary output below interspersed between standard Atlas types such as ‘hive_table’, ‘jms_topic’, etc., will be the current set of user defined Atlas Tags (CUSTOMER & SALES) as shown below: "count": 35, "requestId": "qtp1177377518-81 - c7d4a853-02a0-4

Garbage CollectionBest Practice GC ConfigurationThis is an example implementation of our current recommendation for best practice GC tuning, driven by requirements of the NameNode:export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=8 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Xms1G -Xmx1G -XX:NewSize=128M -XX:MaxNewSize=128M -XX:PermSize=128M -XX:MaxPermSize=256M -verbose:gc -Xloggc:/Users/chris/hadoop-deploy-trunk/hadoop-3.0.0-SNAPSHOT/logs/gc.log-`date +'%Y%m%d%H%M'` -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:ErrorFile=/Users/chris/hadoop-deploy-trunk/hadoop-3.0.0-SNAPSHOT/logs/hs_err_pid%p.log -XX:+HeapDumpOnOutOfMemoryError $HADOOP_NAMENODE_OPTS"This is from a small development environment, so the heap is much smaller (1 GB) than what you’ll see in a real production cluster.In the past, we’ve shared with you what to use for GC settings, but have we ever explained why?GC T

I was looking for a way to easily forward and analyze provenance data that is available in nifi. There were a couple of options available. You could use the nifi-rest api to search for provenance data and then use the results for analysis or storing in a database. The other alternative was to setup a Site2Siteprovenancereportingtask in nifi. This will forward provenance events to a flow in nifi. option 1 is a very techy option , you could point you UI directly to the rest api and present a nice provenance visual with bulk replay capabilities. But, then it makes the developer responsible for keeping up with changes in the nifi rest api. It would be nice if we did not have direct dependency. Also, you might want to lockdown the rest api in production.option 2 is very easy , but it is limited in where i can send those provenance events.The apache nifi eng team resolved this situation with a ScriptedReportingTask controller service. It gives you an easy way of setting up the provenance r

Word Count using Spark Streaming in PysparkThis is a WordCount example with the followingLocal File System as a sourceCalculate counts using reduceByKey and store them in a temp tableQuerying running counts through SQL Setup: Define the function that sets up the StreamingContextThis function has to create a new StreamingContext and set it up with all the input stream, transformation and output operations.But it must not start the StreamingContext, it is just to capture all the setup code necessary for your streaming application in one place. from pyspark.streaming import StreamingContextbatchIntervalSeconds = 10 def creatingFunc(): ssc = StreamingContext(sc, batchIntervalSeconds) # Set each DStreams in this context to remember RDDs it generated in the last given duration. # DStreams remember RDDs only for a limited duration of time and releases them for garbage # collection. This method allows the developer to specify how long to remember the RDDs ( # if the developer wishes

Sentiment CoreNLP Processor[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP -Adding annotator tokenize[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.TokenizerAnnotator- No tokenizer type provided. Defaulting to PTBTokenizer.[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP -Adding annotator ssplit[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP -Adding annotator parse[pool-1-thread-1] INFO edu.stanford.nlp.parser.common.ParserGrammar- Loading parser from serialized file edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz... done [0.4 sec].[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP -Adding annotator sentimentFILE:Header,Header2,Header3Value,Value2,Value3Value4,Value5,Value6Attribute: {"names":"NEGATIVE"}Service Source CodeJUnit Test for ProcessorTo Add Sentiment Analysis to Your NiFi Data Flow, just add the custom processor, CoreNLPProcessor. You can downloada pre-built NAR from the github listed below. Add to yo

Create a HANA table for demo purposes firstCREATE COLUMN TABLE "CODEJAMMER"."STORE_ADDRESS" (ID bigint not null primary key ,"STREETNUMBER" INTEGER CS_INT, "STREET" NVARCHAR(200), "LOCALITY" NVARCHAR(200), "STATE" NVARCHAR(200), "COUNTRY" NVARCHAR(200)) UNLOAD PRIORITY 5 AUTO MERGE ; insert into "CODEJAMMER"."STORE_ADDRESS" (ID,STREETNUMBER,STREET,LOCALITY,STATE,COUNTRY) values(1,555,'Madison Ave','New York','NY','America');insert into "CODEJAMMER"."STORE_ADDRESS" (ID,STREETNUMBER,STREET,LOCALITY,STATE,COUNTRY) values(2,95,'Morten Street','New York','NY','USA');insert into "CODEJAMMER"."STORE_ADDRESS" (ID,STREETNUMBER,STREET,LOCALITY,STATE,COUNTRY) values(3,2395,'Broadway Street','New York','NY','USA'); Configure the SAP JDBC Driver in Spark config to where the driver is saved on ALL the nodes . My example: /tmp/ngdbc.jar You can also load the ngdbc.jar using the zeppelin depency if you dont want to system wide access:z.reset() // clean up previously added artifact an

Prerequisites:* SAP HANA - Instructions to setup a Cloud HANA on AWS or Azurehttps://community.hortonworks.com/content/kbentry/58427/getting-started-with-sap-hana-and-vora-with-hdp-us.html* HDP 2.5.xWe will use Spark shell, scala code and data frames to access HANA using JDBC driver.Start the spark shell with ngdbc.jar driver.spark-shell --master yarn-client --jars /tmp/ngdbc.jarscala> val url="jdbc:sap://xxxx:30015/?currentschema=CODEJAMMER"url: String = jdbc:sap://xxxx:30015/?currentschema=CODEJAMMER scala> | val prop = new java.util.Propertiesprop: java.util.Properties = {} scala> prop.setProperty("user","xxxx")res1: Object = null scala> prop.setProperty("password","xxxx")res2: Object = null scala> prop.setProperty("driver","com.sap.db.jdbc.Driver")res3: Object = null scala> scala> val emp_address = sqlContext.read.jdbc(url,"EMPLOYEE_ADDRESS",prop)emp_address: org.apache.spark.sql.DataFrame = [ID: bigint, STREETNUMBER: int, STREET: string, LOCALITY: string,

Prerequisites: * NIFI 1.0+ * SAP HANA - Instructions to setup a Cloud HANA on AWS or Azure: https://community.hortonworks.com/content/kbentry/58427/getting-started-with-sap-hana-and-vora-with-hdp-us.html SETUP:HANA (Source Database)In this setup we will create a table in HANA table. First follow this HCC article "Demo data in SAP Vora Using Eclipse HANA Modelling tools - Part 3"You will need to download Eclipse Neon - Eclipse IDE for Java Developers to connect to the SAP HANA that we setup in Part 1 . After you setup eclipse we will need to configure Eclipse to install HANA Modelling tools that will allow us to connect to SAP HANA and execute sql scripts to setup demo data that we will use from SAP Vora.After you establish connection to your HANA system. Run this code:DROP TABLE "CODEJAMMER"."EMPLOYEE_ADDRESS"; CREATE COLUMN TABLE "CODEJAMMER"."EMPLOYEE_ADDRESS" (ID bigint not null primary key generated by default as IDENTITY,"STREETNUMBER" INTEGER CS_INT, "STREET" NVARCHAR(200), "

AssumptionsThe following assumptions are made. The openldap clients and server are already installed. The basic setup of the ldap server has been completed and users "nifi admin", "nifi user1" and "nifi user2" are in the ldap database.LDAPS System ConfigurationThe example below is being configured on system nifi-sme-20. The CA certificate being used, aka truststore, is called, all-trusted.jks and the server certificate, aka keystore, is called nifi-sme-20.cert.pfx. There is also the encrypted private key for the server, nifi-sme-20.key.enc.pem which is needed for the configuration of the ldaps service. List the current certificates in the database, the default location is /etc/openldap/certs, using the following command:certutil -d /etc/openldap/certs/ -LOutput will look like the followingIf your CA is in pem format, then it can be imported into the NSS database. If you have a CA that is in jks format, it first must be converted before it can be imported.Converting it can be done

SparkR is an R package that provides a lightweight front end for using Apache Spark from R, thus supporting large-scale analytics on Hortonworks Data Platform (HDP) from the R language and environment. As of Spark 1.6.2, SparkR provides a distributed data frame implementation that supports operations like selection, filtering, and aggregation on large datasets. In addition, SparkR supports distributed machine learning through MLlib.ArchitectureSparkR which was introduced to Spark in version1.4 consists of wrappers over DataFrames and DataFrame-based APIs. In SparkR, the APIs are similar to existing ones in R (or R packages), rather than Python/Java/Scala APIs. The reason is that SparkR is very popular is primarily because it allows users to write spark jobs while staying entirely in the R framework/model. A SparkDataFrame is a distributed collection of data organized into named columns. It is conceptually equivalent to a table in a relational database or a data frame in R, but with ric

OVERVIEWThis facility is available in Zeppelin 0.7.0 based on the JIRA: https://issues.apache.org/jira/browse/ZEPPELIN-1320. The below steps allow you to get this working in HDP 2.5 environment with Zeppelin 0.6.ENVIRONMENT and SETUPI did test the below solution with HDP 2.5.0.0-1245Ambari 2.4.0.1Shiro.ini[users]# List of users with their password allowed to access Zeppelin.# To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISectionsadmin = passworduser1 = password, role1, role2user2 = password, role3#user3 = password4, role2# Sample LDAP configuration, for user Authentication, currently tested for single Realm[main]#activeDirectoryRealm = org.apache.zeppelin.server.ActiveDirectoryGroupRealm#activeDirectoryRealm.systemUsername = CN=Administrator,CN=Users,DC=HW,DC=EXAMPLE,DC=COM#activeDirectoryRealm.systemPassword = Password1!#activeDirectoryRealm.hadoopSecurityCredentialPath = jceks://user/zeppelin/ze

SYMPTOM:hive> create table mytab (col1 int) location '/tmp/abc';FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.security.AccessControlException: Permission denied: user=dbloader, access=WRITE, inode="/user/hive":hive:hadoop:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:219) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1780) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1764) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPathAccess(FSDirectory.java:1738) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.ch

When you build a cluster and enable Kerberos , by default Ambari creates/uses default principal names. here is the steps to enable the Kerberos using Blueprint/APIs. 1. Blueprint: Create a blueprint. API- POST - http://apappu3.hdp.com:8080/api/v1/blueprints/cbp Payload - { "configurations" : [ { "zoo.cfg" : { "properties" : { "dataDir" : "/data/hadoop/zookeeper", "initLimit": "5", "syncLimit": "2", "maxClientCnxns": "0" } } } ], "host_groups" : [ { "components" : [ { "name" : "HIVE_SERVER" }, { "name" : "HISTORYSERVER" }, { "name" : "HIVE_METASTORE" }, { "name" : "ZOOKEEPER_CLIENT" }, { "name" : "WEBHCAT_SERVER" }, { "name" : "ZOOKEEPER_SERVER" }, { "name" : "APP_TIMELINE_SERVER" }, { "name" : "RESOURCEMANAGER" }, {

Permissions are available under existing notebook -> padlock buttonThere are three different permissions: Owners – this is the owner(s) of the notebook. This permissions allows owners to give further - read/write - permission to that notebookReaders – these users can only read scripts created in the section of that notebook. They can neither execute nor edit the scripts - see belowWriters – these users can edit the scripts in notebook. With that permissions the users can also read the notebook. Hence, there is no need to add them to be Reader as wellReader users:When they try to execute the scriptWhen they edit the script - the screen comes up after few seconds. But the content create is not saved

In Zeppelin LDAP Authentication with OpenLDAP and How to Set Up OpenLDAP we've shown how to use LDAP Authentication with Zeppelin. In this article, we'll harden that configuration by ensuring that Zeppelin and OpenLDAP communicate over LDAPS.LDAPS is a secure protocol that uses TLS to assure authenticity, confidentiality, and integrity of communications. This prevents man-in-the-middle attacks that sniff traffic to discover LDAP credentials communicated in plaintext, which could compromise the security of the cluster.The first step is to modify the configuration of the OpenLDAP server, as root, to expose LDAPS connectivity, we'll need to modify /etc/openldap/ldap.conf. Please recall that we created /etc/openldap/certs/myldap.field.hortonworks.com.cert in the How to Set Up OpenLDAP article#TLS_CACERTDIR /etc/openldap/certsTLS_CACERT /etc/openldap/certs/myldap.field.hortonworks.com.certURI ldaps://myldap.field.hortonworks.com ldap://myldap.field.hortonworks.comBASE dc=field,dc=hortonwor

Auto-recovery in Ambari is a useful way of getting cluster components restarted automatically in the event that a component fails (without the need for human intervention).Ambari 2.4.0 introduced dynamic auto-recovery, which allows auto-start properties to be configured without needing an ambari-agent / ambari-server restart. Currently, the simplest way to manage the auto-recovery features within Ambari is via the REST API (documented within this article), although on-going work in the community will bring the feature to the UI: https://issues.apache.org/jira/browse/AMBARI-2330Check Auto-Recovery SettingsTo check if auto-recovery is enabled for all components, run the following command on the Ambari server node:curl -u admin:<password> -i -H 'X-Requested-By: ambari' -X GET http://localhost:8080/api/v1/clusters/<cluster_name>/components?fields=ServiceComponentInfo/component_name,ServiceComponentInfo/service_name,ServiceComponentInfo/category,ServiceComponentInfo/recovery_ena

Whether you're creating an Ambari cluster from scratch, taking over an existing cluster, or growing your cluster over time, it is imperative to tune Ambari and MySQL to work at a large scale of 1000-3000 Ambari Agents.Ambari Server Configs First, increase the memory used by Ambari. For large clusters, 8 GB of memory should be sufficient. If you have more than 10 concurrent users, increase it to 16 GB. Edit /var/lib/ambari-server/ambari-env.sh and change the -Xmn setting. export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms2048m -Xmx8192m Edit /etc/ambari-server/conf/ambari.properties with the following configs # The size of the Jetty connection pool used for handling incoming Ambari Agent requests.# 10 hosts => 25# 50 hosts => 35# 100 hosts => 75# 500 hosts => 100 agent.threadpool.size.max=100# Determines whether current alerts should be cached.# Enabling this can increase performance on large cluster, but can also result in lost alert data# if the cache is not flushed freque

Here is a high level overview of how to deploy Solr if approached cautiously.RequirementsInstallEdit Configuration files to work with HDFSChanging to ‘Solr’ userAdding NodesStarting SolrCloudCreating a collectionVerify & Enjoy#Requirements for this guideInstallation on a Linux server.Solr 5.5.0 requires Java 1.7 and higher.HDP 2.3 or 2.4Apache ZooKeeper — HDP 2.3 or 2.4 and Solr both use Apache ZooKeeper to manage services for the cluster. The ZooKeeper ensemble that you are using for HDP 2.3 or 2.4 can also be used by Solr.This guide doesn’t include Kerberos requirements.#Install Lucidworks-HDPsearch packageyum install lucidworks-hdpsearchAfter installation, the HDPSearch files will be found in this directory./opt/lucidworks-hdpsearch Install only — DO NOT START SOLR SERVICE YETThe Lucidworks HDP Search package should be installed manually on each node of the cluster. Follow this guide on 1 machine first. Then go and start service on each node in cluster (See Adding nodes section

SYMPTOM: HDP upgrade failed on "Finalize". Tried setting the version manually using below command which was failing -ambari-server set-current --cluster-name=<ClusterName> --version-display-name=HDP-2.5.3.0ERROR: Please find error screenshot below - ROOT CAUSE: There was issue with the last "request_id" which was run by Express upgrade process. you can check the last request_id from ambari DB from table name "host_role_command" RESOLUTION:1. Backup the ambari DB2. Find the last "request_id" from table "host_role_command" and check the status of the request_id.3. In my case the status was "Failed"4. Modifying the status of the request_id from "FAILED" to "COMPLETED" and restart ambari server.5. Try running the command againambari-server set-current --cluster-name=<ClusterName> --version-display-name=HDP-2.5.3.0

Preparing a Raspberry PI to Run TensorFlow Image RecognitionI can easily have a Python script that polls my webcam (use official Raspberry Pi webcam) , calls TensorFlow and then sends the results to NiFi via MQTT.You need to install Python MQTT Library (https://pypi.python.org/pypi/paho-mqtt/1.1)For setting up Python, Raspberry PI with Camera, see https://dzone.com/articles/picamera-ingest-real-timeRaspberry Pi 3 B+ preparationBuy a good quality 16 GIG SD Card and from OSX, Run SD Formatter to Overwrite Format the device at FAT, download here: https://www.sdcard.org/downloads/formatter_4/. Download the BerryBoot image from here. Unzip it and then copy it to your complete SD card.For examples of RPi TensorFlow You Can Run:https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/pi_examples/You need to build tensorflow for pi, which took me over 4 hours.See:https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/makefilehttps://github.com/tensorflow/ten