Resources

1. You can update the share lib with the following jars or can be directly passed in oozie workflow.xml. ( Make sure you use 3.2 version not 4.x datanucleus jars )/usr/hdp/current/spark-client/lib/datanucleus-rdbms-3.2.9.jar /usr/hdp/current/spark-client/lib/datanucleus-core-3.2.10.jar/usr/hdp/current/spark-client/lib/datanucleus-api-jdo-3.2.6.jar To copy jars to sharelib#hdfs dfs –put /usr/hdp/current/spark-client/lib/datanucleus-rdbms-3.2.9.jar /user/oozie/share/lib/lib_*/spark/ If you copy the jars to sharelib make sure run oozie sharelibupdate Update oozie sharelib:# oozie admin -oozie http://<oozie-server>:11000/oozie -sharelibupdate Verify the current spark action sharelib with all the above files:# oozie admin -oozie http://<oozie-server>:11000/oozie -shareliblist spark* Make sure you have hive-site.xml in sharelib too and have the following properties in it. Replace the values with your hive-site.xml values.<configuration><property><name>hive.me

When NiFi flow runs in mission critical production environment, customer would concern about how to change NiFi flow logic but don't impact business.One solution was implemented in customer production is as follow steps:1. Add an extra non-logic processor, for example "UpdateAttribute", between ListenPort and main ProcessGroup2. Updated NewProcessGroup, add into canvas3. Stop "UpdateAttribute" process, make sure RunningProcessGroup consume all flowfiles in the previous queue. But client components can still send messages to Listen Port4. After all queued flowfiles are processed by RunningProcessGroup, move the queue end connection to the NewProcessGroup5. Start "UPdateAttribute" process and NewProcessGroup6. At the end, remove the old RunningProcessGroupFollow these steps, the production flow change is transparent for client components, and guarantee no data lost.

A common error to see in initial installations is the following from Accumulo TabletServer logsCaused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /apps/accumulo/data/wal/myhost.mydomain.com+9997/1ff916a2-13d0-4bb7-aa38-c44b69831519 could only be replicated to 0 nodes instead of minReplication (=1).  There are 3 datanode(s) running and no node(s) are excluded in this operation.    at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1649)    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3198)    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3122)    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:843)    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:500)    at org.apache.hadoop.hdfs.prot

Error message:Below error message from dmesg.txt[1294523.092549] python2.6[11900]: segfault at 11 ip 00007f36b7cd4e90 sp 00007ffd31ce5830 error 4 in libcrypto.so.1.0.0[7f36b7baf000+1d5000][1294523.530806] python2.6[11901]: segfault at 11 ip 00007f33ef805e90 sp 00007ffe41443280 error 4 in libcrypto.so.1.0.0[7f33ef6e0000+1d5000][1294523.934818] python2.6[11902]: segfault at 11 ip 00007f29280d5e90 sp 00007fff7b5facc0 error 4 in libcrypto.so.1.0.0[7f2927fb0000+1d5000][1294524.379260] python2.6[11903]: segfault at 11 ip 00007f2cfd50be90 sp 00007ffe17504610 error 4 in libcrypto.so.1.0.0[7f2cfd3e6000+1d5000]Below error message from /var/log/hue/supervisor.log[15/Feb/2017 17:43:58 +0000] supervisor ERROR Exception in supervisor main loopTraceback (most recent call last): File "/usr/lib/hue/desktop/core/src/desktop/supervisor.py", line 414, in main wait_loop(sups, options) File "/usr/lib/hue/desktop/core/src/desktop/supervisor.py", line 424, in wait_loop time.sleep(1) File "/usr/l

For this tutorial we will be using Ubuntu 14.04.5. This setup can further be leveraged with Apache Metron (Incubating). Additional installation instructions for Metron core will be provided in another article.Install Elasticsearch First we'll install Elasticsearch 2.4. You'll need the following prerequisites: wget apt-transport-https Java You can install them by logging into your ES node and executing the following: sudo apt-get updatesudo apt-get install -y wget apt-transport-https# If using oracle jdk 8sudo apt-get install -y software-properties-commonsudo apt-add-repository -y ppa:webupd8team/javasudo apt-get updateecho "oracle-java8-installer shared/accepted-oracle-license-v1-1 select true" | sudodebconf-set-selectionssudo apt-get install -y oracle-java8-installer Now let's install Elasticsearch. Run the following commands on the node where you want to install ES. # Get the Elasticsearch packageswget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key ad

How to connect GetKafka to Kafka through StunnelStunnel is a proxy that can make insecure network transmission secure by wrapping it with SSL. This article contains example and illustrations describing how it works and how to configure it.Most part of it is derived from this informative Git comment I wouldn't be able to set it up without this comment. Thank you for sharing such detailed example. How it works?Let's see how it can be applied to NiFi GetKafka. I used two servers for this experimentation. 0 and 1.server.aws.mine. A single Zookeeper and Kafka broker is running on 0.server. A GetKafka NiFi processor in 1.server consumes messages through Stunnel:Kafka Broker joins the Kafka cluster and declares its address as 127.0.0.1:9092. If Zookeeper is in different server (recommended) and you need to secure this connection via Stunnel as well, then you need to apply the same method as the one used between GetKafka and Zookeeper.GetKafka's Zookeeper Connection String is set to 127.0.0.1

ISSUE : Even after configuring falcon to use https, ambari alert checks Falcon UI on http and fails. Falcon service will be running fine but Ambari shows Falcon WEB UI alert. ROOT CAUSE : Ambari alert script is hardcoded to check Falcon on http WORKAROUND : 1. Login to Ambari UI 2. Open one more tab and go to this UI : http://AMBARI_HOST:8080/api/v1/clusters/CLUSTER_NAME/alert_definitions 3. Search for falcon_server_webui and get the alert definitions ID.Example : { "href" : "http://mon-ravi.ironman.com:8080/api/v1/clusters/ironman/alert_definitions/301", "AlertDefinition" : { "cluster_name" : "ironman", "id" : 301, "label" : "Falcon Server Web UI", "name" : "falcon_server_webui" }301 is the alert definition here. 4. Delete this alert definition using curl curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://AMBARI_HOST:8080/api/v1/clusters/CLUSTER_NAME/alert_definitions/301 5. On the ambari server, edit this file /var/li

Documentation It is recommended to create the following documents as part of the Upgrade Planning and Execution phase. ● Upgrade plan ● Upgrade guide ● Upgrade log (list of issues and their workarounds) ● Validation test results ● Hand-over documents Learnings from the Field Phased approach to Upgrades ● Upgrade Dev/Test cluster first before touching production environment. This will catch issues which are specific to the environments. ● Upgrade Dev/Test cluster first, create a runbook along with all issues which occurred during upgrade ●Just setting up fresh cluster with the new version in Dev and testing applications is not enough. While it does take care of application tests, upgrade issues are not tested. It is recommended to upgrade Dev/Test in line with upgrade path for Production. Updating Configurations ● Any major update to the configurations should be designed and tested in advance. ● For example – Capacity Scheduler ●Recommend to prep

OVERVIEWDocker supplies multiple storage drivers to manage the mutable and immutable layers of images and containers. Many options exist with varying pros and cons. Out of the box, docker uses devicemapper loop-lvm. The loop-lvm storage driver is not recommended for production, but requires zero setup to leverage.When attempting to increase the base size of the mutable layer, it was observed that docker client operations slow. The alternative of using smaller base layers causes failures due to out of storage conditions.The intent of this article is to outline testing that was performed to determine sane defaults for the docker storage driver options.TESTINGThe following testing methodology was used: Build the centos6 image with different combinations of base sizes and storage drivers (build)Create a container from the image (run)Stop the container (stop)Remove the container (rm)Stop dockerDelete/reprovision the docker graph storage locationRepeatThe following scenarios were tested: loo

Considering data is often times being "sent into NIFI" from other processes, and "sent out of NIFI" to other processes, data safety is a legitimate concern. When data is being exchanged between processes, we have the following possible delivery semantics:1. at most once delivery2. at least once delivery3. exactly once deliveryFirst of all, speaking of the communication between multiple NIFI/MINIFi instances via site-to-site protocol, because of the 2 phase commit provided at the framework level, at least once delivery is guaranteed. And not only that, the chance of "exactly once" delivery is maximized. "Exactly once delivery" isn't guaranteed only if your system breaks between the two commits (literally between those two lines of code), or if there was any error in NIFI when committing the state of things. As an example, after data is being delivered to the receiving side, but before your sending side commits, network connection is somehow lost. In that case, sending side will try to s

Storm 1.1.X provide an external storm kafka client that we could use to build storm topology. Please note this is support for Kafka 0.10 onwards. Below is the step by step guide on how to use the API's.Add below dependency to your pom.xml<dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-kafka-client</artifactId> <version>1.1.1-SNAPSHOT</version></dependency>The kafka spout implementation for the topology is configured using KafkaSpoutConfig. Below is a sample config object creation.KafkaSpoutConfig spoutConf = KafkaSpoutConfig.builder(bootStrapServers, topic) .setGroupId(consumerGroupId) .setOffsetCommitPeriodMs(10_000) .setFirstPollOffsetStrategy(UNCOMMITTED_LATEST) .setMaxUncommittedOffsets(1000000) .setRetry(kafkaSpoutRetryService) .setRecordTranslator (new TupleBuilder(), outputFields, topic ) .build();Above class follows builder pattern. bootS

Assumption - HDP 2.5.3 and above versions and kerberized cluster. Create a hplsql-site.xml as per the below template. <configuration><property><name>hplsql.conn.default</name><value>hive2conn</value><description>The default connection profile</description></property><property><name>hplsql.conn.hiveconn</name><value>org.apache.hive.jdbc.HiveDriver;jdbc:hive2://</value><description>HiveServer2 JDBC connection (embedded mode)</description></property><property><name>hplsql.conn.init.hiveconn</name><value>set hive.execution.engine=mr;use default;</value><description>Statements for execute after connection to the database</description></property><property><name>hplsql.conn.convert.hiveconn</name><<value>true</value><description>Convert SQL statements before execution</description><</property&

I found it is a little tricky to get started with a Kerberos enabled Kafka cluster. I created this step by step recipe for securing Kafka with Kerberos, sending and receiving data on console. This is tested on HDP2.5.0 and Ambari 2.4.1.Enabled Kerberos using the Ambari Kerberos setup wizard under Admin -- Kerberos menu.On Ambari Kafka Config UI, change "listeners" property to "PLAINTEXTSASL://localhost:6667".Restart Kafka as requested by Ambari.Create a test topic in Kafka. Must use the kafka service user to do this.$ cd /usr/hdp/current/kafka-broker/bin$ sudo su kafka$ kinit -k -t /etc/security/keytabs/kafka.service.keytab kafka/ip-10-0-1-130.ap-northeast-1.compute.internal$ ./kafka-topics.sh --zookeeper ip-10-0-1-130.ap-northeast-1.compute.internal:2181 --create --topic foo --partitions 1 --replication-factor 1Created topic "bar".Grant permission to user. This can be done using Kafka native ACL mechanism or Apache Ranger. In the example, we use Kafka ACL. User bob needs to be existin

Picture the scene...You've run the HDFS Balancer on your cluster and have your data balanced nicely across your DataNodes on HDFS. Your cluster is humming along nicely, but your system administrator runs across the office to you with alerts about full disks on one of your DataNode machines! What now?The Low-DownUneven data distribution amongst disks isn't dangerous as such, though in some rare cases you may start to notice the fuller disks becoming bottlenecks for I/O. As of Apache Hadoop 2.7.3, it is not possible to balance disks within a single node (aka intra-node balancing) - the HDFS balancer only balances across DataNodes and not within them. HDFS-1312 is tracking work to introduce this functionality into Apache HDFS, but it will not be available before Hadoop 3.0.The conservative approach:Modify the following property to your HDFS configurations or add it if it isn't already there: dfs.datanode.du.reserved (reserved space in bytes per volume). This will always leave this much sp

Use CaseI want to hide text messages inside images. When the images arrive somewhere else, I want to extract those messages.It let's you hide text in images, binaries in images and images in images.I was interesting in hiding text messages in images. After seeing

Linkedin PostApache NiFi is an easy to use, powerful, and reliable system to process and distribute data.Apache NiFi is based on technology previously called “Niagara Files” that was in development and used at scale within the NSA for the last 8 years and was made available to the Apache Software Foundation through the NSA Technology Transfer Program. Some of the use cases include, but are not limited to: Big Data Ingest– Offers a simple, reliable and secure way to collect data streams.IoAT Optimization– Allows organizations to overcome real world constraints such as limited or expensive bandwidth while ensuring data quality and reliability.Compliance– Enables organizations to understand everything that happens to data in motion from its creation to its final resting place, which is particularly important for regulated industries that must retain and report on chain of custody.Digital Security– Helps organizations collect large volumes of data from many sources and prioritize which dat

Many deployments in hadoop use Centrify LDAP proxy to integrate Active Directory users with linux accounts. Centrify ldapproxy uses posix properties like uid, gidNumber, uidNumber, memberUid, etc... and are made available for ldap search attributes to the clients like Apache Ranger. Ranger interacts with Centrify ldapproxy for configuring hadoop authorization policies as well as for authentication to ranger UI. In order to sync users and groups from Centrify, ranger performs ldap search against Centrify using all the configured LDAP attributes. Some of the common issues noticed during this integration can be:Issue #1. Users are unable to login to Ranger UI but were successfully sync'd to ranger:For AD/LDAP authentication, Ranger uses the complete DN of the login user for performing an LDAP search. Ranger also uses group search base and group search filter for group membership for obtaining user role information. Few configuration parameters to verify on ranger are:ranger.ldap.user.dnpa

Demonstrate how easy is to create a simple data flow with NiFi, stream to Hive and visualize via Zeppelin.Pre-requisites Apache NiFi 1.1.0.2.1.0.0-165, included with Hortonworks DataFlow 2.1.0Apache Zeppelin 0.6.0.2.5.0.0-1245, included with Hortonworks Data Platform 2.5.0My repo for Apache NiFi "CSVToHive.xml" template, customer demographics data (customer_demographics.header, customer_demographics.csv), "Customer Demographics.json" Apache Zeppelin notebook, customer_demographics_orc_table_ddl.hql database and table DDLsApache Hive 1.2.1 included with HDP 2.5.0Hive configured to support ACID transactions and demo database and customer_demographics created using customer_demographics_orc_table_ddl.hqlStepsImport NiFi TemplateAssuming NiFi is started and the UI available at <NiFiMasterHostName>:8086:/nifi, import the template CSVToHive.xml:screen-shot-2017-03-06-at-74106-pm.pngCreate Data Folder and Upload Data FilesIn your home directory create /home/username/customer_demographic

1. HA provider for webhdfs is needed in your topology. <provider> <role>ha</role> <name>HaProvider</name> <enabled>true</enabled> <param> <name>WEBHDFS</name> <value>maxFailoverAttempts=3;failoverSleep=1000;maxRetryAttempts=300;retrySleep=1000;enabled=true</value> </param></provider> 2. The namenode service url value should contain your name service ID. (This can be found in your hdfs-default.xml under parameter dfs.internal.nameservices) <service> <role>NAMENODE</role> <url>hdfs://chupa</url></service> 3. Make sure webhdfs url for each namenode is added in your WEBHDFS service area. <service> <role>WEBHDFS</role> <url>http://chupa1.openstacklocal:50070/webhdfs</url> <url>http://chupa2.openstacklocal:50070/webhdfs</url></service> 4. Here is a working topology using the knox defau

In previous post we have seen how to Automate HDP installation with Kerberos authencation on single node cluster using Ambari Blueprints. In this post, we will see how to Deploy multinode node HDP Cluster with Kerberos authentication via Ambari blueprint..Note - For Ambari 2.6.X onwards, we will have to register VDF to register internal repository, or else Ambari will pick up latest version of HDP and use the public repos. please see below document for more information. For Ambari version less than 2.6.X, this guide will work without any modifications.Document - https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-release-notes/content/ambari_relnotes-2.6.0.0-behavioral-changes.html.Below are simple steps to install HDP multi node cluster with Kerberos Authentication(MIT KDC) using internal repository via Ambari Blueprints..Step 1: Install Ambari server using steps mentioned under below linkhttp://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-installation/con

Objective This tutorial will walk you through the process of using Ansible to deploy Hortonworks Data Platform (HDP) on Amazon Web Services (AWS). We will use the ansible-hadoop Ansible playbook from ObjectRocket to do this. You can find more information on that playbook here: ObjectRocket Ansible-Hadoop This tutorial is part 2 of a 2 part series. Part 1 in the series will show you how to use Ansible to create instances on Amazon Web Services (AWS). Part 1 is avaiablle here: HCC Article Part 1 This tutorial was created as a companion to the Ansible + Hadoop talk I gave at the Ansible NOVA Meetup in February 2017. You can find the slides to that talk here: SlideShare Prerequisites You must have an existing AWS account. You must have access to your AWS Access and Secret keys. You are responsible for all AWS costs incurred. You should have 3-6 instances created in AWS. If you completed Part 1 of this series, then you have an easy way to do that. Scope This tutorial was tested using the

Found in version Ambari 2.4.2.0 Generation of SmartSense bundle fails with following error:ERROR 2017-02-24 10:52:53,284 shell.py:95 - Execution of command returned 1. Exception in thread "main" java.lang.IllegalArgumentException: Illegal group reference at java.util.regex.Matcher.appendReplacement(Matcher.java:857) at java.util.regex.Matcher.replaceAll(Matcher.java:955) at java.lang.String.replaceAll(String.java:2223) at com.hortonworks.smartsense.anonymization.BundleAnonymizer.handleApplyGroupPatternKeyValue(BundleAnonymizer.java:690) at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyGroupPattern(BundleAnonymizer.java:673) at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyPropertyRule(BundleAnonymizer.java:612) at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyRules(BundleAnonymizer.java:393) at com.hortonworks.smartsense.anonymization.BundleAnonymizer.anonymizeFolder(BundleAno

Pre-requsite: 1.Make sure /etc/hosts has the correct ip address and the hostnames and the master hostnames can be pinged from each other . 3. OpenLDAP to be setup. 4. Make sure the logs are captured in a valid log file for slapd process as follows: vi /etc/rsyslog.conf Append the follwowing line to log OpenLDAP messages and restart rsyslogd service: # Point SLAPD logs to /var/log/slapd.loglocal4.* /var/log/slapd.logservice rsyslog restart HA OpenLDAP Sync Setup: The following ldif files needs to be created on the users home directory and should be executed on all the master nodes: basednupdate.ldif syncproc_module.ldif syncproc.ldif addladpservers.ldif NOTE: Please change the cn , dc accordingly. vi /tmp/basednupdate.ldif # updated your base dn below:dn: olcDatabase={1}monitor,cn=configchangetype: modifyreplace: olcAccessolcAccess: {0}to * by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth"read by dn.base="cn=Manager,dc=srv

Objective This tutorial will walk you through the process of using Ansible, an agent-less automation tool, to create instances on AWS. The Ansible playbook we will use is relatively simple; you can use it as a base to experiment with more advanced features. You can read more about Ansible here: Ansible. Ansible is written in Python and is installed as a Python module on the control host. The only requirement for the hosts managed by Ansible is the ability to login with SSH. There is no requirement to install any software on the host managed by Ansible. If you have never used Ansible, you can become more familiar with it by going through some basic tutorials. The following two tutorials are a good starting point: Automate All Things With Ansible: Part One Automate All Things With Ansible: Part Two This tutorial is part 1 of a 2 part series. Part 2 in the series will show you how to use Ansible to deploy Hortonworks Data Platform (HDP) on Amazon Web Services (AWS). This tutorial was cr

Business Need: Build Tool Notifications + FunSonicPi is an open source, multiple platform live music synthesizer tool for building music via coding. It's a pretty neat environment and runs on a Mac, RPi, Windows machine or Linux machine. I found a command-line tool that can trigger SonicPi to play your code. I have wired up a dataflow from NiFi to SonicPi, so I can trigger music on demand. My thought is I will tie this into Jenkins and play different songs depending on what's going on. Such as Test Failures, Build Complete, Build Broken, New Deployment, based on various steps in the SDLC.Note: The machine this running on will need Ruby, Gem, Apache NiFi, Sonic Pi and a sound card.Step 1:To trigger Sonic Pi, you must install Sonic Pi Cli, via gem install sonic-pi-cli. Step 2:Run the SonicPI software and have it up and running. As you can see it's a pretty awesome environment, somewhere between a nice IDE and a music tracker.Step 3: ExecuteStreamCommand This calls the com