Community Articles

Cloudbreak 2.7.2 maintenance release is now available. If you are using an earlier version of Cloudbreak, you can upgrade now to pick up the latest bug fixes.If you are new to Cloudbreak, you can get started by launching Cloudbreak on AWS, Azure, GCP, or OpenStack from a template.Useful links:Release notesUpgrade stepsGet started

This has been tested on Ambari 2.6.2.0 and DLM 1.1.2.0If there is broken symlink or unwanted directory under /var/lib/ambari-server/resources on Ambari Server, you get below error while installing mpack(management pack) for Beacon service.[root@XXXXXX ~]# ambari-server install-mpack --mpack /root/beacon-ambari-mpack-1.1.2.0-37.tar.gz --verboseUsing python /usr/bin/python Installing management pack INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Installing management pack /root/beacon-ambari-mpack-1.1.2.0-37.tar.gz INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Download management pack to temp location /var/lib/ambari-server/data/tmp/beacon-ambari-mpack-1.1.2.0-37.tar.gz INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Expand management pack at temp location /var/lib/ambari-server/data/tmp/beacon-ambari-mpack-1.1.2.0-37/ INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO:

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'loginIdentityProvider': FactoryBean threw exception on object creation; nested exception is java.lang.Exception: The specified login identity provider 'kerberos-provider' could not be found.CloudBreak 2.7 built HDF 3.1 has error: 'kerberos-provider' could not be found.The reason is HDF 3.1's blueprint has problem:nifi.security.user.login.identity.provider= is set to kerberos-provider but that provider doesn't exist in login-identity-providers.xml, or is commented out

Usually service can be removed using API calls, but if the service is inconsistent state then API's does not work.so only way to delete is by running SQL queries. here is the list of steps to delete KNOX service.1. delete from serviceconfigmapping where service_config_id in (select service_config_id from serviceconfig where service_name like '%KNOX%') 2. delete from confgroupclusterconfigmapping where config_type like '%knox%'3. delete from clusterconfig where type_name like '%knox%'4. delete from clusterconfigmapping where type_name like '%knox%' 5. delete from serviceconfig where service_name = 'KNOX' 6. delete from servicedesiredstate where service_name = 'KNOX' 7. delete from hostcomponentdesiredstate where service_name = 'KNOX' 8. delete from hostcomponentstate where service_name = 'KNOX' 9.delete from servicecomponentdesiredstate where service_name = 'KNOX' 10.delete from clusterservices where service_name = 'KNOX' 11. DELETE from alert_history where alert_definition_id i

This article will demonstrate how to rapidly launch a Spark cluster on AWS via CloudBreak.The prerequisites are documented here. Once you have a AWS account and credentials, launching a Spark cluster is simple.CloudBreak is your command and control center UI for rapidly launching clusters on AWS, Az\ure, GCP, and on prem. Once the UI is up, add your AWS credentials Select AWS as your cloud provider Select the method for authentication. Key or Role. I prefer role but both work well. Click on the help button and follow the directions on how to setup auth for either method. Now that credentials have been setup, cluster creation may begin. Click on "Clusters" on top left and then click on "Create Cluster" on top right Select Advanced on top leftSelect Credential: Your AWS CredentialsCluster Name: Name your clusterRegion: AWS RegionPlatform Version: HDP 3.0Cluster Type: To run data science and ETL workloads, select HDP 3.0 Data Science blueprintClick Next Choose Image Type: Select Base Ima

Using cloudbreak to launch HDF (specifically NiFi), an error may prevent launching an instance. In the logs (/usr/hdf/current/nifi/logs/nifi-app.log) the following error is produced:pache.nifi.processors.hive.PutHiveStreaming could not be instantiatedjava.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.hive.PutHiveStreaming could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:232) at java.util.ServiceLoader.access$100(ServiceLoader.java:185) .... Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:239)....This is generally caused by by snappy library extracting to /tmp (generally defined in java.io.tmpdir).To fix this, do one of the following Grant nifi user access to /tmp Create a tmp directory where nifi user has access. For example, create tmp directory /home/tmp Then go into ambari and find

Zookeeper is one of the most critical components in an HDP cluster, but it is also one that is given least importance usually when tuning cluster for performance and while troubleshooting slowness in a cluster. Here is a basic checklist for zookeeper health check that one must go through to ensure that Zookeeper is running fine.Let's keep the zookeeper happy to be able to better manage the occupants of the zoo 🙂 1. Are all the Zookeeper servers given dedicated disks for transaction log directory ('dataDir' / 'dataLogDir') ?It is very important to have fast disks to complete 'fsync' of new transactions to the log, where zookeeper writes before any update takes place and before sending a response back to the client. Slower 'fsync' for transaction log is one of the most common reasons seen in the past for slower zookeeper response. Yes, the disk space requirement is usually not very high by the zookeeper and one might wonder if its worth to dedicate a complete disk to zookeeper log dire

I have been struggling to configure Spark through Mongodb with SSL. Unfortunately, the steps are not well documented there. Here is some quick guidance: Command line ( spark-shell / spark-submit / pyspark ) 1) According with the JIRA below: https://jira.mongodb.org/browse/SPARK-115?jql=project%20%3D%20SPARK%20AND%20component%20%3D%20Documentation Copy both Spark Connector and Mongo-Java-Driver among your datanodes, ( Nodes where your spark executors / driver are supposed to be running ). The Spark Connector uses the Mongo-Java-Driver and the driver will need to be configured to work with SSL. See the ssl tutorial in the java documentation. Ex. - spark_mongo-spark-connector_2.11-2.1.0.jar - mongodb_mongo-java-driver-3.4.2.jar OBS: Find yours at the mongodb website. 2) Go to ambari > Spark > Custom spark-defaults, now pass these two parameters in order to make spark (executors/driver) aware about the certificates. Example from my lab: spark.driver.extraJavaOptions=-Djavax.net.ssl.t

Starting with HDP 3.0 release, Hortonworks has stopped CentOS 6 as a supported platform. This would mean existing cluster nodes needs to be migrated to CentOS 7 before install/upgrade to HDP 3.0. However, CentOS has no official support for upgrading OS from CentOS 6 to CentOS 7. This article provides an unofficial guide for migrating cluster nodes to CentOS 7 without losing data. This is downtime migration. We would strongly recommend trying these steps on test clusters. These steps are applicable only for a smaller cluster ( <= 10 nodes ) and larger nodes we would need different strategy + automation for the same.Below are some of the assumptions or pre-requisite to be performed before upgrading the OS.Pre-RequisiteVisit all service configuration for e.x. HDFS, YARN etc make sure none of the configurations are pointing to non-root disk. If the services are pointing to root disk then upgrading the OS will format the root disk and we will lose data. Please make sure to migrate all co

Posting Images with Apache NiFi 1.7 and a Custom ProcessorI have been using a shell script for this since Apache NiFi did not have a good way to natively post an image to HTTP servers su such as the model server for Apache MXNet.So I wrote a quick and dirty processor that posts an image there and gathers the headers, result body, status text and status code and returns them to you as attributes.In this example I am download images from picsum.photos free photo service.To use this new processor, download to your lib directory and restart Apache NiFi, then you can add the PostImageProcessor.Eclipse For Building My ProcessorConfigure the Post Image Processor with your URL, fieldname, imagename and image type.MXNet Model Server ResultsThe Attribute Results From the DataResultsExample Resultspost.header{Server=[Werkzeug/0.14.1 Python/3.6.6], Access-Control-Allow-Origin=[*], Content-Length=[396], Date=[Fri, 05 Oct 2018 17:47:22 GMT], Content-Type=[application/json]}post.results{"prediction":

Short Description:Configure Knox to access Atlas UIArticleHere is the steps to access Atlas UI through Knox. 1. Make sure Knox is configured properly and it works fine.2. ssh to Knox gateway host and go to /var/lib/knox/data-2.6.****/services3. mkdir –p atlas/0.8.0/ mkdir –p atlas-api/0.8.0/4. download the configurations from https://github.com/apache/knox/tree/v0.13.0/gateway-service-definitions/src/main/resources/services/atlas-api/0.8.0 URL to /var/lib/knox/data-2.6.***/services/ atlas-api/0.8.0/5. download the configurations from https://github.com/apache/knox/tree/v0.13.0/gateway-service-definitions/src/main/resources/services/atlas/0.8.0 URL to /var/lib/knox/data-2.6.***/services/ atlas/0.8.0/5. change the owner/Group permissions to Knox for /var/lib/knox/data-2.6.**/services/atlas*/ and subdirectory6. Go to Knox configurations Modify "Advanced topology" with below service tag <service><role>ATLAS</role><url>sandbox.hortonworks.com:21000</url>&

Most data movement use cases do not require a “shuffle phase” for redistributing FlowFiles across a NiFi cluster, but there are few cases where it is useful. For example:ListFile -> FetchFileListHDFS -> FetchHDFSListFTP -> FetchFTPGenerateTableFetch -> ExecuteSQLGetSQS -> FetchS3In each case, the flow starts with a processor that generates tasks to run (e.g. filenames) followed by the actual execution of those tasks. To scale, tasks need to run on each node in the NiFi cluster, but for consistency, the task generation should only run on the primary node. The solution is to introduce a shuffle (aka load balancing) step in between task generation and task execution. Processors can be configured to run on the primary node by going to “View Configuration”-> “Scheduling” and selecting “Primary node only” under “Execution”. The shuffle step is not an explicit component on the NiFi canvas, but rather the combination of a Remote Input Port and a Remot

Objective The objective of this article is to present a workflow to capture and republish data in Kafka topics. Note that NiFi is NOT a Kafka Replication/Backup tool. This is just a workflow based approach to capture kafka data and publish it back to a fresh topic if necessary. Although storage of Kafka data is not mandatory is several cases, it may be essential in cases such as Archival – Dump Kafka Data in HDFS Corrections/Inspections – In cases where an invalid event may disrupt a downstream system, to correct and restore the messages. Start by creating two processor groups – One for the backup workflow, second for the restore workflowBackup Workflow Backup is a long running workflow which archives data to HDFS. A file in HDFS may contain a batch of events or a single event, based on how frequently messages are getting published to the source kafka topic. Hence, user does not have to stop/start this backup workflow frequently.Restore Workflow Restore a manually triggere

Properties File Lookup Augmentation of Data Flow in Apache NiFi 1.7.xA really cool technologist contacted me on LinkedIn and asked an interesting questionTim, How do I read values from a properties file and use them in my flow. I want to update/inject an attribute with this value. If you don't want to use the Variable Registry, but want to inject a value from a properties file how to do it. You could run some REST server and read it or does some file reading hack. But we have a great service to do this very easily!In my UpdateAttribute (or in your regular attributes already), I have an attribute named, keytofind. This contains a lookup key such as an integer or a string key. We will find that value in the properties value and give you that in an attribute of your choosing.We have a Controller Service to handle this for you. It reads from your specified properties file. Make sure Apache NiFi has permissions to that path and can read the file.PropertiesFileLookupService h

Overview Time to continue my series on news author personality detection! This tutorial will look at data science, Zeppelin notebooks with the idea of predicting the popularity of an article based on the personality of an author. This tutorial does assume that you completed part 1 and part 2 of the series. After completing this part, your architecture will look like this: This tutorial will be divided in sections: Section 1: Update Nifi to send article data to Phoenix Section 2: Read Phoenix data with Spark in Zeppelin Note Section 3: Train Linear Regression Model Section 4: Make and store article popularity predictions Note: This exercise is a very simplified way to apply machine learning. I spent very little time selecting the appropriate algorithm, accumulating enough data or even challenging whether or not the personality of an author is a viable predictor for personality. This is a full time job, it's called being a data scientist! But hopefully this will enable

We have seen performance issue and stability issue in heavy workload system, this is due to disk latency/Shared Disk. I.e frequent namenode failover, longer boot time, slower checkpoint, slower logging, Higher fsync will cause session expiry, etc. We never recommend a shared disk for Namenode, Journal node and Zookeeper. All of these services should have a dedicated disk. You can configure following disk type according to your HDFS workload.[dfs.namenode.name.dir] Namenode fsimage directory, dedicated disk => HDD 15K RPM [dfs.journalnode.edits.dir] Namenode edit log directory, dedicated disk => SSD [dataDir from zoo.cfg] Zoookeeper snapshot and transaction logs, for normal usage from ZKFC and HBase => HDD 15K RPM Zoookeeper snapshot and transaction logs, If it used by Nifi/ Accumulo/ Kafka/ Storm/ HBase and ZKFC => SSD If you are using RAID for meta-directory(dfs.namenode.name.dir & dfs.journalnode.edits.dir), then disable RAID and check the Non-RAID performance. Ther

Introduction With HDFS HA, the NameNode is no longer a single point of failure in a Hadoop cluster. However the performance of a single NameNode can often limit the performance of jobs across the cluster. The Apache Hadoop community has made a number of NameNode scalability improvements. This series of articles (also see part 2, part 3, part 4 and part 5) explains how you can make configuration changes to help your NameNode scale better and also improve its diagnosability. Audience This article is for Hadoop administrators who are familiar with HDFS and its components. If you are using Ambari or Cloudera Manager you should know how to manage services and configurations. HDFS Audit Logs The HDFS audit log is a separate log file that contains one entry for each user request. The audit log is a key resource for offline analysis of application traffic to find out users/jobs that generate excessive load on the NameNode. Careful analysis of the HDFS audit logs can also identify app

The service RPC port gives the DataNodes a dedicated port to report their status via block reports and heartbeats. The port is also used by Zookeeper Failover Controllers for periodic health checks by the automatic failover logic. The port is never used by client applications hence it reduces RPC queue contention between client requests and DataNode messages.Steps:1) Ambari -> HDFS -> Configs -> Advanced -> Custom hdfs-site -> Add Propertydfs.namenode.servicerpc-address.<dfs.internal.nameservices>.nn1=<namenode1 host:rpc port>dfs.namenode.servicerpc-address.<dfs.internal.nameservices>.nn2=<namenode2 host:rpc port>dfs.namenode.service.handler.count=(dfs.namenode.handler.count / 2)This RPC port receives all DN and ZKFC requests like block report, heartbeat, liveness report, etc.. Example from hdfs-site.xml,dfs.nameservices=shva dfs.internal.nameservices=shva dfs.ha.namenodes.shva=nn1,nn2 dfs.namenode.rpc-address.shva.nn1=hwxunsecure2641.openstackloc

Objective -To store multiple row versions in HBase to evaluate the impact on performance when doing reading all versions vs. getting the latest version. To put this differently, would storing multiple versions affect the performance when querying the latest version.-Using NiFi to be able to quickly ingest millions or rows into HBaseWarning -Do not store more than a few versions in HBase. This can have negative impacts. HBase is NOT designed to store more than a few versions of a cell.Step 1: Create Sample Workflow using NiFi to ingest data into HBase table -Create HBase TableDataset: https://www.citibikenyc.com/system-datacreate 'venkataw:nycstations','nycstationfam'0 row(s) in 1.3070 secondshbase(main):014:0> desc 'venkataw:nycstations'Table venkataw.nycstations is ENABLEDvenkataw.nycstationsCOLUMN FAMILIES DESCRIPTION{NAME => 'nycstationfam', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE',

1. Create keystore file first.keytool -genkey -keyalg RSA -alias ooziehost -keystore keystore.jks -validity 360 -keysize 20482. Login to Ambari server and go to Oozie configurations. 3. Add below configurations under "Advanced oozie-env"export OOZIE_HTTPS_PORT=11443export OOZIE_HTTPS_KEYSTORE_FILE=/tmp/keystore.jksexport OOZIE_HTTPS_KEYSTORE_PASS=bigdata4. Restart Oozie service from UI.5. Now launch the Oozie UI from quick links - you should be able to access the URLhttps://OOZIEHOST:11443/oozie/Note: Make sure to import the cert into Ambari Truststore.

If you have seen my article on Microservice on Hive, this is the Phoenix version. Phoenix seems to be a better option for REST microservice. I like having HBase as my main data store, it's very performant and highly scalable for application style queries.See: https://community.hortonworks.com/articles/53629/writing-a-spring-boot-microservices-to-access-hive.htmlThis microservices is a Spring Boot REST service on top of the data loaded by this NiFi data flow.See: https://community.hortonworks.com/content/kbentry/54947/reading-opendata-json-and-storing-into-phoenix-tab.htmlPom<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.dataflowdeveloper</groupId> <artifactId>phoenix</artifactId> <version>

The following link has some sample code to connect to a secure Hive Server2 via the default connection string and via knox. https://github.com/vspw/hiveJDBCIt uses the following Jar files:-commons-configuration-xx.jar (UGI set configurations and metrics)commons-logging-xx.jar (*Not mandatory)hadoop-auth-xx.jarhadoop-common-xx.jar (UGI stuff, hadoop configurations)hive-jdbc-1.2xx-standalone.jarlog4j-xx-api-xx.jar (for log4j)log4j-api-xx.jar (for log4j)log4j-core.xx.jar (for log4j)xercesImpl-xx.jar It also reads a bunch of properties like "keytabLocationWindows", "jdbcConnStringDirect", "hiveQuery" etc. from a properties file and logging is enabled with log4j2.xml.The class: "HiveJDBCKnox" , has some instructions to make knox connectivity work on windows platform as well.

Got error [User home directory not found] when creating WorkFlow Manager view in AmbariResolve:sudo -u hdfs hdfs dfs -mkdir /user/adminsudo -u hdfs hdfs dfs -chown admin /user/adminResule:

SYMPTOMOozie web UI doesn't work on Internet explorer in Kerberized environment, it gives below error message even after making sure that Ext JS libraries are installed correctly.Oozie web console is disabled.To enable Oozie web console install the Ext JS library. ROOT CAUSEhttps://issues.apache.org/jira/browse/OOZIE-2322 - Because of empty space as a value of oozie.authentication.cookie.domainWORKAROUND1. Set value of oozie.authentication.cookie.domain to name of your corporate domain in oozie-site.xml via Ambari UI.Example:<property><name>oozie.authentication.cookie.domain</name><value>hortonworks.com</value></property>2. Stop Oozie server via Ambari.3. Move oozie.war to some other location.mv /usr/hdp/current/oozie-server/oozie-server/webapps/oozie.war /root/4. Move Oozie webapp directory to some other location. mv /usr/hdp/current/oozie-server/oozie-server/webapps/oozie /root/5. Regenrate Oozie war file using below command/usr/hdp/current/oozie-s

Related: ApacheCon 2018 in MontrealWith my talk in a dual language city, I thought perhaps I should do my talk in French. My college French is very rusty and my accent is very New Jersey. The two don't mix well. So let's have Apache NiFi do it for me. After publically debating this on Twitter, I decided to see if I could implement a solution.Secret:Most of the heavy lifting is done by Python which calls Google Translate API under the covers, automagically.My Presentation is here: https://www.slideshare.net/bunkertor/apache-deep-learning-101-apachecon-montreal-2018-v031Flow to Extract FrenchApache Tika extracts the text from PDF or PPTX and converts to text or HTML. ( I chose text). Run the Translate Python with a sentence extracted from the PDF or PPTX.Let's send that translated french line to it's own slack channel.Let's send the english to another.And there it is:runtranslate.shpython3.6 -W ignore /Volumes/TSPANN/2018/talks/IOT/translate.py "$1" 2>/dev/nulltranslate.py