Community Articles

From Database MetaData to Data Ingest in a Few Quick Steps A company wants to know when new tables are added to a JDBC source (say a RDBMS). Using the ListDatabaseTables processor we can get a list of TABLEs (also views, system tables and other database objects), but for our purposes we want tables with data. I have used the ngdbc.jar from SAP HANA to connect and query tables with ease. For today's example I am connecting to MySQL as I have a MySQL database available for use and modification. Pre-Step mysql -u root -p test < person.sqlCREATE USER 'nifi'@'%' IDENTIFIED BY 'reallylongDifficultPassDF&^D&F^Dwird';GRANT ALL PRIVILEGES ON *.* TO 'nifi'@'%' WITH GRANT OPTION;COMMIT;mysql> show tables;+----------------+| Tables_in_test |+----------------+| MOCK_DATA || mock2 || personReg || visitor |+----------------+4 rows in set (0.00 sec) I created a user to use for my JDBC Connection Pool in NiFi to read the metadata and data. These ta

Introduction Finally, we get to put all this together! This is less of a tutorial, and more of a show case of the capabilities of this solution. Please refere to previous articles to build this on your own. The meat of this article is showcasing the React/NodeJS application you can build to communicate with Nifi, Cloudbreak and your ephemeral cluster running Zeppelin and Spark. Architecture Below is a high-level architecture of the solution: You can find all the code for this here: https://github.com/paulvid/bmq-app The code includes: Latest Zeppelin Prediction Notebook All Nifi templates All Blueprints/Recipes for Cloudbreak The source code of the app Agenda This article is divided in the following sections: Section 1: Monitoring your current data with Nifi Section 2: Launching an ephemeral cluster with Cloudbreak Section 3: Running a prediction model with Spark Zeppelin Section 1: Monitoring your current data with Nifi   Section 2: Launching an ephemeral cluster with Cloud

Introduction It's been a while, but I'm finally finishing this series on Best Mode Quotient. This article is fairly straight forward as we look how to build the infrastructure of model training using Zeppelin and Spark.Pre-Requisites In order to run this install, you will have to have deployed a BMQ ephemeral cluster as detailed in this article and this repository. Moreover you will have to have gathered data on your fitness as explained in part 1 of these series. Based on this data, I created 3 indexes: Intensity Index: How intense was the workout I did that day (based on distance, pace and elevation) Fatigue Index: How much rest I had on that day (based on hours of sleep and rest heart rate) BMQ Index: How my BMQ fairs compare to the max BMQ (5) You can then agglomerate these 3 indexes, having BMQ and Fatigue on the same day correlating to the Intensity Index of the previous day. All data will be shared in the last article of the series. You will also notice that the model uses

Customizing Atlas: Summary of Work to DateArticleKey PointsCustomized Type(s) DevelopedPart 1: Model governance, traceability and registry Quick primer on Atlas types, entities, attributes, lineage, searchQuick primer on customizing AtlasUse the Atlas Rest API to customize any type, entity, attributes you wishCustomizations integrate seamlessly with out of the box Atlas lineage and search capabilitiesNotes on operationalizing model to represent your deployed data science and complex Spark ETL models (what was deployed, which version, when, what are its concrete artifacts, etc)Part 2: Deep source metadata & embedded entities Use the Atlas Rest API to customize any type/entity/attributes you wishYou can use a hyplerlinked entity (vs text) as a value to an attribute (embedded entity pattern)HDFS entities can hold deep metadata from source device to represent a hardware device (in this case gene sequencing device)gene_sequence to represent gene sequencing data landed in HDFS, as well

In Apache NiFi, Controller Services are shared services that can be used by Processors, Reporting Tasks, and other Controller Services. In order to modify a Controller Service, all referencing components must be stopped. This means that the user attempting to modify the Controller Service, must also have permissions to modify all referencing components. With the introduction of multi-tenant authorization, Controller Service scoping was updated to ensure service usage does not become so broad that it becomes impossible to update the Controller Service because no users are allowed to modified all components that reference it. Within the dataflow, to scope a Controller Service and limit it's availability, a Controller Service can be created within any Process Group. Controller Services created within a Process Group will be available to all descendent components. This means that the service will be available to all Processors and Controller Services defined in that Process Group and below

CVE-2018-8012 vulnerability is reported here, https://nvd.nist.gov/vuln/detail/CVE-2018-8012   o Affected Versions: Apache ZooKeeper before 3.4.10, and 3.5.0-alpha through 3.5.3-beta. o Mitigations: - The Apache JIRA (ZOOKEEPER-1045) is fixed @ Zookeeper 3.4.10. - Currently, the Fix for ZOOKEEPER-1045 (Zookeeper 3.4.10) isn't planned for any HDP release as the vulnerability is less of a vulnerability and more of a lack of a feature - Zookeeper version 3.4.6 is present from HDP 2.3.0 to HDP 2.6.x   o Workaround: - The workaround for this is to implement firewall rules to disallow connections from ports other than 2181. - Customer may try to move the ZooKeeper Election and Follower ports into the privileged port range, but this would require additional logic to start ZK as root and then drop privileges which does not exist as of now. - As such, the feasible workaround for this CVE is to configure firewall to allow communication [for Leader Election] within cluster. A possibl

Introduction SoChain provides a fast set of public freely available APIs (don't abuse them) to access information on various networks. If you need this for critical work, please donate: https://chain.so/address/devfund. One of the things you will see in this simple flow is that NiFi excels in ingesting REST and working with JSON. As you can see with split it up, shred it, filter it, manipulate and extract from it. With the resulting usable objects we build a schema that will also us to do record processing. Once we have a set of records with a schema I can store it to I just hosted a Future of Data Princeton Meetup in Woodbridge New Jersey with some amazing speakers sponsored by ChainNinja. While this was all about Blockchain for Enterprise and no cryptocurrency was involved it made we want to investigate some cryptocurrency data. As you can see, manipulating complex JSON data, filtering, modifying, routing and scripting with it's values is trivial in Apache NiFi.

Less than 5% of the support cases have recommendations to engage professional services. I have seen this message create mixed emotions in the customer's mind. I would like to wear both customers and support engineer's hat to see how this can be handled efficiently.As a customer, I will be delighted if solutions come instantly, a setting like “hive.run.query.faster=true or spark.fix.error=true” would be very ideal. Requesting customers to connect with a different team could often be seen as a deflection strategy from support. Also, professional service engagement has its own sales cycle and costs are attached to it. This results in frustration.Let’s take an example of one such redirection, Customer requests support’s help refine their use case and suggest the right technology stack. From a support point of view, this request is considered out of scope and the engineer would direct the customer to engage professional services. As a professional services consultant in my past life, use ca

Working with a Proximity Beacon Network Part 1Introduction:In a retail environment, we want to be able to interact with people within the store. Our beacons can provide hyperlocalized information and also help us determine what's going on in the store for traffic patterns. Our beacons are also giving us temperature and other reading as any sensor we may have in our IoT retail environment.I have set up three Estimote Proximity beacons in an indoor environment broadcasting both Estimote and IBeacon messages.iBeacon is a Bluetooth advertising protocol by Apple that is built into iPhones. In this article we ingest, filter and route the data based on the beacon IDs. In a following article we will stream our data to a data store(s) and run machine learning and analytics on our streaming time series data.I tested the BLE library from the command line with a Python script:Cloud SetupSince we are using our own IoT gateways and networks, I am only using the Estimote Cloud to check the beac

The Data Load ProcessWe looked at the performance that these engines have in the last article, now it’s time to look at how the data got loaded in. There are trade-offs here to be aware of when loading data into each of these engines, as they use different mechanisms to accomplish the task.Hive LoadThere is no immediate need for a Schema-on-Write data load when you are using Hive with your native file format. Your only “load” operation is copying the files from your local file system to HDFS. With schema-on-read functionality, Hive can instantly access data as soon as its’ underlying file is loaded into HDFS. In our case the real data load step was in converting this Schema-on-Read external Hive table data into optimized ORC format, therefore loading it from an external table to a Hive-managed table. This was a relatively short process, coming in much under an hour.HBase LoadContrast that with HBase, where a bulk data load for our sample data set of 200M rows (around 30GB of disk size

As we know that the logs are important however many times we see that the logs consumes a lots of disk space. So if we want to use the log4j provided approach to control the logging behavior in a much efficient way, then in that case we can take advantage of the "apache-log4j-extra" package. More information about the extra packages can be found in :https://logging.apache.org/log4j/extras/In this article we will see how to use the log compression (zip) feature of log4j to compress the NameNode logs on a daily basis automatically.In order to achieve the same we will need the following Steps:Step-1). As the extras features are not shipped with the default log4j implementation hence the users will need to download the "Apache Extras™ for Apache log4j" (Like: apache-log4j-extras) : https://logging.apache.org/log4j/extras/download.htmlExample: For example download the jar "apache-log4j-extras-1.2.17.jar" and place it inside the Hadoop library location./usr/hdp/2.4.2.0-258/hadoop/lib/apache-

Overview There are many useful articles as well official Cloudbreak documentation covers everything in great depth. This short article walks you through how to deploy the Cloudbreak instance within the existing VPC and Subnet using the AWS Quickstart deployment. Cloudbreak deployment options The cloudbreak deployment options are explained in detail here. If you notice the AWS specific Networking options, the Quickstart by default creates the new VPC and for the custom VPC the production install is recommended. In case you are doing poc and quickly want to try the Quickstart option but wanted to use the existing VPC, you can do that by enhancing the CloudFormation template which is described in next session. CloudFormation template changes When you launch the CloudFormation template for AWS Quickstart by default it selects the existing CloudFormation template https://s3.amazonaws.com/cbd-quickstart/cbd-quickstart-2.7.0.template. Instead of using the default template use t

I need to parse Kerberos KDC Log files (including the currently filling file) to find users with their host that are connecting.It seems using Grok in NiFi we can parse out a lot of different parts of these files and use them for filtering and alerting with ease.This is what many of the lines in the log file look like:Jan 01 03:31:01 somenewserver-310 krb5kdc[28593](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.237.220: ISSUE: authtime 1546278185, etypes {rep=18 tkt=16 ses=18}, nn/somenewserver-310.field.hortonworks.com@HWX.COM for nn/somenewserver-310.field.hortonworks.com@HWX.COMState of the Tail ProcessorTail a FileWe also have the option of using the GrokReader listed in an article included to immediately convert matching records to output formats like JSON or Avro and then partition into groups. We'll do that in a later article.In this one, we can get a line from the file via Tail, read a list of files and fetch one at a time or generate a flow file for testing. Once we had

The following parameters decide if ACID is enabled or disabled in your cluster. ACID Enable:hive.support.concurrency=true;hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;hive.enforce.bucketing=true;hive.exec.dynamic.partition.mode=nostrict;*** The following parameters are required for standalone hive metastore ***hive.compactor.initiator.on=true;hive.compactor.worker.threads=1NOTE: Even though HiveServer2 runs with an embedded metastore, a standalone Hive metastore is required for ACID support to function properly. If you are not using ACID support with HiveServer2, you do not need a standalone metastore.ACID Disable:If not using ACID, then make sure the value of the parameters are set as below to avoid any hive locking issueshive.support.concurrency=false;hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager;hive.enforce.bucketing=false;hive.exec.dynamic.partition.mode=strict;hive.compactor.initiator.on=false;hive.compactor.worker.threads=0;

Problem:some of the rest call you might not face this issue , but some cases where A service to service call is made , for ex. hbase makes call to ranger admin to download the policy using following rest call.https://localhost:6182/service/plugins/policies/download/in this case ranger admin trust store should have certificate of Client who is trying to download the policy. so for example if hbase tries to download the policy then since we had already setup the ssl for ranger and plugins and ranger admin is having certificate for hbase plugin in ranger admin trust store so this call will work finebut if you try to make this call using curl from your own rest client then it will failcurl -k -u admin:admin 'https://localhost:6182/service/plugins/policies/download/' it will throw following kind of error:ERROR org.apache.ranger.common.ServiceUtil (ServiceUtil.java:1376) - Unauthorized access. Unable to get client certificate. serviceName=cl1_hadoop2016-07-06 05:51:46,264 [http-bio-6182-exec

Here is a self explanatory tutorial. The official Cloudbreak documentation explains how to retrieve your Cloudbreak token via curl or python (see here).I have been playing with Node.JS on my side, so here is how to do it in Node.JS using request:process.env["NODE_TLS_REJECT_UNAUTHORIZED"] = 0;var request = require("request");var options = { method: 'POST', url: 'https://[YOUR_CB_URL]/identity/oauth/authorize', qs: { response_type: 'token', client_id: 'cloudbreak_shell', 'scope.0': 'openid', source: 'login', redirect_uri: 'http://cloudbreak.shell', accept: 'application/x-www-form-urlencoded' }, headers: { 'Content-Type': 'application/x-www-form-urlencoded', accept: 'application/x-www-form-urlencoded' }, body: 'credentials={"username":"[YOUR_USER]","password":"[YOUR_PASSWORD]"' };request(options, function (error, response, body) { if (error) throw new Error(error); const querystring = require('querystring'); console.log(querystring.parse(response.h

Note: this article only deals with the disk space of each format, not the performance comparison. When using Hive as your engine for SQL queries, you might want to consider using ORC or Parquet file formats for your data. There are numerous advantages to consider when choosing ORC or Parquet. Firstly, both will give you columnar compression of the data within, whereas a plain text file will have no compression at all. Secondly, indexes within ORC or Parquet will help with query speed as some basic statistics are stored inside the files, such as min,max value, number of rows etc. Both formats can also express complex data structures (such as hierarchies) which a plain CSV file cannot do. The combination of these can boost your query many times over. Have a look at the article here to read more about the underlying mechanics of both formats: https://www.datanami.com/2018/05/16/big-data-file-formats-demystified/ More information on the ORC file format: https://cwiki.apache.org/confluenc

The number can be found in the database which is used to store hive meta data.If MySql is the backend db, select count(*) from TBLS;

In Data Science, R is commonly used for analytics and data exploration. When moving to a Hadoop architecture and a connected data platform a big question is what happens to my already existing R scripts?You can transition nicely to Hadoop using the rHadoop package for R which allows you to read from hdfs and get data back into a dataframe in R. To enable this you first need to get the R package:install.packages("rhadoop")You can also wget the packagewget https://cran.r-project.org/src/contrib/Archive/rHadoopClient/rHadoopClient_0.2.tar.gzand then install.packages("/path/to/package")now you can read a file in using the rHadoopClient:rHadoopClient::read.hdfs("/path/to/data.csv")That's all you need to get started. This allows you to change your file read steps in your R scripts to point to HDFS and still run your R scripts as you are used to doing.

1. Introduction1.1 HDP 2.6.3 provides Apache Spark 2.2 with Apache ORC 1.4 Since Apache Spark 1.4.1, Spark supports ORC as one of its FileFormat. This article introduces how to use another faster ORC file format with Apache Spark 2.2 in HDP 2.6.3. First, in order to show how to choose a FileFormat, Section 1.2 will show an example which writes and reads with ORCFileFormat. Section 2 shows a brief performance comparison, and Section 3 explains more use cases and ORC configurations. Section 4 summarizes ORC-related Apache Spark fixes included in HDP 2.6.3.1.2 Usage Example: Write and Read with ORCFileFormat %spark2.spark// Save 5 rows into an ORC file.spark.range(5).write.format("orc").mode("overwrite").save("/tmp/orc")// Read a DataFrame from the ORC file with the existing ORCFileFormat.spark.read.format("orc").load("/tmp/orc").count// Read a DataFrame from the ORC file with a new ORCFileFormat.spark.read.format("org.apache.spark.sql.execution.datasources.orc").load("/tmp/orc").countres

Abstract With native SQL Commands and the publically available Hivemall components we are able to build an advanced analytics and machine learning process for a recommendation system.Setup the EnvironmentWe will setup this environment with the latest version of the Hortonworks Sandbox and the Hivemall Repo as shown below to prepare our infrastructure for eg some relatively complicated Machine Learning concepts with a relatively simple environment.Step 1- Download and install Hortonworks Sandbox HDP 3.0.1 Step 2 - Download the Hivemall artefacts Hivemall Repo and copy into your HDFS directory (so all nodes in the cluster have access to the same)login as user: hive with the su -hive commandwget https://search.maven.org/remotecontent?filepath=org/apache/hivemall/hivemall-all/0.5.2-incubating/hivemall-all-0.5.2-incubating.jar -O hivemall-all-0.5.2-incubating.jarhdfs dfs -mkdir /apps/hive/warehouse/libhdfs dfs -put hivemall-all-0.5.2-incubating.jar /apps/hive/warehouse/lib Step 3A - Dow

Disclaimer: This article is based on my personal experience and knowledge. Don't take it as a standard guidelines, understand the concept and modify it for your environmental best practices and use case.Its always suggest contact Hortonworks Support if you have trouble in your production cluster Problem Statement: My Cluster is upgraded from ambari-2.7.1 to ambari-2.7.3 and HDP-3.0.1 to HDP-3.1. Whenever I am trying to change some configs and save it. the Below Error is shown : Error message: </strong>Stack Advisor reported an error . Exit Code: 2. Error: KeyError: 'beeline_jdbc_url_default' StdOut file: /var/run/ambari-server/stack-recommendations/12/stackadvisor.outStdErr file: /var/run/ambari-server/stack-recommendations/12/stackadvisor.err and in Ambari-server.log I can see Caused by: org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException: Stack Advisor reported an error. Exit Code: 2. Error: KeyError: 'beeline_jdbc_url_default'StdOut file: /var/run/ambar

AbstractMachine learning and supervised learning at scale is a common task that can be addressed with Apache Hive by leveraging Hivemall features and user-defined functions (UDFs) for HiveQL, which is strongly optimized for machine learning (ML) and data science. For example, logistic regression models, one of the basic tenants of ML, can also be easily built with Hivemall. The following example should explain the basic usage of Hivemall with examples of supervised learning of a simple regressor.Setup the environment-> HDP 3.0.1 Sandbox - Hivemall 0.5.2 see Machine Learning with Apache Hive SQL using HivemallExplore the Training Data Database: foodmart - Table: sales_fact _dec_1998 and customer Create Table customer_training asselect s1.customer_id, c1.country, c1.gender, c1.member_card,c1.num_cars_owned, count(s1.product_id) as purchase_cntfrom sales_fact_dec_1998 s1 join customer c1WHERE s1.customer_id = c1.customer_idgroup by s1.customer_id, c1.country, c1.gender, c1.member_c

Kafka's mirroring feature makes it possible to maintain a replica of an existing Kafka cluster. This tool uses Kafka consumer to consume messages from the source cluster, and re-publishes those messages to the target cluster using an embedded Kafka producer. Running Mirror Maker To set up a mirror, run kafka.tools.MirrorMaker. The following table lists configuration options. At a minimum, MirrorMaker requires one or more consumer configuration files, a producer configuration file, and either a whitelist or a blacklist of topics. In the consumer and producer configuration files, point the consumer to the the source cluster, and point the producer to the destination (mirror) cluster, respectively.bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config sourceCluster1Consumer.config --consumer.config sourceCluster2Consumer.config --num.streams 2 --producer.config targetClusterProducer.config --whitelist=".*"Pa

Docker on YARN is relatively easy to set up on an existing cluster, but full clusters are not always available.My Ember Project was created to simplify dev/test for technologies managed by Ambari and Cloudera Manager. It provides utilities for creating dockerized clusters that use far fewer resources than a full bare metal or VM-based cluster. Additionally, by using pre-built images, the time it takes to get a cluster up and running can be reduced to less than 10 minutes.The following four commands are all that is necessary to download and run a ~5GB image that comes preinstalled with Ambari, Zookeeper, HDFS, and YARN with Docker on YARN pre-configured. Docker containers spawned by YARN will be created on the host machine as peers to the container with YARN inside. All containers are launched into the "ember" docker network by default. Once the container is downloaded, it takes less than 5 minutes to start all services.curl -L https://github.com/SamHjelmfelt/Ember/archive/v1.1.zip -o E