Resources

Ambari database cleanup - Speed upambari-server db-cleanup -d 2016-09-30 --cluster-name=TESTHDPI have ran this on a ambari database used by on one of the 500 node cluster HDP cluster. It ran for more than 15 hours but without any success. I analyzed the ambari server logs to check where it is taking most of the time, it seems to be spending more time on batch deletes on ambari.alert_notice, ambari.alert_current ,ambari.alert_history table.To improve the performance of the db cleanup, I have created the index on the ambari.alert_notice table..-bash-4.1$ psql -U ambari -d ambari Password for user ambari: psql (8.4.20)Type "help" for help.ambari=> CREATE INDEX alert_notice_idx ON ambari.alert_notice(history_id);After this i ran re-loaded my ambari database from the backup and ran the db-cleanup, it took only less than 2 min to complete the cleanup.Also to reclaim the disk space and reindex after the cleanup, i ran the following commands as super user "postgres"Vacuum full;reindex da

This article explains on how to build a spark application in eclipse that can run in local mode on a kerberized hadoop cluster.Environment:Eclipse : Scala IDERequisites:krb5.conf Copy /etc/krb5.conf from any one of the node of your cluster where keberos client is installed to local machinecore-site.xmlCopy core-site.xml from /usr/hdp/current/conf/core-site.xml to local machinehdfs-site.xmlCopy hdfs-site.xml from /usr/hdp/current/conf/core-site.xml to local machineyarn-site.xmlCopy yarn-site.xml from /usr/hdp/current/conf/core-site.xml to local machinekeytabCopy keytab file of the user from cluster to local machine. This keytab file should be of the corresponding user principal.Steps:1. Add namenode(s) hostname to host file on local machine(In case of HA, add both namenodes to host file).2. Add hostname of KDC server to host file on local machine.3. Right click on your project in Eclipse and add source folder, rename it to resources.4. Copy core-site.xml, hdfs-site.xml and yarn-site.xml

Copy, paste and run the following code: val data = Array(1, 2, 3, 4, 5) // create Array of Integersval dataRDD = sc.parallelize(data) // create an RDDval dataDF = dataRDD.toDF() // convert RDD to DataFramedataDF.write.parquet("data.parquet") // write to parquetval newDataDF = sqlContext. read.parquet("data.parquet") // read back parquet to DFnewDataDF.show() // show contents If you run this code in a Zeppelin notebook you will see the following output data: data: Array[Int] = Array(1, 2, 3, 4, 5)dataRDD: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:31dataDF: org.apache.spark.sql.DataFrame = [_1: int]newDataDF: org.apache.spark.sql.DataFrame = [_1: int]+---+| _1|+---+| 1|| 2|| 3|| 4|| 5|+---+

If you need to install the specific version of HDP using Blue prints use the Following steps. Install ambari-server and ambari-agent and register it.Use the appropriate os version. Here I have used the os version as centos/redhat 7 Make sure you do it on a clean host with no existing hadooppackages etc.Create 4 files under say /root [root@vijaybluetest1 ~]# lsblueprint.json cluster.json repo.json utils.jsonblueprint.json :- Contains cluster definition.1) cat blueprint.json{ "host_groups" : [ { "name":"host_group_1", "components":[ { "name" : "NODEMANAGER" }, { "name" : "DATANODE" }, { "name" : "ZOOKEEPER_SERVER" }, { "name" : "HISTORYSERVER" }, { "name" : "JOURNALNODE" }, { "name" : "APP_TIMELINE_SERVER" }, { "name" : "RESOURCEMANAGER" }, { "name" : "MAPREDUCE2_CLIENT" }, { "name" : "YARN_CLIENT" }, { "name" : "HDFS_CLIENT" }, { "name" : "ZOOKEEPER_CLIENT" }, { "name" : "NAMENODE" }, { "name" : "SECONDARY_NAMENODE"} ], "cardinality":"1" } ], "Blueprints" : {

Why Hortonworks Schema Registry?Both Apache NiFi and Hortonworks Streaming Analytics Manager (SAM) are extremely powerful tools in the data-in-motion space. To make them truly enterprise ready, Hortonworks DataFlow (HDF), which is a cutting edge data-in-motion platform, offers a set of enterprise foundational services. Apache Ambari is one example, which provides cluster management and provisioning capabilities; Apache Ranger is another example, which provides a centralized place for users to manage policies/permissions across HDF and HDP components. In HDF 3.0, Hortonworks Schema Registry (SR) became a GA offering as part of the foundational services.SR offers a centralized place to store and manage reusable schemas. This becomes extremely handy when you have data sharing the same schema either generated by different producers, or being delivered to different consumers. Think of data coming from different ATM machines, different trucks, different producers, or being delivered to diffe

For now, it is not possible to encrypt the data during its transmission from Sqoop.This is a known limitation, Jira SQOOP-917 is already in place for the feature.

sqoop-validate.pdf

With HDF 1.1.2.1 release, HDF supports accessing Kerberos enabled kafka topics. For a stanalone NiFi node, following instructions can be used.1. new jass file should be created with following entries.Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="./conf/nifi.keytab" useTicketCache=false principal="nifi@EXAMPLE.COM”; };KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=false renewTicket=true serviceName="kafka" useKeyTab=true keyTab="./conf/nifi.keytab" principal="nifi@EXAMPLE.COM";};*** Both the KafkaClient and Client configs should use the same principal and key tab ***2. bootstrap.conf file should be changed to include following line.java.arg.15=-Djava.security.auth.login.config=/<path>/zookeeper-jaas.confRestart NiFi node after changes to bootstrap.conf.3. GetKafka/PutKafka processors should be modified to include a new property security.protocol and value for this property should be set

Use Case :1 . ADDING a member to another group and being able to manage them internally without having to deal with outside or additional products.2 . To be able to easily determine what members reside in what groups instead of having to scroll down page after page to see what members are in what groups especially when you have hundreds of users to keep track of. 3 . To easily administer various groups without having the hassle of creating more and more Active Directory/LDAP associations and having to submit change control requests to other departments for something we should be able to administer on our own. Ranger User Sync Process supports reading users and group information from one of the following sources: Unix Text file - CSV or JSON formatLADP/ADCSV Format :If the filename does not end with .json, each line in the file will be treated as a delimiter separated fields of the following format. Default delimiter is a comma; this can be changed using configuration shown above.user-1

A region is decided to be split when store file size goes above hbase.hregion.max.filesize or according to defined region split policy.At this point this region is divided into two by region server.Region server creates two reference files for these two daughter regions.These reference files are stored in a new directory called splits under parent directory.Exactly at this point, parent region is marked as closed or offline so no client tries to read or write to it.Now region server creates two new directories in splits directory for these daughter regions.If steps till 6 are completed successfully, Region server moves both daughter region directories under table directory.The META table is now informed of the creation of two new regions, along with an update in the entry of parent region that it has now been split and is offline. (OFFLINE=true , SPLIT=true)The reference files are very small files containing only the key at which the split happened and also whether it represents top h

Apache Storm is a distributed real-time computation system for processing large volumes of high-velocity data. Debugging a distributed system is inherently hard since there are many moving parts spread across a large number of hosts in a cluster. Tracing failures to a particular component or a host in the system is hard and requires one to collect and analyze a lot of logs and debug/trace processes running in the distributed cluster. In the past the support for debugging topologies in an Apache storm cluster was very minimal. Users would have to first turn on debug logs, restart the workers, redeploy the topologies and then collect logs from different worker nodes and analyze them. If they wanted to take a jstack dump or profile the workers, they would have to login to the worker hosts and run the commands manually which was painful. If one wanted to inspect the tuples flowing through the topology pipeline, they would have to add special “debug” bolts to log the tuples to some log a

IntroductionFor NLP, mostly I want to do two things, Entity Recognition (people, facility, organizations, locations, products, events, art, language, groups, dates, time, percent, money, quantity, ordinal and cardinal)Sentiment AnalysisSo basically what is it and why don't people like it. These two features are very useful as part of a real-time streaming processing of social, email, logs and semistructured document data. I can use both of these in Twitter ingest via Apache NiFi or Apache Spark. Don't confuse text entity recognition with image recognition that we looked at with TensorFlow previously. You can certainly add that to your flow as well, but that is working with images and not text.My debate with sentiment analysis is do you give numbers, really general terms like Neutral, Negative or Positive. Or do you get more detailed like Stanford CoreNLP which has multiple of each.There are a lot of libraries available for NLP and Sentiment Analysis. The first two decisions

The PutSolrContentStream processor in Apache NiFi makes use of Solr's ContentStreamUpdateRequest which means it can stream arbitrary data to Solr. Typically this processor is used to insert JSON documents, but it can be used to stream any kind of data. The following tutorial shows how to use NiFi to stream data to Solr's Extracting Request Handler.Setup SolrDownload the latest version of Solr (6.0.0 for writing this) and extract the distributionStart Solr with the cloud example: ./bin/solr start -e cloud -nopromptVerify Solr is running by going to http://localhost:8983/solr in your browserSetup NiFiDownload the latest version of NiFi (0.6.1) and extract the distributionStart NiFi: ./bin/nifi.sh startVerify NiFi is running by going to http://localhost:8080/nifi in your browserCreate a directory under the NiFi home for listening for new files:cd nifi-0.6.1mkdir datamkdir data/inputCreate the NiFi FlowCreate a simple flow of GetFile -> PutSolrContentStream -> LogAttribute:The GetFil

Cloudbreak is a cloud agnostic Hadoop As A Service solution. It is easy to install, only one command in the command line, and an other one to start the service itself. During startup Cloudbreak Deployer tries to figure out the underlay infrastructure and sets required environment variables with competent default values, but some cases it isn't enough. In this article i collected all of the possible customizations, default values and some hint what does the variable configure in Profile file. ADDRESS_RESOLVING_TIMEOUT 120000 - DNS lookup timeout for internal service discoveryAWS_ACCESS_KEY_ID "" - Access key of the AWS accountAWS_ROLE_NAME cbreak-deployer - Name of the AWS role for the `cbd aws [generate-rol, show role]` commandsAWS_SECRET_ACCESS_KEY "" - Secret access key of the AWS accountAZURE_SUBSCRIPTION_ID - Azure subscription ID for interactive login in Web UIAZURE_TENANT_ID - Azure tenant ID for interactive login in Web UICAPTURE_CRON_EXPRESSION - SmartSense

The MiniFi flow executes two scripts: one to call TensorFlow Python that captures an OpenCV Raspberry Pi Camera and runs Inception on it. That message is formatted as JSON and sent on. The second script reads GPS values from a USB GSP sensor and outputs JSON. Get File reads the Pi Camera image produced by the ClassifyImages process. Cleanup logs is a standalone timed script that cleans up old logs on the Raspberry Pi.Using InferredAvroSchema I created a schema for the GPS unit and stored it in the Hortonworks Schema Registry.This is the provenance event for a typical GPS message sent. You can see what shell script we ran and from what host.In Apache NiFi we process the message, routing to the correct place, setting a schema and querying it for a latitude. Then we convert the AVRO record to ORC to save as a Hive table.MiniFi requires we change the NiFi created template to a configuration file via the command-line MiniFi Toolkit.minifi-toolkit-0.2.0/bin/config.sh transform

Technology: CSV, AVRO, Hortonworks Schema Registry, Apache NiFi, Streaming Analytics Manager, Kafka, Hadoop HDFSThis is a simple example of reading CSV data, using it's schema to convert it to AVRO, sending it via Kafka to SAM. SAM then reads it and stores it to HDFS. This is a simple flow, but a start to setting up any level of complex flow.A number of questions have come up on how to setup this basic flow.Gotchas: Must send AVRO, beware of nulls, set your schema, put your schema name somewhere, create your kafka topic and make sure you have permissions.Building Sample CSV Datacurl "http://api.mockaroo.com/api/ANID?count=1000&key=AKEY" > "simple.csv"****You need to sign up for mockaroo and create your own schema. You will then need to add the key and account id to the above link.Import this schema to your account<strong>{ "id": 76671, "num_rows": 1000, "file_format": "csv", "name": "simple", "include_header": true, "columns": [ { "name": "first",

haveibeenpwned has downloadable files that contains about 320 million password hashes that have been involved in known data breaches.This site has a search feature that allows you to check whether a password exists in the list of known breached passwords. From a security perspective, entering passwords into a public website is a very bad idea. Thankfully, the downloadable files make it possible to perform this analysis offline.Fast random access of a dataset that contains hundreds of millions of records is a great fit for HBase. Queries execute in a few milliseconds. In the example below, we'll load the data into HBase. We'll then use a few lines of Python to convert passwords into a SHA-1 hash and query HBase to see if they exist in the pwned list.On a cluster node, download the files:wget https://downloads.pwnedpasswords.com/passwords/pwned-passwords-1.0.txt.7zwget https://downloads.pwnedpasswords.com/passwords/pwned-passwords-update-1.txt.7zwget https://downloads.pwnedpasswords.com/

This past year Hadoop celebrated it 10th birthday. At Hadoop Summit we talked about how far Hadoop has come from its origins as a Google whitepaper to its initial implementation at Yahoo to where it is today, powering more than half of fortune 500 companies and being one, if not the most disruptive technologies of this century so far. In that time, we’ve seen platform improvements such as the elimination of the single point of failure in the Name Node, YARN opened up the platform for mixed workloads and the proliferation of new distributed computing frameworks beyond MapReduce and advances in hardware has increased speed and computing density of worker nodes.For those of you not in the know, when you write to HDFS, Hadoop’s distributed file system, your documents and data files are broken into blocks. For each of those blocks 2 additional copies are created and all the blocks are distributed across the data nodes in the cluster. There is a scheme to how the blocks are scattered but it’

PROBLEM:After upgrade from IOP (v 4.2.5) to HDP (v 2.6.x) in a kerberized setup, HSI start fails with following in HSI start log :WARN impl.LlapZookeeperRegistryImpl: The cluster is not started yet (InvalidACL); will retryERROR impl.LlapZookeeperRegistryImpl: Unable to start curator PathChildrenCacheorg.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /llap-sasl/user-hive at org.apache.zookeeper.KeeperException.create(KeeperException.java:121) ~[zookeeper-3.4.6.2.6.2.0-173.jar:3.4.6-173--1] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.2.6.2.0-173.jar:3.4.6-173--1] at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) ~[zookeeper-3.4.6.2.6.2.0-173.jar:3.4.6-173--1] at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:232) ~[curator-client-2.7.1.jar:?] at org.apache.curator.utils.EnsurePath$InitialHelper$1.call(EnsurePath.java:148) ~[curator-client-2.7.1.jar:?] at org.apache.curator.RetryLoop.cal

PROBLEM:After upgrade from IOP (v 4.2.0) to HDP (v 2.6.x) in a kerberized setup, HSI install fails with Configuration parameter 'hive.llap.daemon.service.principal' missing in dictionary.Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 616, in <module> HiveServerInteractive().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 75, in install import params File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/params.py", line 27, in <module> from params_linux import * File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py", line 697, in <module> hive_llap_principal = (config['configuratio

PROBLEM:After upgrade from IOP (v 4.2.0) to HDP (v 2.6.x) in a kerberized setup, HSI install fails with Configuration parameter 'hive.llap.daemon.service.principal' missing in dictionary.Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 616, in <module> HiveServerInteractive().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 75, in install import params File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/params.py", line 27, in <module> from params_linux import * File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py", line 697, in <module> hive_llap_principal = (config['configuratio

In case of a Blueprint based deployment Ambari stores deployment tasks in topology tables: topology_request, topology_logical_request, topology_host_request, topology_host_task, topology_logical_task.In case of some errors during deployment task creation these tables could be left in an inconsistent state, which has been fixed in 2.5.0 (AMBARI-19929). Ambari 2.5.0 introduced (removed in 2.5.2) two DB consistency checks related to topology tables:1. Check topology_request, topology_logical_request tables are consistent, by comparing the row counts returned:select count(tpr.id) from topology_request tpr;select count(DISTINCT tpr.id) from topology_request tpr join topology_logical_request tlr on tpr.id = tlr.request_id; In case row counts are different it means there’s a missing row in topology_logical_request table. To fix this a new row has to be inserted in topology_logical_request table: get next available: next_topology_logical_request_id = select max(id) + 1 from topology_l

Various open-source tools and applications are available to do performance, scalability and reliability testing for various popular HDP components. Here is a list of some of the popular ones:HDFSTestDFSIOMeasure the I/O performance of HDFS in your cluster. Source code for the tool can be found here.NameNode BenchmarkApplies load on the namenode by performing continuous read, write, rename and delete operations on small files. Source code for this tool can be found here.Synthetic Load GeneratorThe synthetic load generator (SLG) is a tool for testing NameNode behavior under different client loads. The user can generate different mixes of read, write, and list requests by specifying the probabilities of read and write. The user controls the intensity of the load by adjusting parameters for the number of worker threads and the delay between operations. More information on the tool can be found here.YARN/MRTeraSortMeasure performance by measuring time to sort 1TB of data. The test runs in t

Cloudbreak is now available directly from the Azure Marketplace. The goal of this article is to help you get started: quickly launch Cloudbreak from the Azure Marketplace, log in to the Cloudbreak UI and create a Cloudbreak credential using the interactive method. Let's get started!UpdateThis article was written in 2017. As of today, Azure Marketplace no longer includes the latest version of Cloudbreak. Fo the latest version use this quickstart.OverviewWe will be performing the following steps:Meet the Prerequisites.Launch Cloudbreak.Log in to the Cloudbreak UI.Create a Cloudbreak credential using the interactive method. Meet the Prerequisites 1.In order to launch Cloubdreak on the Azure Marketplace, you must have a Microsoft Azure account. If you don't have an account, you can set it up at https://azure.microsoft.com.2.In order to provision clusters on Azure, Cloudbreak must be able to assume a sufficient Azure role ("Owner" or "Contributor") via Cloudbreak credential:Your account mus

Sometimes you want to trigger events with a click on a special touchpad or device mounted somewhere. This could be in a factory, on a door or at your desk.For me it's a small device on my desk that I can use to trigger events. I have this running every 15 seconds looking for gestures. I could put it in an infinite loop, but Python and RPI could leak some memory. We are very constrained, so I am trying to keep it a little more minimal. I keep my batch duration to 15 seconds and my run schedule to 15 seconds.Let's Build A Simple NiFi Flow to Receive The JSON Data and React To ItIn my RouteOnContent, I just look for the word "center". I have thought of many options for running sql, doing a backup, etc.. Build The MiniFi Flow in NiFiDownloaded minifi 0.2.0 and minifi 0.2.0 toolkit (you can use newer, but make sure you install the same version on the device you are going to move your config.yml to).minifi-toolkit-0.2.0/bin/config.sh transform minififlick.xml config.yml Then SCP t