Community Articles

Topic: IoT Edge Processing with Apache NiFi and MiniFi and Multiple Deep Learning LibrariesPart 1: Multiple Devices with DataKeywords: Deep Learning On The Edge, GPS Ingestion, Sense-Hat and Rainbow Hat Sensor Ingest, WebCam Image IngestIn preparation for my talk at Strata in NYC, I am updating my IoT demos for more devices, more data types and more actions.I have three streams coming from each device including web camera images. When we are sending data from a MiniFi agent we need to define a port on an Apache NiFi server/cluster to receive it.So I design my MiniFi flow in the Apache NiFi UI (pretty soon there will be a special designer for this). You then highlight everything there and hit Create Template. You can then export it and convert to config.yml. Again, this process will be automated and connected with the NiFi Registry very shortly to make this less clicks.This is an example. When you connect to it in your flow you design in Apache NiFi UI you will connect to

Ingesting Blockchain data from btc.com and blockchain.com(.info).Like everything in Apache NiFi, this is trivial easy to ingest all these different feeds, process them, route them and store them for SQL access.API:blockchain.comhttps://blockchain.info/latestblockhttps://api.blockchain.info/charts/transactions-per-second?timespan=5weeks&rollingAverage=8hours&format=jsonhttps://blockchain.info/blocks/BTC.com?format=jsonhttps://api.blockchain.info/pools?timespan=5dayshttps://api.blockchain.info/statshttps://blockchain.info/tickerhttps://blockchain.info/tobtc?currency=USD&value=10000Blocks For Today

A quick example NiFi flow to demonstrate sending logs to syslog from NiFi.Steps:1. Wait for new compressed files (gz) to land in folder (/tmp/nifi/syslog/src/*.gz) with log lines.2. Decompress the file3. Split the file into individual lines4. Send data to syslogNiFi Template : Download the following NiFi template and import the flow into your NiFi instance. Instructions to import a NiFi template is available here - Importing NiFi templatessend-to-syslog.xmlSend to Syslog Template Walkthrough :The Flow is broken down into 3 sectionsSection 1 : Test Data GenerationSection 2 : Processing (decompress files and send to local syslog)Key HighlightsExtractText Processor :We are using the extract text processor to get the content of the flowfile into the flowfile attribute "msg". We are grabbing the entire content of the flowfile into "msg" by using the regular expression (.*). You may get only specific fields by providing appropriate regular expression here.PutSyslog processor :We are using t

This article mainly focuses mainly on what all supported options available in User sync and how they can be used to target specific use cases. For some common use cases examples and how ranger can be configured in those cases, please refer to Configuring Ranger UserSync with AD/LDAPUser based search without enable group sync: This configuration can be used in cases where customers want to sync users based on a OU or a user search filter and get all the groups the users belong to.This case is supported only when the user object has the group membership attributes like “memberof” or “ismemberof”. This option is available only in Full sync case (i.e., when Incremental sync is disabled in Ranger User sync configuration)User based search with enable group sync: This configuration can be used in cases where - customers want to sync users based on a OU or a user search filter and want to limit the groups that these user belong to based on the OU or a group search filter.Group membership attri

Short Description:In this tutorial we will install the browser connected version of RServer on to the HDP Sandbox.ArticleInstalling RStudio on HDP Sandbox IntroductionRStudio is an Integrated Development Environment (IDE) for the R language which includes a direct code execution console, as well as tools for plotting and debugging, you can find more information about the RStudio features here.RStudio is used as the primary tool in the Predicting Airline Delays using SparkR Hortonworks tutorial in which you will learn to train and analyze Machine Learning models to predict Airline delays.PrerequisitesDownload the latest HDP SandboxComplete the Learning the Ropes of the Hortonworks Sandbox tutorial1. SSH on to the SandboxUse the following command to SSH on to the Sandbox as root user:ssh root@sandbox-hdp.hortonworks.com -p 2222NOTE: If this is your first time signing on the default password is hadoop.2. Begin installationOn CentOS 7, the base OS for the Sandbox, R is available through th

1. Installed expect first:# yum install expect -y (or use your Linux distribution's package manager if you're not Using CentOS or RHEL)2. Then create and run the following expect script: # cat /tmp/ambari-server-sync-ldap-unattended.sh#!/usr/bin/expect set timeout 20spawn /usr/sbin/ambari-server sync-ldap --groups=/etc/ambari-server/ambari-groups.csvexpect "Enter Ambari Admin login:" { send "admin\n" }expect "Enter Ambari Admin password:" { send "notTheRealPasswordOfCourse\n" }interact3. If customer wants password to NOT be in plain text, ask them to look at something like Ansible which handles decrypting passwords from a file.

Tokens are wire-serializable objects issued by Hadoop services, which grant access to services. Some services issue tokens to callers which are then used by those callers to directly interact with other services without involving the KDC at all. Block Tokens A BlockToken is the token issued for access to a block; it includes:(userId, (BlockPoolId, BlockId), keyId, expiryDate, access-modes)Block Keys Key used for generating and verifying block tokens. Block Keys are managed in the BlockTokenSecretManager, one in the NN and another in every DN to track the block keys to which it has access.How this Works:1. Client asks NN for access to a path, identifying via Kerberos or delegation token. 2. Client talks to DNs with the block, using the Block Token. 3. DN authenticates Block Token using shared-secret with NameNode. 4. if authenticated, DN compares permissions in Block Token with operation requested, then grants or rejects the request. The client does not have its identity checked b

Roaring Elephant Podcast – Episode 92 – Roaring newshttps://roaringelephant.org/2018/06/12/episode-92-roaring-news/Another week, another edition of Roaring Big Data News. This time, Dave talks about driving teens and Jhon takes a detailed look at an Eventbrite data pipeline article. Breaking NewsPlay in new window | Download (Duration: 46:08 — 31.9MB)Dave Driver monitoring isn’t just for teens; adults can benefit, too https://arstechnica.com/cars/2018/05/buicks-smart-driver-explains-why-my-gas-mileage-sucks-and-my-editors-doesnt/ Jhon Looking under the hood of the Eventbrite data pipeline! https://www.eventbrite.com/engineering/looking-under-the-hood-of-the-eventbrite-data-pipeline/ Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

A distributed compute engine for pricing financial derivatives using QuantLib with Spark running in Docker containers on YARN with HDP 3.0.   Introduction Modern financial trading and risk platforms employ compute engines for pricing and risk analytics across different asset classes (equities, fixed income, FX etc.) to drive real-time trading decisions and quantitative risk management. Pricing financial instruments involves a range of algorithms from simple cashflow discounting to more analytical methods using stochastic processes such as Black-Scholes for options pricing to more computationally intensive numerical methods such as finite differences, Monte Carlo and Quasi Monte Carlo techniques depending on the instrument being priced – bonds, stocks or their derivatives – options, swaps etc. and the pricing (NPV, Rates etc) and risk (DV01, PV01, higher order greeks such as gamma, vega etc.) metrics being calculated. Quantitative finance libraries, typically written in low level p

Context Hortonworks tutorials are a great resource and one I use often to explore new topics or re-enforce my current knowledge. Recently, I went through the Predicting Airline Delays using SparkR tutorial. This tutorial is using SparkR and RStudio, running on the HDP Sandbox. For those, like me, who want to run rstudio on an HDP node running on something else than the sandbox, the configuration of rstudio differs a little from the article linked in this tutorial, which is why I thought of writing this quick article. This tutorial will thus detail the installation and unit testing of RStudio Server for the following environment versions: Hortonworks Data Platform: HDP 2.6 OS: Centos 7 64bit Cloud Infrastucture: OpenStackInstalling rstudio-server on Centos 71. Connect to the server onto which you want to install rstudio-server:$ ssh [your_user]@[your_server]Note: the user with which you connect must have sudo priviledges.2. Install R:$ sudo yum install R -y3. Once connected, fo

The ProblemTraditional 'distcp' from one directory to another or from cluster to cluster is quite useful in moving massive amounts of data, once. But what happens when you need to "update" a target directory or cluster with only the changes made since the last 'distcp' had run. That becomes a very tricky scenario. 'distcp' offers an '-update' flag, which is suppose to move only the files that have changed. In this case 'distcp' will pull a list of files and directories from the source and targets, compare them and then build a migration plan.The problem: It's an expensive and time-consuming task. Furthermore, the process is not atomic. First, the cost of gathering a list of files and directories, along with their metadata is expensive when you're considering sources with millions of file and directory objects. And this cost is incurred on both the source and target namenode's, resulting in quite a bit of pressure on those systems.It's up to 'distcp' to reconcile the difference between

ProblemI have a problem with accessing spark history data for killed streaming jobs. The problem looks like the one mentioned here:When I click on History under Tracking UI for the FINISHED Spark Job I am redirected to: http://dan3.dklocal:18080/history/application_1491987409680_0013/jobs/ - Spark History URL However, when doing the same for the KILLED one I am redirected to: http://dan3.dklocal:8088/cluster/app/application_1491987409680_0012 - application log on RM UI. And the need is to be redirected for the KILLED jobs to Spark History URL.SolutionKilling the spark job using 'yarn application -kill' is the not the right way of doing it.When doing: 'yarn application -kill' - the job got killed (status in RM UI is State: KILLED, FinalStatus: KILLED), the '.inprogress' suffix got removed. But the History does not go to Spark History but RM log. When having done: kill -SIGTERM <Spark Driver>, the job also get killed. In RM UI is State: FINISHED, FinalStatus: SUCCEEDED, t

HiveServer2 is a thrift server which is a thin Service layer to interact with the HDP cluster in a seamless fashion. It supports both JDBC and ODBC driver to provide a SQL layer to query the data. An incoming SQL query is converted to either TEZ or MR job, the results are fetched and send back to client. No heavy lifting work is done inside the HS2. It just acts as a place to have the TEZ/MR driver, scan metadata infor and apply ranger policy for authorization. HiveServer 2 maintains two type pools 1. Connection pool : Any incoming connection is handled by the HiveServer2-handler thread and is kept in this pool which is unlimited in nature but restricted by number of threads available to service an incoming request . An incoming request will not be accepted if there are no free HiveServer2-handler thread to service the request. The total no of threads that can be spawnned with the HS2 is controlled by parameter hive.server2.thrift.max.worker.thread.2. Session pool (per queue) : t

Key Take away :1. For a hiveServer2 client the connection time seen is the total time to interact with AD(TGT + Zookeper Service ticket + HiveServer2 Service Ticket) + Zookeeper + HiveServer2 (mysql + YARN allocation).2. In case your AD is slow, the hive connection will take long time. 3. time beeline -u "zookeeper String" -e "select 1" can be used to find how much time the beeline is taking.4. In general it takes 4 to 10 seconds for connection to establish. 5. Neither AD, zookeeper or HiveServer2 can ever deny a connection, the connection time can be more but it can never be denied ideally.6. Clients can only have a timeout (configurable parameter in most of clients like HUE, SAS, Alation, health check scripts ), as neither zookeeper of HS2 can ever deny a connection. 7. HiveServer2 will try to allocate resources in YARN before acking that it has accepted connection. In case your queue is full The connection time will be impacted. 8. Kindly set hive.server2.tez.initialize.defaul

Scanning Documents into Data Lakes via Tesseract, Python, OpenCV and Apache NiFiSource: https://github.com/tspannhw/nifi-tesseract-pythonThere are many awesome open source tools available to integrate with your Big Data Streaming flows.Take a look at these articles for installation and why the new version of Tesseract is different.I am officially recommending Python 3.6 or newer. Please don't use Python 2.7 if you don't have to. Friends don't let friends use old Python.Tesseract 4 with Deep Learning https://www.learnopencv.com/deep-learning-based-text-recognition-ocr-using-tesseract-and-opencv/ Github: https://github.com/spmallick/learnopencv/tree/master/OCRFor installation on a Mac Laptop:brew install tesseract --HEADpip3.6 install pytesseractbrew install leptonicaNote: if you have tesseract already, you may need to uninstall and unlink it first with brew. If you don't use brew, you can install another way.Summary Execute the run.sh (https://github.com/tspannhw/nifi-tessera

A common question when planning a new implementation or an upgrade is “what’s supported?” We’ve traditionally handled this with documentation, but as we release more products and support more platforms, it’s been tough for our customers to understand product interoperability. To solve this problem, we’ve created a new interactive web application that Hortonworks Support users can use to quickly answer questions like the following:Which version of Ambari works with HDF 3.1.2?Which Hortonworks products are certified with RHEL 7.4?Which databases are supported with both HDP 2.6.4 and HDF 3.1.2?Usually, these questions would be tough to answer as multiple documentation sections would have to be referenced and compared, but with the Hortonworks Support Matrix, answers are only a few clicks away.Just navigate to https://supportmatrix.hortonworks.com and log in using your Hortonworks Support Portal credentials; then click one or more products to see what we support. It’s important to note t

SummaryUsing Apache Solr as the indexing and search engine for Metron requires the Metron REST service to perform queries to multple collections. If the Ranger plugin is active there is currently a gotcha ( = Ranger Solr plugin bug). If you don't want to give the Metron user full access to all Solr collections here is a workaround.The Problem2+ Solr collections that are being queried: metaalert, cef,.... (and other parser collections):1 user: metron1 Ranger policy: user: "metron", access type: "Read", "Write", collections: "metaalert", "cef"Query of metaalert collection returns content of metaalert collection as expected and logs event successfully in Ranger audit.curl -k --negotiate -u : "http://solr_url:solr_port/solr/metaalert/search?q=*"Query of cef collection returns content of cef collection as expected and logs it successfully in Ranger audit.curl -k --negotiate -u : "http://solr_url:solr_port/solr/cef/search?q=*"Query of metaalert and cef will return a "403 Unauthorized request

You can follow the below steps to upgrade your Spectrum Scale transparency.1. Download the latest HDFS Transparency connector (gpfs.hdfs-protocol.rpm)2. Delete the old rpm from your local repository. There should be only one hdfs-protocol rpm in the RPMs directory.3. Place the latest rpm downloaded in Step 2 in the directory.4. Refresh the local repository to pick up the latest rpm.cd <local rpm repository dir>createrepo . 5. Stop all services from ambari Ambari -> Actions -> Stop All6. Upgrade transparencyAmbari -> Spectrum scale -> Service Actions -> Upgrade Transparency7. Restart ambari serverambari-server restart8. Start all services from ambariAmbari -> Actions -> Start AllYou can check SpectrumScale installation here.Uninstallation hereDownload Transparency connector from hereHope this helps 🙂

Error:Cloudbreak cluster fails with "Infrastructure creation failed. Reason: Failed to retrieve the server's certificateSolution:The most common reason for this error is that your security group configuration is too restrictive. Notably, CB needs access to port 9443 from outside the local network.Refer to https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.7.0/content/security/index.html#default-cluster-security-groups for information on ports that must be open on cluster security groups.

Error:Cloudbreak cluster fails with "Infrastructure creation failed. Reason: Failed to retrieve the server's certificateSolution:The most common reason for this error is that your security group configuration is too restrictive. Notably, CB needs access to port 9443 from outside the local network. Refer to https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.7.0/content/security/index.html#default-cluster-security-groups for information on ports that must be open on cluster security groups.

Ambari Server version: 2.5.2.0-298HDP version: 2.6.4.0-91Ansible version: 2.4OS version: centos7There are options of deployment tools to do automatic HDP installation, one of the reasons for why ansible is popular is ansible is lightweight and it totally relies on ssh connection without any physical agent. But ansible script has its organizational structure that we will not talk here in detail, and for people who are not familiar with its grammar and its running logic, can reference to this quick tutorialhttps://serversforhackers.com/c/an-ansible2-tutorial but when you delve into full deck of scripts for hdp installation, you'd better to have the document along withhttps://docs.ansible.com/ansible/2.4/porting_guide_2.4.htmlAnd beware of composing the full ansible scripts from scratch is a little bit boring, to save time you can build your own on some other existing scripts, for hdp installation we can start with the following link, but to fit for your case, still, need to do tailoringh

Capture Images from PicSum.com Free ImagesProcess All the Images via TensorFlow Processor, SSD Predict via MMS and SqueezeNet v1.1 via MMSApache Zeppelin SQL Against tblsqueeze11Example Output from Squeeze v1.1Storing Generic Data in HDFS via SchemaExample SSD Data JSONHigh Level Flow From ServerApache NiFi Server Flows to StoreConvert to Apache ORCExtract AttributesConvert JSON Arrays to OtherExample Data Derived From TensorFlow ProcessorSchemas in Schema RegistryCreate Table in ZeppelinQuery Table in ZeppelinPython Librariesgit clone https://github.com/awslabs/mxnet-model-server.gitpip install opencv-python -Upip install scikit-learn -Upip install easydict -Upip install scikit-image -Upip install numpy -Upip install mxnet -Upip3.6 install opencv-python -Upip3.6 install scikit-learn -Upip3.6 install easydict -Upip3.6 install scikit-image -Upip3.6 install numpy -Upip3.6 install mxnet -UExample Runs - Squeeze v1.1mxnet-model-server --models squeezenet=squeezenet_v1.1.model --service mm

Error: Failed to verify the credential: User: arn:aws:iam::<roleid>:user/assume-only-user is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::<roleid>:role/<rolename>Reason:The Cloubdreak instance is not authorized to assume the role that you are trying to register for the Cloudbreak credential. How to fix:If you are using hosted Cloudbreak, you may be entering an incorrect AWS account ID when creating the role for Cloudbreak credential. Contact the Cloudbreak team to confirm the AWS account ID.

I have tried copying data between non-kerberized cluster to kerberized cluster using Snapshot diff, but it was failing with following error in HDP 2.5.3Currently distcp only supports hdfs:// RPC protocol for snapshot based diff copy. If you use webhdfs either in source or target you will encouter this error. Command:$hadoop distcp -diff s1 s2 -update webhdfs://nonsecure_cluster:50070/source  hdfs://secure_cluster:8020/targetError: java.lang.IllegalArgumentException: The FileSystems needs to be DistributedFileSystem for using snapshot-diff-based distcpat org.apache.hadoop.tools.DistCpSync.preSyncCheck(DistCpSync.java:86)at org.apache.hadoop.tools.DistCpSync.sync(DistCpSync.java:124)at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:180)at org.apache.hadoop.tools.DistCp.execute(DistCp.java:155)at org.apache.hadoop.tools.DistCp.run(DistCp.java:128)at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)at org.apache.hadoop.tools.DistCp.main(DistCp.java:462)

We can configure the Zeppelin UI for access over SSL. Step 1 : Create a keystoreFor Self-Signed Certificate:   ### Generate a keystore# keytool -genkey -alias zeppelin -keyalg RSA -dname "CN=$HOSTNAME,OU=IT,O=HWX,L=Bangalore,S=KA,C=IN" -keystore /etc/zeppelin/conf/zeppelin-keystore.jks -keysize 2048 -validity 365 -keypass hadoop -storepass hadoop ### Export self-signed certificate to a pem file# keytool -exportcert -keystore /etc/zeppelin/conf/zeppelin-keystore.jks -alias zeppelin -file zeppelin.pem -rfc   For CA-Signed Certificate:  ### Get certificate from you CA and create a PKCS12 keystore# openssl pkcs12 -export -inkey zeppelin.key -in zeppelin.pem -certfile /path/to/ca/certificate/ca.pem -out zeppelin.pfx### Convert pkcs12 keystore to jks keystore# keytool -v -importkeystore -srckeystore zeppelin.pfx -srcstoretype PKCS12 -destkeystore /etc/zeppelin/conf/zeppelin-keystore.jks -deststoretype JKS -srcalias 1 -destalias $(hostname)### Val