Community Articles

https://roaringelephant.org/2018/04/24/episode-85-dataworks-summit-community-showcase-exhibitor-soundbites/This is the final part of our coverage of the DataWorks Summit Berlin 2018. Normally we would not have had an episode this week, since we were in Berlin last week, but we had lightning interviews with the vendors in the Community Expo Are and used that coverage to make this episode. Audio Player 00:00 00:00 Play in new window | Download (Duration: 30:34 — 21.0MB)So less of “Dave & Jhon” and more “ecosystem tech” snippets this time. Even though this does stray a bit from our usual content, we still hope it is useful.This was recorded in a hotel room and on the expo floor so the audio quality is not up to our usual standards, we hope you’ll forgive us!Here is a timestamped list of the lightning interviews:02:41 Hortonworks https://hortonworks.com/06:28 Alation https://alation.com/08:45 Arcadia Data https://www.arcadiadata.com/11:12 Attunity https://www.attunity.com/

https://roaringelephant.org/2018/04/19/episode-84-dataworks-summit-berlin-day-2-recap/And with the end of day two of the 2018 DataWorks Summit in Berlin comes the end of this years Europe Summit. But never fear, we have an extra 90 minutes of DataWorks goodness for you to consume on your way home. Audio Player 00:00 00:00 Play in new window | Download (Duration: 1:30:26 — 62.3MB)No real editing on this one, recording in a hotel room so audio quality may not be up to our usual standards, we hope you’ll forgive us!Enjoy! Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

https://roaringelephant.org/2018/04/18/episode-83-dataworks-summit-berlin-day-1-recap/Another year, another European Dataworks Summit, and yes, another daily recap show from Jhon and Dave. We walk through the keynotes and sessions we attended and give our thoughts and views. This should be useful for anyone who wasn’t able to attend or those seeking to peek into sessions they couldn’t make. Audio Player 00:00 00:00 Play in new window | Download (Duration: 1:23:45 — 57.8MB)No real editing on this one, recording in a hotel room so audio quality may not be up to our usual standards, we hope you’ll forgive us!Enjoy! Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

It's coming up more often now, the need to "add" host to an existing cluster without using the Ambari UI. I once thought the Blueprints were only useful to "initially" provision a cluster. But they're also quite helpful to extend your cluster as well.Provision the Initial Cluster using Auto-DiscoveryIn the following example, we'll use a new feature in Ambari 2.1, called "Auto-Discovery" to provision the first node in our cluster.Do NOT manually register the Host with Ambari Server yet. We're going to demonstrate "Auto-Discovery" to initialize new hosts.Check that NO hosts have been registered with the cluster.curl -i -H "X-Requested-By: ambari" -u admin:xxxx -X GET http://<ambari-server:port>/api/v1/hostsBlueprint has be registered with Ambari { "Blueprints": { "stack_name": "HDP", "stack_version": "2.3" }, "configurations": [], "host_groups": [ { "cardinality": "1", "components": [ { "nam

Continuing with the theme of one-click installs from the last post, this time we will see how one can provision an HDP 2.6.4 cluster equipped with features for data governance and security through Apache Atlas and Apache Ranger using Cloudbreak in private/public cloud, again of course, with a single command!To begin with, you’d need: Access to a supported cloud provider (AWS, Azure, GCP, OpenStack) with following resources configured / identified for the cluster deployment:Region / Availability zoneVirtual Private Network / Security GroupVM types / sizes identified for master and worker rolesSSH keypairAccess to a Hortonworks Cloudbreak instance setup with credentials for the cloud provider(s) of choice [eg: OpenStack, AWS]Cloudbreak CLI installed on your machine, in path and configured pointing to the Cloudbreak instance aboveYou can use this tool to setup Cloudbreak with CLI configured locally on your workstation with just a couple of commands.Once this is done, you should have your

Disclaimer!Yes, despite some great articles around this topic by the finest folks here at Hortonworks, this is YALC [Yet Another article on spinning up Local HDP Clusters].But before you switch off, this article will show how to spin up HDP/HDF HA, secure clusters, even using customizable OS images, on your local workstation by just executing a single command!!…Glad this caught your attention and you’re still around!Quick Peek!For starters, just to get a taste, once you have ensured you are on a workstation that has at least 4 cores and 8 GB RAM and has open internet connectivity and once you have the environment setup with the pre-requisites,1. Execute the following in a terminalgit clone https://github.com/amolthacker/hwx-local-cluster && cd hwx-local-cluster && ./scripts/create-n-provision.sh2. Grab a coffee as this could take some time (unfortunately the script does not make coffee for you … yet)3. Once the script’s completed, you should have a basic HDP 2.6.4 clus

Issue: When you upgrade HDF to 3.0.2, you might experience NiFi service unable to start via Ambari due to the OOM exception in the below stack traceTraceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 309, in <module> Master().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 177, in start self.configure(env, is_starting = True) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 120, in locking_configure original_configure(obj, *args, **kw) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 150, in configure params.nifi_flow_config_dir, params.nifi_sensitive_props_key, is_starting) File "/var/lib/ambari-agent/cache/common-services/NIFI

This is for people preparing to attend my talk on Deep Learning at DataWorks Summit Berling 2018 (https://dataworkssummit.com/berlin-2018/#agenda) on Thursday April 19, 2018 at 11:50AM Berlin time.This is for running Apache MXNet on an HDF 3.1 node with Centos 7.Let's get this installed!git clone https://github.com/apache/incubator-mxnet.gitThe installation instructions at Apache MXNet's website (http://mxnet.incubator.apache.org/install/index.html) are amazing. Pick your platform and your style. I am doing this the simplest way on Linux path.We need to install OpenCV to handle images in Python. So we install that and all the build tools that OpenCV requires to build it and Apache MXNet.HDF 3.1 / Centos 7 Setupsudo yum groupinstall 'Development Tools' -ysudo yum install cmake git pkgconfig -ysudo yum install libpng-devel libjpeg-turbo-devel jasper-devel openexr-devel libtiff-devel libwebp-devel -ysudo yum install libdc1394-devel libv4l-devel gstreamer-plugins-base-devel -ysudo yum i

In order to debug pairing DLM, you'll need the following pre-req:1) Root access to the DPS VMProblem statement - have you received an error when pairing a cluster? Follow these step-by-step instructions to access the DLM log, to gain granular log information that will help you debug:1) Run the "sudo docker ps" command to gain the container id for "dlm-app":In the image above, the container id for "dlm-app" is "83d879e9a45e". 2) Once you receive the container id, you can run the following command "sudo docker exec -it 83d879e9a45e /bin/tailf /usr/dlm-app/logs/application.log"This will give you insight into the DPS-DLM application, in the example above you'll see "ERROR". The error log will post once you click "pair" in the DLM UI. Using the information from the log, you'll be able to troubleshoot your issue.

Ethereum: Accessing Feeds from Etherscan on Volume, Price and Transaction (Apache NiFi) - Cryptocurrency Part 2As opposed to EtherDelta that we accessed in our first post (https://community.hortonworks.com/content/kbentry/191146/accessing-feeds-from-etherdelta-on-trades-funds-bu.html) Etherscan has a familiar SSL HTTP REST call returning clean JSON. We are reading three REST APIs for acquiring three sets of interesting Ethereum data including logs, current supply and current price.Overview of FlowREST API Ingest feed for getLogsWe split into individual JSON records for processing.Ether Supply and Ether Last Price FlowsFor these two flows they return very minimal amounts of data with some flags I don't care about (status, message). So I use EvaluateJsonPath to extract just the good fields. Then with UpdateAttribute I add a just created time stamp (now()). Finally I use AttributesToJSON to replace the existing flow file with a new JSON file built from the fields I want only. Y

“In meetings philosophy might work,
[but in] the field practicality works.” –Amit KalantriShort Description:A quick tutorial on how to query cryptocurrency transactions with Spark StreamingAbstract:This tutorial covers how once can use Nifi to stream public cryptocurrency transactional data to Kafka and consequently query the transactional stream with Spark Streaming. We will be using the Pyspark API for demonstration purposes. It is strongly recommended that one use Java or Scala for production grade applications; as most of the generally available (GA) features in Spark are published to support those languages foremost. Figures are used as a user graphical aid. Steps are highlighted by bullet points.What is Spark? Why is it well suited for Streaming at scale?Apache Spark is a distributed processing engine for data science and data engineering at scale. It features several out of the box libraries for machine learning, map reduce, graph analysis, micro batching, and structured query l

Within VirtualBox, navigate to Settings > Shared Folders. Choose a directory path on the host machine, give the directory a name and choose applicable options (such as Auto Mount). For this example, I used the name 'share'. Note this doesn't require restarting your VM. Now establish an SSH session to your guest. Run the following command (see this article) to mount the shared directory to the local directory ~/host on the guest. You can now access and transfer files from guest to host and vice versa. sudo mount -t vboxsf share ~/host

Note: First made your topology file. Please find an attached example. knox-topology-file.xmlknox-ad-ldap-upgraded-docus.pdfAbove PDF file covered all practical concepts and some theory part.Step 1:- Install Knox on edge node or any node on the cluster.Step 2:- Start Knox service from Ambari,make sure your Ambari Server is already sync with LDAP.Step3:- Search your LDAP Server via below commandldapsearch -W -H ldap://ad2012.ansari.net -D binduser@ansari.net -b "dc=ansari,dc=net"ldapsearch -W -H ldaps://ad2012.ansari.net -D binduser@ansari.net -b "dc=ansari,dc=net"Step 4:- Create a master password for Knox:/usr/hdp/current/knox-server/data/security/keystores/gateway.jks/usr/hdp/2.6.4.0-91/knox/bin/knoxcli.sh create-master --forceenter password then verify it Note:-(2.6.4.0-91 is my HDP versions select your hdp version /usr/hdp/XXXXXXX/)Step 5: Validate your topology file (your cluster name and toplogy file name should be same):-/usr/hdp/2.6.0.3-8/knox/bin/knoxcli.sh validate-topology --c

ProblemInterpreters do not work through zeppelin with no internet access.Checking out /usr/hdp/current/zeppelin-server/local-repo - the folder is empty or contains “org” folder with no jars Running hive through Zeppelin returns%jdbc(hive)show tables; ...org.apache.hive.jdbc.HiveDriver class java.lang.ClassNotFoundException...SolutionThe permanent solution is currently planned to be delivered in HDP 2.6. Here is a workaround to follow:a) I have tarred /usr/hdp/current/zeppelin-server/local-repo into zeppelin-local-repo.tar.gz and located on https://drive.google.com/drive/folders/0B-YVWxQz56HubWhUdEdVWGZ1Mms?usp=sharing. As this is my google drive I can allow you to access it after receiving the request from youb) download the file into /tmp/zeppelin folder c) extract it - this should created local-repo folder. So, you will get /tmp/zeppelin/local-repo with all the subfolders in it d) copy the content of /tmp/zeppelin/local-repo into /usr/hdp/current/zeppelin-server/local-repo e) change

Integrating Darknet YOLOv3 Into Apache NiFi WorkflowsDarknet has released a new version of YOLO, version 3. This one is a faster and perhaps more accurate. It's new and shiny and I had to try it. I am liking the results.Flow to Execute ScriptWe call the shell script, then I route out the empty results. I use SplitText to split into individual lines. I use Extract Text with ([^:]+):(.*) to split into our name, value pairs.We also want to process the images produced by YOLOv3.We grab the newest ones from the output directory. I also add meta data extraction and Tensorflow analysis. This data is stored in attributes and can be saved independently via using AttributesToJSON to build a new flow file that we save off separate probably converting into Apache ORC and storing in HDFS for Apache Hive querying. The image file we can store in the cloud, another file system, send to a front end or save in HDFS. Or even email it to someone.The parsed YOLOv3 results in Apache NiFi Att

By default, hiveserver2 doesn't have an entry in the hive-env.sh that provides a way to change the type of GC. The CMS collector is designed to eliminate the long pauses associated with the full GC cycles of the throughput and serial collectors. CMS stops all application threads during a minor GC, which it also performs with multiple threads.In order to change the type of GC:Ambari > Hive > Configs > Advanced hive-env > hive-env template > Add the follow properties. if [ "$SERVICE" = "hiveserver2" ]; then if [ -z "$DEBUG" ]; then export HADOOP_OPTS="-server -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hive/$USER/hs_err_pid%p.log -XX:NewSize=200m -XX:MaxNewSize=200m -Xloggc:/var/log/hadoop/$USER/gc.log-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -Xms1024m -Xmx1024m" else export HADOOP_OPTS="-server -XX:P

A quick glance at NiFi’s 252+ processors shows that it can solve a wide array of use cases out of the box. What is not immediately obvious is the flexibility that its attributes and expression language can provide. This allows it to quickly, easily, and efficiently solve complex use cases that would require significant customization to solve in other solutions.For example, sending all of the incoming data to both Kafka and HDFS while sending 10% to a dev environment and a portion to a partner system based on the content of the data (e.g. CustomerName=ABC). These more complex routing scenarios are easily accommodated using UpdateAttribute, RouteOnAttribute, and RouteOnContent. Another example of NiFi’s flexibility is the ability to multiplex data flows. In traditional ETL systems, the schema is tightly coupled to the data as it moves between systems, because transformations occur in transit. In more modern ELT scenarios, the data is often loaded into the destination with minimal transfo

SYMPTOMS: During HDP upgrades, namenode restart would fail leading to upgrade failure. Following errors are usually seen:-Traceback (most recent call last):File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/jmx.py", line 42, in get_value_from_jmxreturn data_dict["beans"][0][property]IndexError: list index out of rangeTraceback (most recent call last):File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 420, in <module>NameNode().execute()File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in executemethod(env)File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 720, in restartself.start(env, upgrade_type=upgrade_type)File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 101, in startupgrade_suspended=params.upgrade_suspended, env=env)File "/usr/lib/python2.6/site-packages/amba

Can not start run livy jobs from Zeppelin The following can be seen in livy log file: INFO: Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: http://rangerkms.example.com:9292/kms/v1/?op=GETDELEGATIONTOKEN&doAs=w20524%4542test.tt&renewer=r m%2Fsktudv01hdp02.test.tt%40CCTA.DK&user.name=livy, status: 403, message: ForbiddenRoot Cause: Missing proxy users for livy in kms. The solution is to add the following into Ambari Custom kms-sitehadoop.kms.proxyuser.livy.users=* hadoop.kms.proxyuser.livy.hosts=* hadoop.kms.proxyuser.livy.groups=*

1. Install Java 8. Mac should have the Java 6 installed with the OS. We recommend you switch to Java 8 as Hadoop trunk branch does.2. Install Xcode from the App Store3. Install homebrew, the missing package manager for Mac OS X 4. Install all necessary system tools/libraries (mainly maven, make) brew install maven autoconf automake cmake libtool 5. Install the protobuffer package which is heavily used in Hadoop IPC layer.Install homebrew versions brew tap homebrew/versionsInstall the protobuf package version 2.5 brew install homebrew/versions/protobuf250 6. Clone Hadoop git repository git clone git://git.apache.org/hadoop.git 7. Build the Hadoop project using maven mvn clean package -Pdist,native -Dtar -DskipTests=true -Dmaven.site.skip=true -Dmaven.javadoc.skip=true 8. Install IntelliJ IDEA IDE. This is much better than Eclipse. 9. Create a new project using the existing source. Do use the Maven option if asked. 10. Start coding! Bonus followup: there is a known bug in the TestGetTr

Performing a sqoop import using 'hive-import' into a data type char() or varchar() fails with17/06/05 14:04:44 INFO mapreduce.Job: Task Id : attempt_1496415095220_0016_m_000002_0, Status : FAILEDError: java.lang.ClassNotFoundException: org.apache.hadoop.hive.serde2.SerDeException at java.net.URLClassLoader.findClass(URLClassLoader.java:381) Working example. Create the Teradata and hive tables as follows:Terdata> create table td_importme_into_hive (col1 int not null primary key, col2 char(30));Hive> create table td_import (col1 int, col2 char(30)); Executesqoop import --connection-manager {connection info} \ --table td_importme_into_hive --hive-import --hive-table td_import \ -m 1 --split-by col1 This will fail as char / varchar are not supported Hive datatypes for sqoop import from Teradata.Create the hive table with datatype string instead of char() or varchar().

Accessing Feeds from EtherDelta on Trades, Funds, Buys and Sells (Cryptocurrency Analysis)EtherDelta lets you trade Ether or Ethereum-based tokens. Ethereum (https://www.ethereum.org/) is an open-source block chain platform for running smart contracts. They provide a fast web-socket feed of all data coming through the system. We can tap this websocket feed with Apache NiFi to examine and ingest all the trades, funds, buys and sells coming through the system as JSON. Once we ingest, clean up, parse and schematize the data we can run queries on it with Apache Spark SQL and Apache Hive in Apache Zeppelin notebooks. Now the data scientists have a continuing stream of data to play with. Next we will start adding additional feeds from other Ethereum exchanges, Bit Coin APIs and other sources of data. Some of the APIs are REST, some are websockets and some are SDKs. All of these are easy to ingest with Apache NiFi.Initial Ingest from EtherDelta via Web Sockets APIRoute To The

Iy you have deployed and secured your multi-node-cluster with an MIT KDC running on a Linux box (dedicated or not), this can also be applied on a single node cluster.Below is a step by step procedure to grant a group of user(s) on the Edge node with access to services in the cluster. Assumption KDC is running KDC is created KDC user and a master password is available REALM: DEV.COMUsers: user1 to user5Edge node: for users Kerberos Admin user is root or sudoer A good solution security-wise is to copy the generated keytabs to that users' home directory. If these are local Unix users NOT Active directory then create the keytabs in e.g /tmp and later copy them to their respective home directories and make sure to change the correct permissions on the keytabs. A good practice is to ensure a node dedicated to users usually called an EDGE NODE all client software is installed here and not on the Data or Name Nodes! Change directory to tmp# cd /tmp If you have root access, no need for sudo, s

In this article, I'm going to cover a simple solution to control the data processing in NiFi serially or based on an event trigger.This article use Wait and Notify processors and DistributedMapCache controller service to achieve this functionality. Follows the links to the Usage Guide of these processors and Controller Services.Wait ProcessorNotify ProcessorDistributedMapCache ServerDistributedMapCache client serviceNiFi FlowThe flow that I am going to present in this article has a very simple use case. Get the data from the source, at whatever rate, but process it one flow file at a time. Once a flow file is processed, it should trigger the processing of the next flow file. Follows a step by step explanation of what I have implemented to solve this problem.First, the pre-requisites to start the flow.We will be storing the DistributedMapCache to store and retrieve the "signals" to process the flow files and hence we would need the Server and Client Service for that. Follows a quick des

This article is based on the following Kaggle competition: https://www.kaggle.com/arjunjoshua/predicting-fraud-in-financial-payment-services It is a Scala-based implementation of the data science exploration written in Python. In addition to training a model, we also have the ability to batch-evaluate a set of data stored in a file through the trained model. Full configuration, build, and installation instructions can be found at the GitHub repo: https://github.com/anarasimham/anomaly-detection When you execute the model training, you'll get various lines of output as the data is cleaned and the model is built. To view this output, use the link provided by the Spark job console output. This will look like the following: 18/05/10 14:33:58 INFO Client:client token: N/Adiagnostics: N/AApplicationMaster host: <HOST_IP>ApplicationMaster RPC port: 0queue: defaultstart time: 1525962717635final status: SUCCEEDEDtracking URL: http://<SPARK_SERVER_HOST_NAME>:8088/proxy/applicatio