Resources

Druid Kafka Integration Service + Hive This tutorial will focus on was of ingesting raw streaming data into druid and exposing the data in hive for high performance analytics. For Reference, this tutorial was built with CentOS 7, HDP 3.0.1 w/ LLAP enabled and Kafka installed and running This tutorial will be divided into 6 sections: Ambari Configuration Kafka Setup Setup Data Stream Generating and Running the Supervisor Spec Hive Integration Create Streaming Druid Data Sources through HiveKafka Integration Service With Kafka Ingestion Service, Druid can guarantee that all records received will the processed and stored. That’s in contrast to the older push method of Tranquility. If an event was received after the window period with tranquility, the record was dropped. This meant that a secondary batch process needed to be run to true up the data. With Kafka Ingestion Service, if an event is received late, a segment will be created for it regardless of when it was received. This is h

Below are some FAQ's which helps you to quickly identify some important info for DPS-DLM deployment.Pre-req's for DPS and DLMDB version- postgres 9.3 to 9.6OS - RHEL 7.0 and aboveAmbari - 2.6.2HDP 2.6.5Distcp should work b/w source and target clusters.Beacon user should be created in AD there is no choice of using custom user in this DLM 1.1 releasethe onboarded service user for your application should exists in AD and needsto be resolved(id <username> on both source and target clusters .docker versionRequired ports needs to be open b/w source and target clusters and also to access DPS UIwhere to install DLM and DPS software components?DLM engine needs to be installed as m-pack on both clusters using Ambari serverDLM app is dockerized container needs to be installed on DPS hostWhich URL needs to be given to register cluster in DPS UIthe Ambari URL integrated with knox http://<>:8443I'm unable to see the DLM icon in DPS UI after enabling DLM component in DPSUser needs to be

Introduction Knowing how and when to leverage cloud infrastructure and when to use on premises equipment is at the heart of today's mature enterprise data strategies. I decided to dive deeper into the subject, and, like I did with the Personality Detection data science and engineering platform I built, I wanted to use an example to illustrate how to approach how to architect a hybrid cloud infrastructure. These types of endeavors can be quite theoretical (read hard to follow and potentially boring), so I decided to attach this hybrid cloud platform demonstration to something very concrete: my own health. I signed up for a marathon that will take place March 29, and I am starting to train November 5th. I plan to use my training, sleep, nutrition and general health data to evaluate and eventually predict my perceived level of fatigue that I coined under the term: "Beast Mode Quotient" or BMQ. This article is an introduction to the architecture and data flows I will put in place. It

My Environment details:HDP Version - 2.4.2.0Ambari - 2.2.2.0Mysql Version - 5.1.34.Hive shell was taking lot of time to load and when I checked Hive Metastore logs, I found below exception:com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1 .Complete StackTrace:NestedThrowablesStackTrace: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.re

In this article we will see how to add sample users and enable password based authentication for the Zeppelin UI.By default when we access Zeppelin we see that we are able to access it as "anonymous" user. (means users are not challenged to provide credentials).1. Login to Ambari UI and then navigate to :Ambari UI --> Zeppelin Notebook --> Configs --> Advanced --> Advanced zeppelin-shiro-ini2. Then add the users inside the "[users]" section as following:[users]# List of users with their password allowed to access Zeppelin.# To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISectionsadmin = adminuser1=user1pwduser2=user2pwd3. Also edit the "[urls]" section and add "authBasic" as following to tell which all URL patterns needs to be protected:[urls]# anon means the access is anonymous.# authcBasic means Basic Auth Security# To enfore security, comment the line below and uncomment the next one /

This article is designed to extend my articles Twitter Sentiment using Spark Core NLP in Apache Zeppelin and Connecting Solr to Spark - Apache Zeppelin Notebook I have included the complete notebook on my Github site, which can be found on my GitHub site. Step 1 - Follow the tutorial in the provide articles above, and establish an Apache Solr collection called "tweets" Step 2 - Verify the version of Apache Spark being used, and visit the Solr-Spark connector site. The key is to match the version of Spark the version of the Solr-Spark connector. In the example below, the version of Spark is 2.2.0, and the connector version is 3.4.4%spark2scsc.version</p><p>Step 3 - Include the Solr-Spark dependency in Zeppelin. Important note: This needs to be run beforethe Spark Context has been initialized</p><pre>%dep z.load("com.lucidworks.spark:spark-solr:jar:3.4.4")//Must be used before SparkInterpreter (%spark2) initialized//Hint: put this paragraph before any S

I've just stared the third edition of the fastai course with Jeremy Howard. There’s been a lot of buzz around fast.ai and how the non-profit is making deep learning accessible to hundreds of thousands of developers. The latest mention was in the Economist: https://www.economist.com/business/2018/10/27/new-schemes-teach-the-masses-to-build-ai. For me the most exciting part of the course is learning how to get cutting edge results (in the 90%+ accuracy range) with just a few lines of code using fastai library methods that have best practices baked in.Below, I'll present a few lines of code that allow you to quickly classify different breeds of cats and dogs. You may recall that distinguishing between cats and dogs was big just a few years ago but now it's too easy. Thus, we’re using the Oxford-IIIT Pet Dataset http://www.robots.ox.ac.uk/~vgg/data/pets/.The code example references the latest fastai library v1.0.x built on top of PyTorch. See the github repo for more details https://github

Objectives HDPSearch 4.0 was recently announced (Blog) which upgrades Solr from 6.6 to 7.4. The HPDSearch 4.0 Ambari management pack will install HDPSearch 3.0 for HDP 2.6 and HDPSearch 4.0 for HDP 3.0. HDP 3.0 is required for HDPSearch 4.0 because the HDFS and Hive libraries have been updated for Hadoop 3.1. Using Cloudbreak 2.8 Tech Preview (TP) you can install an HDP 3.0 cluster that includes HDPSearch 4.0 using Cloudbreak's management pack extensions. Cloudbreak 2.8 is a Tech Preview release and is not suitable for production usage. Similarly CB 2.8 TP doesn't officially support deploying HDP 3.0 clusters. The intent is to become familiar with the process for when Cloudbreak 2.9 is released. This tutorial is designed to walk you through the process of deploying an HDP 3.0 cluster which includes HDPSearch 4.0 components on AWS using a custom Ambri blueprint. Prerequisites You should already have an installed version of Cloudbreak 2.8. You can find the documentation on Cloudbreak h

In the previous article, we showed how to deploy a cluster running HDF 2.x or 3.x. Here we will look into enabling SSL for Apache Nifi on the cluster setup previously and optionally setup identity mappings. This approach also sets up users/authorizations using Nifi's file-based authorizer (as opposed to Ranger based authorizer).Tested with HDF 2.x, 3.0, 3.21. Configure Nifi for SSL There are 2 options for configuring SSL for Apache Nifi via Ambari:i). Use Nifi CA to generate self-signed certificates (good for quick start/demos)ii). Use existing certificates (used in production envs)Option i) - Use Nifi Certificate Authority (CA) to generate self-signed certificates:Assuming Nifi CA is already installed (via Ambari when you installed NiFi), you can make the below config changes in Ambari under Nifi > Configs > “Advanced nifi-ambari-ssl-config” and click Save to commit the changes: a) Enable SSL? Check boxb) Clients need to authenticate? Check boxc) NiFi CA Token - Set this to long

This article is designed to extend the great work by @Ali Bajwa: Sample HDF/NiFi flow to Push Tweets into Solr/Banana, HDFS/Hive and my article Connecting Solr to Spark - Apache Zeppelin Notebook I have included the complete notebook on my Github site, which can be found on my Github site. Step 1 - Follow Ali's tutorial to establish an Apache Solr collection called "tweets" Step 2 - Verify the version of Apache Spark being used, and visit the Solr-Spark connector site. The key is to match the version of Spark the version of the Solr-Spark connector. In the example below, the version of Spark is 2.2.0, and the connector version is 3.4.4 %spark2scsc.versionres0: org.apache.spark.SparkContext = org.apache.spark.SparkContext@617d134ares1: String = 2.2.0.2.6.4.0-91 Step 3 - Include the Solr-Spark dependency in Zeppelin. Important note: This needs to be run before the Spark Context has been initialized. %depz.load("com.lucidworks.spark:spark-solr:jar:3.4.4")//Must be used before SparkI

The idea of this article is to help admins detect artifacts(files/folders) in the cluster that are older than certain days. Also, in certain cases, there may be empty directories that are lying in the cluster which are not used and hence contribute to the small file issue. Hence, we have the attached script which performs, 1. Identifies files older than X days. 2. Identifies folders older than X days. 3. Deletes empty folders.Script ExecutionScript name is "findAll.sh" which expects 2 parameters which is 1. Age of the artifact (file/folder) in terms of days. 2. Actual location of the artifact (file/folder) in HDFS. Based upon the type of artifact and kind of operation you would have to choose one of the three operations. NOTE: 1. Please make sure the user running the script has permissions to execute the command on the artifacts that is passed as parameter to script. 2. Also, running this script once may take some time based upon the size/hierarchy of the folders. But once

Objectives The release of Cloudbreak 2.7 enables you to deploy Hortonworks Data Flow (HDF) clusters. Currently there are two HDF cluster types supported: Flow Management (NiFi) and Messaging Management (Kafka). Cloudbreak expects HDF clusters to be deployed with security (LDAP, SSL). However, for testing purposes, many people would like to deploy a cluster without having to go through the steps of setting up SSL, LDAP, etc. Therefore, we'll need to modify the default HDF Flow Management blueprint to loosen the security configuration. This is not recommended for production use cases. This tutorial is designed to walk you through the process of of deploying an HDF 3.1 Flow Management Cluster on AWS using Cloudbreak 2.7 using a custom blueprint. Prerequisites You should already have an installed version of Cloudbreak 2.7. You can find the documentation on Cloudbreak here: Cloudbreak Documentation You can find an article that walks you through installing a local version of Cloudbreak wit

When you install HDP cluster and if you select 'MySQL' database to store ambari or hive metastore or Ranger or oozie databases. The important aspect is to setup MYSQL replication and below step will illustrate how to setup MYSQL database replication.Mysql Master-Master-Active-Passive Replication : ================================================ step 1- Configure Master-Slave Replication ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ On server A: ------------1) Shutdown the mysql server process if its running. #mysqladmin -u root shutdown -p2) Edit my.cnf file with following values:# cat /etc/my.cnf [mysqld] datadir=/hadoop/mysql socket=/var/lib/mysql/mysql.sock user=mysql # Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links=0 # New parameters for Ranger partitioned tables innodb_file_per_table=1 innodb_buffer_pool_size=4G innodb_flush_log_at_trx_commit=2 innodb_flush_method=O_DIRECTlog_bin=mysql-bin binlog_format=ROW server_id=10 innodb_support_xa=

So I log into the cloudera website, and than start clicking links, and suddenly notice that I am not logged in any more.  I keep visiting pages that ask me to sign up or sign in.  This happens whether I open a link in another tab or in current tab.  Also, sometimes I am on the page I want to view, but cloudera forces me to log in.  After I log in, I am no longer on the page I wanted to see.  If I hit the back button in my browser, I can't get back to where I was.  I can't figure out how to get back to the link after I logged in.  It's really frustrating. The cloudera website keeps forcing me to enter my personal information: name, email, phone, etc.  It would not be so bad if it forced me to do this once, but it keeps forcing me to do this when I want to see training.  That is true even if I have logged in already.  It is really annoying. Is there anything I can do?

Configure Livy in Ambari Until https://github.com/jupyter-incubator/sparkmagic/issues/285 is fixed, set livy.server.csrf_protection.enabled ==> false in Ambari under Spark Config - Advanced livy-confInstall Sparkmagic Details see https://github.com/jupyter-incubator/sparkmagic Install Jupyter, if you don't already have it: $ sudo -H pip install jupyter notebook ipython Install Sparkmagic: $ sudo -H pip install sparkmagic Install Kernels: $ pip show sparkmagic # check path, e.g /usr/local/lib/python2.7/site-packages $ cd /usr/local/lib/python2.7/site-packages $ jupyter-kernelspec install --user sparkmagic/kernels/sparkkernel $ jupyter-kernelspec install --user sparkmagic/kernels/pysparkkernel Install Sparkmagic widgets $ sudo -H jupyter nbextension enable --py --sys-prefix widgetsnbextensionCreate local Configuration The configuration file is a json file stored under ~/.sparkmagic/config.json To avoid timeouts connecting to HDP 2.5 it is important to add "livy_server_

Motivation:When Knox is configured for perimeter security, the end users need to depend heavily on cURL tool or the browser to access the Hadoop services exposed via Knox. Similarly, Hive queries can be submitted by using WebHCat (Templeton) service via Knox. User can also set various parameters required for Hive job to run correctly.cURL Command Syntax:Here's the cURL command syntax which can be used to submit a Hive Job via Knox:$curl -ivk -u <username>:<password> -d <Hive parameters> [-d ...] https://<knox-server-FQDN>:8443/gateway/<topology>templeton/v1/hive"Complete list of Hive parameters can be found in WebHCat cURL Command Reference.The most important Hive parameters are: Hive Query : -d execute="<Hive-Query>" ORHive Program : -d file="/hdfs/path/to/hive/program"Specifies a Hive query string using 'execute' OR HDFS file name of Hive program to run using 'file'. It is mandatory to provide either "execute" OR "file" option. Hive Configuration :

Objective In a separate HCC article, I detailed how to offload nodes using the NiFi UI. This article covers how to perform the same operation using the NiFi Toolkit CLI. Note: This tutorial assumes you have already setup a running NiFi cluster. The examples included are for a 2 node NiFi cluster.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.8.0 Apace NiFi Toolkit 1.8.0Offload Nodes via CLI TutorialQueue FlowFiles First, generate some queued flowfiles: The queued flowfiles are distributed between both nodes in the cluster: Disconnect Node From a terminal window, navigate to the directory where the NiFi Toolkit was installed. From the /nifi-toolkit-1.8.0/bin directory, run: ./cli.sh Enter the command nifi get-nodes to retrieve the node IDs: #> nifi get-nodes# Node ID Node Address API Port Node Status- ------------------------------------ ------------ --------

Deep learning offers the opportunity to improve clinical outcomes in medicine, enabling healthcare providers to serve growing patient populations, and providing care givers with the tools they need to focus on patients with critical conditions. The applications for the use of deep learning in medicine extend across all areas where visual diagnosis is used in patient care. This includes radiology (arteriography, mammography, radiomics), dermatology, and oncology, as well as clinical and biotech R&D. In this series, we will explore these applications, including the staging of data, training of deep learning models, evaluating models, the use of distributed compute, and the operationalization of our models. In this article, we will take a look using deep learning to classify Melanoma. Specifically, we will use images of skin, to classify whether the patient associated with the image, has a malignant tumor. The diagram below shows a classification model based on Stanford M

If MySQL Driver does not exist by default, or not installed when MySQL is installed, then the provisioned cluster has the following errors. In this case, create a Recipe in Cloudbreak to install MySQL jdbc driver at pre-ambari-start time:During Create Cluster, use Advance option. Pick the recipe pk-install-mysql-driver-pre from the drop down and click Attach. This will ensure that the jdbc driver is installed before Ambari is started. This should then result in a clean install.

If you see errors related to YARN Registry DNS Bind Port. Where the YARN Registry DNS is stopped, most probably due to a port conflict. Then go into the Advanced Configurations. Look for the parameter RegistryDNS Bind Port and change is to a port that does not have a conflict. I changed it from 53 to 553.Save configuration changes. Restart all components that are impacted. This should take care of the issue and YARN Registry DNS will start successfully.

OBJECTIVE: Delete the cluster added in the Dataplane using utility script provided in the product. OVERVIEW:DPS-1.1 doesn't support the deletion of added clusters from the UI. If the users want to delete the clusters added in the Dataplane , an utility script is provided which can be used for deleting the HDP Clusters added. PREREQUISITES: Install jq : jq is a lightweight and flexible command-line JSON processor. STEPS: By default, "rm_dp_cluster.sh" script will be located at /usr/dp/current/core/bin. Execute the rm_dp_cluster.sh script by providing the following parameters DP_JWT : Value of dp_jwt cookie from a valid users browser sessionnHADOOP_JWT : Value of hadoop-jwt cookie from a valid users browser sessionDP_HOST_NAME : Hostname or IP address of the DataPlane serverCLUSTER_NAME: Ambari Cluster Name of the cluster to deleteDATA_CENTER_NAME : Name of the cluster datacenter to delete For e.g:Format:./rm_dp_cluster.sh <DP_JWT> <HADOOP_JWT> <DP_HOST_NAME> <CLUST

Q. What is Delegated Admin Privilege? Instead of having a single central authority creating security policies for the entire Hadoop resources, Apache Ranger provides a feature, “Delegated-Admin” to delegate the security policy management to other resource administrators. For example, the permissions associated with hdfs:/app/finance folder and all its contents can be entirely managed finance-admin group by granting “delegated-admin” permission to finance-admin group on “hdfs:/app/finance with recursive=true” In simple-terms, Delegated-Admin permission allows other resource administrators to manage permissions for their managed-resources. Q. What are the roles assigned to a user within ranger-ui? In Apache Ranger, an user can be set with one of the following role: ADMIN, USER, KEYADMIN. When a user is assigned with KEYADMIN role, he/she will have the ability to manage all ranger-kms policies and kms-audit information within the ranger-admin UI. Also, he/she can provide

Problem: How to enable LDAP for users to view Activity Explorer Dashboard. We may have shiro.ini enabled for LDAP for Zepplin but it is different for Activity Explorer Dashboard.Solution: On SmartSense Activity Explorer node, under [main] sectioncd /etc/smartsense-activity/conf/vi shiro.ini[main]sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManagersecurityManager.sessionManager = $sessionManagersecurityManager.sessionManager.globalSessionTimeout = 86400000shiro.loginUrl = /api/loginldapRealm = org.apache.zeppelin.realm.LdapRealmldapRealm.contextFactory.systemUsername = <fill your details>ldapRealm.contextFactory.systemPassword = <fill your details>ldapRealm.contextFactory.authenticationMechanism = SIMPLEldapRealm.contextFactory.url = <fill your details>ldapRealm.pagingSize = 200ldapRealm.authorizationEnabled = trueldapRealm.searchBase = <fill your details>ldapRealm.memberAttributeValueTemplate = <fill your details>ldapRealm.userSearch

General Recommendations - Only fail tuples if there's a recoverable exception - Use an Exception/Error Logging Bolt/Stream and send the output to a Kafka topic so all logs are consolidated in place - Be careful about supervisor saturation / overallocation i.e. running more threads than available CPU resources - Configure parallelism based on throughput requirements (use Throughput Equation to calculate) - Be aware of data spikes and plan for them in your topology / use cases / operations - If using Kafka, setup Kafka Producer-Consumer Lag analysis and dashboard - Benchmark topologies to assess "health" operations numbers for individual bolt performance ## Troubleshooting topology performance issues: - Check the topology stats from the Nimbus UI - Look for Failures (in spout or specific bolt) if any that module is having issues. - If a topology has Kafka error bolt then open a console consumer to looks at the error logs from Kafka - Now find out what supervisor the worker is running on

When Oozie launcher(map only mapreduce job) gets scheduled to run on RHEL7 node in a mixed OS environment, it may get failed with below ERROR(stderr section of Oozie launcher logs):Container: container_e1XX_XXXXXXX_0X_00000X on XXXXXX_XXX_XXXXXXLogAggregationType: AGGREGATED===============================================================================================================LogType:stderrLog Upload Time:Tue XXXXXXXXXXXXLogLength:XXLog Contents:SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".SLF4J: Defaulting to no-operation (NOP) logger implementationSLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details./usr/bin/env: bash: No such file or directoryFailing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.impl.MetricsSystemImpl).log4j:WARN Please initialize the log4j system properly.log4j:WARN See http://logging.apache.org/l