Community Articles

Description: Once you have a Hadoop cluster that is using Kerberos for authentication, you have to do the following in Ambari to configure Knox to work with that cluster.1) Grant proxy privileges for Knox in core-site.xml hadoop.proxyuser.knox.groups=*hadoop.proxyuser.knox.hosts=*2) Grant proxy privileges for Knox in webhcat-site.xmlwebhcat.proxyuser.knox.groups=*webhcat.proxyuser.knox.hosts=*3) Grant proxy privileges for Knox in oozie-site.xmloozie.service.ProxyUserService.proxyuser.knox.groups=*oozie.service.ProxyUserService.proxyuser.knox.hosts=*4) Update hive-site.xml and set the following properties on Hive Server2 hosts. Some of the properties may already be in the hive-site.xml. Ensure that the values match the ones below.hive.server2.allow.user.substitution=truehive.server2.transport.mode=httphive.server2.thrift.http.port=10001hive.server2.thrift.http.path=cliserviceNOTE: The properties shown above allows all users and all hostnames. Please consider security and update the valu

In Apache NiFi 1.2, there are processors to Get and Put data to an MQTT broker, which is popular in IoT because of it's small footprint and speed. MQTT is supported by Eclipse and IBM.I created an example on the HDP 2.6. I downloaded and installed the latest Apache NiFi 1.2 there as well as an example MQTT Broker (Mosquitto) http://mosquitto.org/.To Install Mosquitto on HDP 2.6 (Centos 7.x)sudo wget http://download.opensuse.org/repositories/home:/oojah:/mqtt/CentOS_CentOS-6/home:oojah:mqtt.reposudo cp *.repo /etc/yum.repos.d/sudo yum -y updatesudo yum -y install mosquittoTo Verify the Settings and Prepare Logs[root@sandbox opt]# cat /etc/mosquitto/mosquitto.conf# Place your local configuration in /etc/mosquitto/conf.d/pid_file /var/run/mosquitto.pidpersistence truepersistence_location /var/lib/mosquitto/#log_dest file /var/log/mosquitto/mosquitto.loginclude_dir /etc/mosquitto/conf.d[root@sandbox opt]# vi /etc/mosquitto/mosquitto.conf[root@sandbox opt]# mkdir -p /var/log/mosquitto[ro

I have written a small Java 8 Spring Boot application to view Twitter tweets that I have ingested with NiFi and stored as JSON files in HDFS. I have an external Hive table on top of those, from those raw tweets I ran I Spark Scala job that added Stanford CoreNLP Sentiment and saved it to an ORC Hive Table. That is the table I am querying in my Spring Boot visualization program. To show something in a simple AngularJS HTML5 page, I have also queried that microservice which has a method for calling Spring Social Twitter to get live tweets. For this you will need JDK 1.8 and Maven installed on your machine or VM. I used Eclipse as my IDE, but I usually use IntelliJ, either will work fine.Java BeanI have a few of the fields specified, this is to put our Hive data into and transport to AngularJS as JSON serialized.public class Twitter2 implements Serializable { private static final long serialVersionUID = 7409772495079484269L; private String geo; private String unixtime; private Stri

When users run hive query in zeppelin via jdbc interperator then it is going to some anonymous user not an actual user.INFO [2017-11-02 03:18:20,405] ({pool-2-thread-2} RemoteInterpreter.java[pushAngularObjectRegistryToRemote]:546) – Push local angular object registry from ZeppelinServer to remote interpreter group 2CNQZ1ES5:shared_processWARN [2017-11-02 03:18:21,825] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2058) – Job 20171031-075630_2029577092 is finished, status: ERROR, exception: null, result: %text org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Unable to fetch table ushi_gl. org.apache.hadoop.security.AccessControlException: Permission denied: user=anonymous, access=EXECUTE, inode=”/apps/hive/warehouse/adodb.db/ushi_gl”:hive:hdfs:drwxr-x— at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTrave

I've seen some postings (including This one) where people are using CSVSerde for processing input data. CSVSerde is a magical piece of code, but it isn't meant to be used for all input CSVs. tl;dr - Use CSVSerde only when you have quoted text or really strange delimiters (such as blanks) in your input data - otherwise you will take a rather substantial performance hit...When to use CSVSerde: For example: If we have a text file with the following data:col1 col2 col3----------------------121 Hello World 4567232 Text 5678343 More Text 6789Pipe delimited it would look like:121|Hello World|4567|232|Text|5678|343|More Text|6789|but Blank delimited with quoted text would look like (don't laugh - Progress database dumps are blank delimited and text quoted in this exact format): 121 'Hello World' 4567232 Text 5678343 'More Text' 6789Notice that the text may or may not have quote marks around it - text only needs to be quoted if it contains a blank. This is a particularly nast

The steps to set up Atlas with Ldaps (ssl) are below. Download the AD CA certificate on Atlas instance echo -n | openssl s_client -connect ad.xx.xx.com:636 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /root/ldaps-ca.cert View the certificate openssl x509 -noout -text -in /root/ldaps-ca.cert Import the AD CA certificate to trust-store in Atlas instance. $JAVA_HOME/bin/keytool -import -alias AD -keystore /home/nixon/ad-truststore -file /root/ldaps-ca.cert #remember the password used while importing into trust-store Verify the certificate added to trust-store. $JAVA_HOME/bin/keytool -list -alias AD -keystore /etc/atlas/conf/ad-truststore Change ownership of trust-store if needed chown atlas:hadoop /etc/atlas/conf/ad-truststoreAdd the following config to Atlas Advanced atlas-env metadata_opts via Ambari, so that ATLAS_OPTS is set within atlas-env.sh-Djavax.net.ssl.trustStore=/etc/atlas/conf/ad-truststore -Djavax.net.ssl.trustStorePassword=<password> Or can be

Apache Kafka Network Bandwidth Quotas Quotas enable cluster servers to impose limits on the volume of data or number of requests served to clients, hence mitigating the effects of DOS [1] attacks and preventing ill behaved clients from becoming a source of DOS towards other clients.In Apache Kafka there are two types of quotas. Network bandwidth quotas (since v0.9) impose byte rate thresholds and are specified in bytes/s. Request rate quotas are specified as a percentage of CPU utilization.Managing Quotas In Kafka quotas are unlimited by default. It is possible to set quota defaults and overrides for tuples (user,client-id), users and clients [2]. Client quotas can be enforced in secure and non-secure clusters using the client property client.id. User and (user,client-id) quotas are suitable for secure environments. All the clients that belong to the same logical grouping [3] will be subject to the quota for that grouping. Furthermore, the quota will be shared across all the instanc

For this article, I am using Ambari-2.5.2.0 and trying to upgrade HDP-2.5.3 to HDP-2.6.2.If you have a large HBase cluster, it can take a long time to do a HBase snapshot. As part of upgrade, this is one of the steps which Ambari will perform for you. However, for a large cluster, this step can actually lead to a timeout from Ambari UI and may result in further inconsistencies just before the Finalize Upgrade step.To overcome this, a lot of people have started performing a manual HBase Snapshot before the upgrade.However, not many have found a way to force Ambari to skip this step and save some time instead of waiting for it to timeout in order to proceed to next step during the upgrade.Here is how you can skip HBase Snapshot step altogether (in case you want to perform it manually before the upgrade):Caution: The steps described below are a hack! It is not recommended that we go about making changes to upgrade XML files based on our needs. Please exercise caution and measure your risk

For this edge use case we are using NVidia's TensorRT as well as Apache MXNet. From TensorRT I am using imageNet for image recognition and detectNet for object localization. For Apache MXNet, I am using their image classifier. So we have multiple deep learning frameworks run on the same capture from an attached USB webcam. For this example I am using a Logitech HD1080, while the Jetson TX1 supports 6+ concurrent high end cameras for those with high end use cases. They also have a more powerful Jetson TX2 for more intense use cases as it has more RAM and a better GPU. Quick Hardware Breakdown NVIDIA Maxwell™ GPU with 256 NVIDIA® CUDA® Cores 4 GB LPDDR4 Memory Python Script# 2017 load pictures and analyze# https://github.com/tspannhw/mxnet_rpi/blob/master/analyze.pyimport timeimport sysimport datetimeimport subprocessimport sysimport urllib2import osimport datetimeimport tracebackimport mathimport random, stringimport base64import jsonfrom time import gmtime, strftimeimport m

OverviewPartner demo kit is built and maintained by the Hortonworks Partner Solutions team. The purpose of the demo kit is to enable the partners to:Quickly bring up a HDP environments with pre-built demosLeverage available demos to understand the capabilities of the platformUse the demos as part of business conversation to demonstrate the art of possibleThe remainder of this article provides a short description of the 3 demos packaged within the demo kit and step by step instruction on:How to launch the demo kit on AWS or on private cloudHow to execute the demos provided with the demo kitOther VersionsThe Security/Governance Demo kit for HDP 2.6 can be found hereThe previous version of demo kit (for HDP 2.5) can be found herePre-requisitesWhen using AWS, you must already have created your Amazon Web Services account. Sample steps for doing this can be found here. If you have an AWS promo code, you can apply it to your account using the steps here.For running the sentiment demo, you mu

In order to utilize the directory server's nested groups information across hadoop, nested groups information must be retrieved for hadoop group mapping as well as for Ranger policy authorization. Retrieving nested group information for Ranger is explained here. This article covers the configuration required for retrieving nested group information for hadoop LdapGroupMapping.Nested group membership in LdapGroupMapping is introduced as part of HADOOP-12291 and is available in HDP 2.6. Let's take the same directory server structure example that is used for Ranger.Sample Active directory structure with Nested groupsUsecase: Admin wants "Marketing Group" and "AMER Marketing Group" to be represented as hdfs groups for users "Adam Will", "John Doe", and "Mary Sam"In the above example, user "John Doe" is a member of "US Marketing Group". But the directory server structure also contains multiple nested group levels like - “US Marketing Group” is a member of “AMER Marketing Group” which again i

I have been working on EDW for last 10 years. Applying well established relational concepts to Hadoop I have seen many anti-patterns. How about some patterns which work? Lets get to work.Slowly changing dimensions are a known and well established design pattern. Patterns were established on relational theory. Why? Those were the dominant database tech used by virtually everyone. This article in no way expresses the only way to do SCD on Hadoop. I am sharing with you a few patterns which lead to victory.What is relational theory you ask?"In physics and philosophy, a relational theory is a framework to understand reality or a physical system in such a way that the positions and other properties of objects are only meaningful relative to other objects." - wikiSo now we have a challenge. Hadoop and all the integrated animals were not based or found on relational theory. Hadoop was built on software engineering principles. This is extremely important to understand and absorb. Do

INGESTING RDBMS DATAI previously posted an article on ingesting and converting data (https://community.hortonworks.com/articles/64069/converting-a-large-json-file-into-csv.html). Once you have a SQL database loaded, you will eventually need to store your data in your one unified datalake. This is quite simple with NiFi. If you have a specialized tool that reads from your RDBMS logs and sends them to Kafka or JMS, that would be easy to ingest as well. For those wishing to stay open source, NiFi works great. If you don't have a good increasing key to use, you can add an article one that increases on every insert. Almost every database supports this from MariaDB to Oracle.ALTER TABLE `useraccount` ADD COLUMN `id` INT AUTO_INCREMENT UNIQUE FIRST; For mine, I just added an autoincrement id column to be my trigger.For Apache NiFi, you will need connections to all your sources and sinks. So I need a DB Connection Pool for Apache Phoenix and MySQL (DBCPConnectionPool) as well as Hi

Update:This is an update to my previous article on the same topic. This covers the new features added in HDP 2.6 (and Zeppelin 0.7). Motivation:Starting HDP 2.6, a new Shiro configuration implementation has been added in Zeppelin to handle LDAP/Active Directory authentication and authorization. It fixes lot of known issues (Bind issue, limited search/filter options, Group based authorization etc.) present in earlier versions and this should be used for any kind of LDAP/AD authentication + authorization going forward.Configuration:1. While most of the configuration steps remain same from the previous article, the following "shiro_init_content" is where the most of the magic happen:Note: Before pasting this configuration in your Zeppelin configuration, please change the Active Directory details to suit your AD environment.# Sample LDAP configuration, for Active Directory user Authentication, currently tested for single Realm[main]ldapRealm=org.apache.zeppelin.realm.LdapRealmldapRealm.con

DescriptionDuring HDP Upgrade, Hive Metastore restart step fails with message - "ValueError: time data '2017-05-10 19:08:30' does not match format '%Y-%m-%d %H:%M:%S.%f'"Following is the stack trace:Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 211, in <module> HiveMetastore().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute method(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 841, in restart self.pre_upgrade_restart(env, upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 118, in pre_upgrade_restart self.upgrade_schema(env) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 150, in upgrade_schema status_params.tmp_dir) File "/usr

IntroductionAdding a group as a member to another group is called nesting. If a group (groupA) is a member of another group (groupB), then the users belonging to the member group (groupA) are part of the parent group (groupB) as well. Nesting can be very useful in delegating access through inheritance. Several large enterprises have their groups in LDAP/AD nested within other groups. Security admins want the users in those nested groups to be associated in Ranger so that they are available for policy authoring in Ranger Admin. In HDP 2.6 (RANGER-1735), ranger usersync introduced support of nested group membership representation for policy authoring. Note:- In order to utilize the nested group mapping across hadoop, this feature must be configured for hadoop LdapGroupMapping as well. Configuring hadoop LdapGroupMapping for nested groups is explained here.Sample Active directory structure with Nested groupsUsecase: Admin wants to give access to some resources for all the users under “AME

H2O is an open source deep learning technology for data scientists. Sparkling Water allows users to combine the fast, scalable machine learning algorithms of H2O with the capabilities of Spark. In this tutorial, I will walk you through the steps required to setup H2O Sparkling Water (specifically PySparkling Water) along with Zeppelin in order to execute your machine learning scripts. Here are a few points to note before I get started:1.) There is a known issue when running Sparkling Water within Zeppelin. This issue is documented in this Jira (AttributeError: 'Logger' object has no attribute 'isatty'). To bypass this issue, I use Zeppelin combined with Livy Server to execute the Sparkling Water jobs. If you are not familiar with Apache Livy, it is a service that enables easy interaction with a Spark cluster over a REST interface.2.) Testing was performed within the following environment:Hortonworks HDP 2.6.2CentOS Linux release 7.2.1511 (Core)Python 2.7.5Spark 2.1.1Zeppelin 0.7.2Now

Pre-requisite Working Metron cluster - deployed via ansible-playbook or via Ambari + Mpack. The node on which opentaxii service is being deployed should have access to HBASE.Step 1 - Deploy Opentaxii Role (Optional - if not deployed) a) Create a playbook to deploy the opentaxii role [root@metron-test ~]# cat metron/metron-deployment/playbooks/install-opentaxii.yml- hosts: metron become: true roles: - role: opentaxii b) Deploy using ansible-playbook [root@metron-test ~]# ansible-playbook -i ~/metron-deployment/inventory/metron_example playbooks/install-opentaxii.yml -e ansible_python_interpreter=python -e ansible_user=root -e ansible_ssh_private_key_file=/path/to/private-keypair.pem -vvv c) Verify the service has been deployed successfully using the command: service opentaxii status This should show the list of subscribed services along with threat feed counts. Here is a sample output: [root@metron-test]# service opentaxii statusguest.phishtank_com

Objective Using correct HDP repositories is a requirement when building Spark production applications that run on HDP. Hence I decided to create this article to help those creating new spark applications using Eclipse with maven that may not know how to reference the Hortornworks repositories instead of the default ones. How-To Following video goes step by step on how to create a simple spark application using the Hortonworks repositories. I will share the content of pom.xml and Hello scala class bellow.Perquisites 1. From market place you need to install Scala IDE for Eclipse. Site: http://scala-ide.org/docs/current-user-doc/gettingstarted/index.html 2. Second install the maven integration for Scala IDE plugin. Site: http://alchim31.free.fr/m2e-scala/update-site/ 3. Finally you need to add the archtype Remote Catalog - Url: http://repo1.maven.org/maven2/archetype-catalog.xml The pom.xml <project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XM

In HDFS, while using Internet Explorer 11 to run a command to create a folder or set file permission with the Files View in Ambari , the command runs successfully. However, the Files View in IE11 are not displayed in the new folder or file permissions. The HDFS displays that the folders are created when using command line or different browsers such as Firefox and Chrome. To resolve this issue, clear the cache to see the newly created folders or folder permissions list in the Files View. For a permanent resolution, do the following to disable the webpage cache in IE 11: From the top right corner of Internet Explorer 11, click the Settings (Gear) icon From the Menu, select Internet Options.On the General tab, locate the Browsing history section, and click Settings.On the Temporary Internet Files tab, confirm that Every time I visit the webpage is selected.On the Caches and Databases tab, confirm that Allow website caches and databases is not selected. Click OK.

We are pleased to announce the availability of Hortonworks Data Flow (HDF) version 3.0.2. This maintenance release of HDF is the final version of our flagship 3.0 product line and includes critical patches and bug fixes. An increasing number of Hortonworks customers are using HDF to meet their needs for enterprise flow and streaming use cases. The team is very pleased to deliver a high quality, well tested release before move onto the next big release. The next release of HDF, version 3.1, is currently in development.For additional information, please refer to:Hortonworks Data Flow (bits) (docs)

We are pleased to announce the certification of IBM Data Science Experience (DXS) Local V1.1.2.01 with Hortonworks Data Platform (HDP) 2.6.3 / Ambari 2.6 on RHEL 7. As an important part of the agreement between the two companies, IBM and Hortonworks collaborated on an extensive set of test cases to specifically validate IBM DSX with both Hortonworks Data Platform and Ambari. This version of IBM DSX integrates with Zeppelin 0.7.3 and allows users to configure Livy interpreter to run workloads on HDP clusters (both secure and unsecure). Users also have the option to launch their DSX jobs on either Spark1 or Spark2.This certification is win-win for both DSX and HDP customers as it brings a production-ready data science experience to HDP customers and at the same time provides DSX customers access to information stored in HDP data lakes with an enterprise grade compute grid. An increasing number of Hortonworks customers are using data science to get a greater value from their data and supp

Objective Using correct HDP repositories is a requirement when building Spark production applications that run on HDP. Hence I decided to create this article to help those creating new spark applications using IntelliJ which don't know how to reference the Hortornworks repositories instead of the default ones. How-ToFollowing video goes step by step on how to create a simple spark application using the Hortonworks repositories. I will share the content of build.sbt and Hello scala class bellow.The build.sbtname := "sparkTest"version := "0.1"scalaVersion := "2.11.12"libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.1.2.6.1.0-129"resolvers := List("Hortonworks Releases" at "http://repo.hortonworks.com/content/repositories/releases/", "Jetty Releases" at "http://repo.hortonworks.com/content/repositories/jetty-hadoop/")Important Note: We will use the HDP 2.6.1 Spark 2.1.1 dependencies to build the project. If you are running different HDP version you need to check and cor

1. IntroductionThis article is an extension of the one created by @Dan Zaratsian - H2O on Livy2. Environment DetailsHere are the environment details I did test it:HDP: 2.6.1Ambari: 2.5.0.3OS: 7.3.1611python: 2.7.5IMPORTANT NOTE: H2O requires python ver. 2.7+3. Installing H2OGo to Zeppelin node and do the following:$ mkdir /tmp/H2O$ cd /tmp/H2O$ wget http://h2o-release.s3.amazonaws.com/sparkling-water/rel-2.1/16/sparkling-water-2.1.16.zip$ unzip sparkling-water-2.1.16.zip4. Testing H2O from CLIGo to Zeppelin node where you downloaded and installed H2O$ export SPARK_HOME='/usr/hdp/current/spark2-client'$ export HADOOP_CONF_DIR=/etc/hadoop/conf$ export MASTER="yarn-client"$ export SPARK_MAJOR_VERSION=2$ cd /tmp/H2O/sparkling-water-2.1.16/bin$ ./pysparkling>>> from pysparkling import *>>> hc = H2OContext.getOrCreate(spark)My test[root@dkozlowski-dkhdp262 bin]# ./pysparklingPython 2.7.5 (default, Jun 17 2014, 18:11:42)[GCC 4.8.2 20140120 (Red Hat 4.8.2-16)] on linux2Type "

Scenario - Cluster managed by Ambari with Ranger Plugins enabled and if Ambari upgrade process fails, below are the things to consider.1) Updating stack_features property in cluster-env config typeGet the current cluster-env version using configs.sh utility./var/lib/ambari-server/resources/scripts/configs.sh get <AMBARI_HOST> <CLUSTER_NAME> cluster-env cluster_env.jsonOpen cluster_env.json and check for "ranger_xml_configuration" feature in stack_features property. If it is not present add below content and save the cluster_env.json.{\n \"name\": \"ranger_xml_configuration\",\n \"description\": \"Ranger code base support xml configurations\",\n \"min_version\": \"2.3.0.0\"\n }Now update the latest cluster-env,/var/lib/ambari-server/resources/scripts/configs.sh set <AMBARI_HOST> <CLUSTER_NAME> cluster-env cluster_env.json 2) Updating Ranger url in plugin properties.Update "ranger.plugin.<SERVICE>.policy.rest.url" property value und