Resources

The Apache Spark - Apache HBase Connector (SHC) is a library to support Spark accessing HBase table as external data source or sink. It provides high performance HBase access via SparkSQL and DataFrames. SHC implements the standard Spark data source APIs, and leverages the Spark catalyst engine for query optimization. It bridges the gap between the simple HBase Key Value store and complex relational SQL queries and enables users to perform complex data analytics on top of HBase using Spark. With the data frame support, SHC leverages all the optimization techniques in catalyst, and achieves data locality, partition pruning, predicate pushdown, Scanning and BulkGet, etc. For the detailed information, please refer the README in SHC github, which is kept up-to-date.

How to enable SSL for Storm UI on an unsecured cluster: 1. Generate keystore and certificate:root@beautiful-storm2 ~]# /usr/jdk64/jdk1.8.0_77/bin/keytool -genkeypair -alias certificatekey -keyalg RSA -validity 7 -keystore keystore.jksEnter keystore password:Re-enter new password:What is your first and last name? [Unknown]: stormWhat is the name of your organizational unit? [Unknown]: stormWhat is the name of your organization? [Unknown]: stormWhat is the name of your City or Locality? [Unknown]: stormWhat is the name of your State or Province? [Unknown]: stormWhat is the two-letter country code for this unit? [Unknown]: stormIs CN=storm, OU=storm, O=storm, L=storm, ST=storm, C=storm correct? [no]: yesEnter key password for <certificatekey> (RETURN if same as keystore password):Re-enter new password:2. Add below properties via Ambari to custom storm-site ui.https.key.password=bigdataui.https.keystore.password=bigdataui.https.keystore.path=/keystore.jks (This is path to your keysto

SYMPTOM: On invoking a mapreduce job or invoking a hive shell (incase where we use tez execution engine, which is default) which involves spinning up a mapreduce container on the mapreduce queue), we get the below error : [hive@sumeshhdpn2 root]$ hadoop jar /usr/hdp/2.2.4.2-2/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 1 10Number of Maps = 1Samples per Map = 10Wrote input for Map #0Starting Job16/08/15 04:12:47 INFO impl.TimelineClientImpl: Timeline service address: http://sumeshhdpn2:8188/ws/v1/timeline/16/08/15 04:12:47 INFO client.RMProxy: Connecting to ResourceManager at sumeshhdpn2/172.25.16.48:805016/08/15 04:12:48 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 129 for hive on ha-hdfs:sumeshhdp16/08/15 04:12:48 INFO security.TokenCache: Got dt for hdfs://sumeshhdp; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:sumeshhdp, Ident: (HDFS_DELEGATION_TOKEN token 129 for hive)16/08/15 04:12:53 INFO input.FileInputFormat: Total input paths to process : 116/08/15 04:12:

Setting up Log Search SSL and HTTPS A keystore and a truststore are required for this setup. These instructions assume that you have already created .jks for the keystore and truststore 1. Create keystore location a. Keystore Setup Place the keystore in /etc/security/certs/ and else by using a symlink you can point to another location of your keystore.jks b. Ensure log search user can read the keystore chown logsearch:hadoop *.keyStore.jks2. Create a truststore for logsearch: a. Cert signed by CA: i. Copy the keystore into <host>.trustStore.jks ii. Create a symlink to this similar to the keystore /etc/security/certs/truststore.jks -> /etc/security/certs/<host>.trustStore.jks b. Ensure log search user can read the trust store chown logsearch:hadoop *.trustStore.jks3. Update Ambari configuration a. Update logsearch UI Protocol to https b. Update Trust store location ( logsearch_t

Error:Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '115' for key 'PRIMARY'ROOT CAUSE: corrupted ambari db-- seeing following errors in Hive View Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '115' for key 'PRIMARY' SOLUTION: we truncated the following tables from amabri db DS_FILERESOURCEITEM_1 | | DS_JOBIMPL_3 | | DS_SAVEDQUERY_6 | | DS_STOREDOPERATIONHANDLE_5 | | DS_TESTBEAN_4 | | DS_UDF_2 Restart ambari server and we are able to access views and run query

Issue:Solr service check fails from ambari when Namenode HA is enabled with error:Unable to create core [collection1_shard2_replica1] Caused by: Connection refusedSolution 1:add this line "SOLR_HDFS_CONFIG=/etc/hadoop/conf" to solr-config-env content at the end of fileRestart Solr and now the service checks should pass.Solution 2:Another workaround is to edit the following file on your Ambari Server: /var/lib/ambari-server/resources/mpacks/solr-ambari-mpack-5.5.2.2.5/common-services/SOLR/5.5.2.2.5/package/scripts/solr.py, and then make this change: https://github.com/lucidworks/solr-stack/commit/7b79894b37b862b86d80c64b34230bc9fed6e54a. Then restart the Ambari Server. With this change in place the Solr instances will be started and will work fine with NN HA.This will be fixed in HDP 2.6

Issue:If there is a custom PID location configured for services and a non standard service account other than user "ranger", before finalizing upgrade, Ambari shows Ranger service as stopped.Solution:Confirm that the Ranger process is running:ps -ef | grep rangerThe pid and the chown values are hard coded in the /usr/bin/ranger*.From this location, start and stop are called.The pidf and the chown values are hard coded in the /usr/bin/ranger-admin from where start and stop are called if you invoke the same from ambari UI.after changing those values to custom parameters ambari reported the service as running from UI.Previous value:cd /usr/bincat ranger-admin | grep -i pid pidf=/var/run/ranger/rangeradmin.pidNew value:vi ranger-adminpidf=<custom location>Save and quitAfter changing these values restart ranger service from ambari

Error:%jdbc (Hive) java.util.ServiceConfigurationError: javax.xml.parsers.DocumentBuilderFactory: Provider org.apache.xerces.jaxp.DocumentBuilderFactoryImpl not found at java.util.ServiceLoader.fail(ServiceLoader.java:239) at java.util.ServiceLoader.access$300(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at javax.xml.parsers.FactoryFinder$1.run(FactoryFinder.java:294) at java.security.AccessController.doPrivileged(Native Method) at javax.xml.parsers.FactoryFinder.findServiceProvider(FactoryFinder.java:289) at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:267) at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:120) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2549) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:25

Error:at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:302)at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:120)at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)... 43 moreCaused by: KrbException: Identifier doesn't match expected value (906)at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60)at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)... 49 more16/11/29 13:13:12 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.Cause:The cause for this issue was that there were multiple accounts in the Active Directory that had a servicePrincipalName value containing the Zookeeper principal names - "zookeeper/<

PROBLEM :Currently in Ranger Audit UI (HDP 2.4) , we do not have a feature or a search filter so that one can pull out a report which answers the question "who made a change to a particular policy" without scrolling through all the pages of audit. A search by Policy Id / Policy Name should solve that.RESOLUTION :We have a Internal Feature request raised to track the same

Synopsis You are a marketing analyst tasked with developing a BI dashboard for management that reports on website activity in realtime throughout the day for Black Friday, the day after Thanksgiving. Unfortunately, all of the data engineers are busy with other projects and cannot immediately help you. Not to worry. With Hortonworks DataFlow, powered by Apache NiFi, you can easily create a data flow in minutes to process the data needed for the dashboard. Source Data: Fictitious online retailer Initech Corporation JSON Formatted Clickstream data Your company stores the raw Clickstream log files in HDFS. The JSON formatted log files are written to HDFS directories partitioned by day and hour. For example: /data/clickstream/2016-11-20/hour=00 /data/clickstream/2016-11-20/hour=01 /data/clickstream/2016-11-20/hour=... /data/clickstream/2016-11-20/hour=23 There could also be subdirectories under the hour This directory structure or a similar variant is commonly fo

Error:DEBUG [master:hbasrmaster01:60000.oldLogCleaner] master.ReplicationLogCleaner: Found log in ZK, keeping: hbasregion.abcd.com%2C60020%2C1446216523936.1475050795365 The duration for which WAL files are kept is controlled by hbase.master.logcleaner.ttl - default is 10 minutes. If the replication peer can be dropped (for longer than the above config value), WAL files would be cleaned. After that the replication peer can be added back. Issue: Hbase OldWAL's in /apps/hbase/data/oldWALs not getting deletedRoot Cause: Hbase had a peer that prevents the deletion of old files.In hbase shell: To List Peers: list_peers To Disable Peers: disable_peer("1") To Remove Peer: remove_peer ("1") List agin to verify removal: list_peers Next tail the Hbase Master log file to ensure the deletion was working. Also look at hbase replication folder in hdfs. Lastly we re-added the peer: add_peer '<n>', "slave.zookeeper.quorum:zookeeper.clientport.:zookeeper.znode.parent"

SummaryHDFS Rolling Upgrade facilitates software upgrade of independent individual components in an HDFS cluster. During the upgrade window, HDFS will not physically delete blocks. Normal block deletion resumes after the administrator finalizes the upgrade. A common source of operational problems is forgetting to finalize an upgrade. If left unaddressed, HDFS will run out of storage capacity. Attempts to delete files will not free space. To avoid this problem, always finalize HDFS rolling upgrades in a timely fashion.This information applies to both Ambari Rolling & Express Upgrade.Rolling Upgrade Block HandlingThe high-level workflow of a rolling upgrade for the administrator is: Initiate rolling upgrade.Perform software upgrade on individual nodes.Run typical workloads and validate new software works.If validation is successful, finalize the upgrade.If validation discovers a problem, revert to the prior software via one of 2 options: Rollback - Restore prior software and re

The following steps explain how to configure LDAP for Zeppelin1) Make sure you can do an ldapsearch with the System Username that has AD permissions to query your OU.Example:ldapsearch -h 10.1.1.10:389 -D adsystem@ABC.YOURCO.COM -w abc123 -b OU=users,DC=ABC,DC=YOURCO,DC=COM dn2) Using Ambari go into Zeppelin Configs and Advanced Zeppelin-env.3) Edit the shiro_ini_content by adding the following parameters (remove existing first and replace with new):[users] admin = yourpassword,admin [main] adRealm = org.apache.shiro.realm.activedirectory.ActiveDirectoryRealm adRealm.url = ldap://10.1.1.10 adRealm.searchBase = OU=users,DC=ABC,DC=YOURCO,DC=COM adRealm.systemUsername = adsystem@ABC.YOURCO.COM adRealm.systemPassword = abc123 adRealm.principalSuffix = @ABC.YOURCO.COM adRealm.authorizationCachingEnabled = true sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager securityManager.sessionManager = $sessionManager securityManager.sessionManager.globalSessionTimeout

This tutorial is a follow on to the Apache Spark Fine Grain Security with LLAP Test Drive tutorial. These two articles cover the entire range of security authroization capabilities available for Spark on the Hortonworks Data Platform.Getting StartedInstall an HDP 2.5.3 Cluster via Ambari. Make sure the following components are installed:HiveSparkSpark Thrift ServerHbaseAmbari InfraAtlasRangerEnable LLAPNavigate to the Hive Configuration Page and click Enable Interactive Query. Ambari will ask what host group to put the Hiveserver2 service into. Select the Host Group with the most available resources.With Interactive Query enabled, Ambari will display new configurations options. These options provide control of resource allocation for the LLAP service. LLAP is a set of long lived daemons that facilitate interactive query response times and fine grain security for Spark. Since the goal of this tutorial is to test out fine grain security for Spark, LLAP only needs a minimal allocation of

Running a hadoop client on Mac OS X and connect to a Kerberized cluster poses some extra challenges. I suggest to use brew, the Mac package manager to conveniently install the Hadoop package: $ brew search hadoop$ brew install hadoop This will install the latest (apache) Hadoop distro, (2.7.3 at the time of writing). Minor version differences to your HDP version will not matter. You may test the installation by running a quick 'hdfs dfs -ls / ' on HDFS. Without further configuration a local single node 'cluster' will be assumed. We now have to point the client to the real HDP cluster. In order to do so you need to copy the full contents of the config files below from any HDP node: Source: /etc/hadoop/{hdp-version}/0/hadoop-env.sh/etc/hadoop/{hdp-version}/0/core-site.xml/etc/hadoop/{hdp-version}/0/hdfs-site.xml /etc/hadoop/{hdp-version}/0/yarn-site.xml Target: /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/hadoop-env.sh/usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/core-site.xml/u

The table definition "LINES TERMINATED BY" only supports newline '\n' right now. This is a known issue and Jira Hive 11996 has already been raised for the issue: To handle the newline characters within the data, you can use the Omniture Data format which uses a EscapedLineReader which gets around Omniture's pesky escaped tabs and newlines. Please note that the data files need to include '\' characters before the newline character within the data and run the below command in sequence and required jars are attached along with data file:add jar /tmp/omnituredata-1.0.2-SNAPSHOT-jar-with-dependencies.jar;add jar /tmp/omnituredata-1.0.2-SNAPSHOT-javadoc.jar;add jar /tmp/omnituredata-1.0.2-SNAPSHOT-sources.jar;add jar /tmp/omnituredata-1.0.2-SNAPSHOT.jar; (Note: jars are available on the HDFS /tmp folder).CREATE TABLE test8(id string,desc string)ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'STORED AS INPUTFORMAT 'org.rassee.omniture.hadoop.mapred.OmnitureDataFileInputFormat'OUTPUTFORMAT '

1Nifi Custom Processor OverviewApache nifi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Its very simple to use product using which you can build "data flow" very easily. As of version 3.0 it has 90 prebuilt processors but then again you can extend them by adding your own custom processors.In this article I am going to talk about how you can build a custom nifi processor in your local machine and then move the final finished processor which is a nar file to nifi.This article is based on a video from youtube and here is the link for that https://www.youtube.com/watch?v=3ldmNFlelhw2Steps to build Customer processorHere are the steps involved to build the custom processor for nifi. I used my mac to build this processor 2.1Required Software Two software that you would need in you location machine are1.maven 2.JavaHere is how you can quickly check if you have them installed mvn -versionjava -versionHere are the results from my ma

SYMPTOM: Not able to login to Ambari UI. The Ambari ui was hung. When checked in logs we found below error -ERROR: +++++++ 16 Sep 201613:41:21,377 WARN [qtp-client-10408] ObjectGraphWalker:209 - The configuredlimit of 1,000 object references was reached while attempting to calculate thesize of the object graph. Severe performance degradation could occur if thesizing operation continues. This can be avoided by setting the CacheManger orCache <sizeOfPolicy> elements maxDepthExceededBehavior to"abort" or adding stop points with @IgnoreSizeOf annotations. Ifperformance degradation is NOT an issue at the configured limit, raise thelimit value using the CacheManager or Cache <sizeOfPolicy> elementsmaxDepth attribute. For more information, see the Ehcache configurationdocumentation. +++++++ ROOT CAUSE: As per above logs we found that the issue is with Ambari cache.This is related to JIRA - https://issues.apache.org/jira/browse/AMBARI-13517RESOLUTION: We have two wa

ObjectiveAccept a parameter -DfileType=[csv|tsv] from the Oozie command line. Use Oozie'sdecision node functionality to simulate an if-then-else conditional operation. If the value of the fileType variable equals tsv, execute a Hive 2 action which will execute theload_policestationstsv.ddl which will in-turn load a tab-separated-value filepolicestations.tsv into a Hive table named policestationstsv. Else, if the value of thefileType variable equals csv, execute a Hive 2 action which will execute theload_policestationscsv.ddl which will in-turn load a comma-separated-value filepolicestations.csv into a Hive table named policestationscsv. These Hive 2 actions will drop any pre-existing policestationstsv or policestationscsv tables from Hive as a preparatory step each time this workflow is run.ProcedureOn an edge node containing the oozie client, change users to the oozie user[root@jyoung-hdp234-1 ~]# su - oozieAuthenticate to the KDC using the oozie service account kerberos keytab[oozie@

ObjectiveLoop over a list of primary crime types ("THEFT", "STALKING", "GAMBLING", "DOMESTIC VIOLENCE"). For each crime type, query hive, get the sum of that primary crime type from the crime table, insert the sum into a new table (crimespertype).This demo uses the oozieloop project written by Jeremy Beard in order to emulate looping in Oozie via sub-workflows. Please see the oozieloop project home page for a detailed explanation of how it works: https://github.com/jeremybeard/oozieloopIMPORTANT!: This demo requires the crime database table existing in Hive as created in the first exercise: Hive-2-Action-in-a-Kerberized-clusterProcedureOn an edge node containing the oozie client, install git if it doesn't already exist[root@jyoung-hdp234-1 ~]# yum install git-allChange users to the oozie user[root@jyoung-hdp234-1 ~]# su - oozieAuthenticate to the KDC using the oozie service account kerberos keytab[oozie@jyoung-hdp234-1 ~]$ kinit -kt /etc/security/keytabs/oozie.service.keytab oozie/jyou

Here I have created a series of demos to highlight workflows that cover examples of using:Hive 2 Action in a Kerberized clusterJeremy Beard's oozieloop project to simulate looping with sub-workflowsDecision nodes to simulate conditional operators (if-then-else)These workflow demo hands-on wikis and source code are hosted on my GitHub repository at: https://github.com/jlyoung/advancedoozieworkflows

SYMPTOMS:HDP running has recently been upgraded to Isilon 8.0.1.0 and kerberized.When trying to start services through Ambari, user receives the following stderr:Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 289, in <module> Resourcemanager().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 124, in start self.wait_for_dfs_directories_created(params.entity_groupfs_store_dir, params.entity_groupfs_active_dir) File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 246, in wait_for_dfs_directories_created self.wait_for_dfs_directory_created(dir_path, ignored_dfs_dirs) File "/usr/lib/python2.6/site-packages/resource_management/libraries/

SYMPTOMS: Starting Falcon server from Ambari is successful, but when user tries to access Falcon web UI. Logs show one or more cluster entities trying to initialize: INFO - [main:] ~ Initializing FS: hdfs://fc1.support.com:8020 for cluster: falconcluster1INFO - [main:] ~ Initializing FS: hdfs://ft2.support.com:8020 for cluster: falcontest2 Start up is delayed with numerous attempts to contact cluster entity. 2016-12-05 13:49:17,240 INFO - [main:] ~ Retrying connect to server: ft2.support.com/xxx.xxx.xxx.xx:8020. Already tried 0 time(s); maxRetries=45 (Client:835)........2016-12-05 14:19:52,971 INFO - [main:] ~ Retrying connect to server: ft2.support.com/xxx.xxx.xxx.xx:8020. Already tried 44 time(s); maxRetries=45 (Client:835)2016-12-05 14:20:13,000 ERROR - [main:] ~ Failed to initialize FS for cluster : (SharedLibraryHostingService:200)org.apache.falcon.FalconException: Failed to initialize FS for cluster : falcontest2 at org.apache.falcon.service.SharedLibraryHostingService

HDP Environment:Secure cluster (kerberos enabled)Users managed by Active Directory (AD)SYMPTOMS:When user tries to access the Falcon UI with the following address, they are prompted to enter their username and password:http://<falcon server host>:15000/index.html?user.name=admin#/After entering the correct AD username and password, user gets this exception in the UI:Using curl to negotiate with Falcon UI URL has no issues.ROOT CAUSE: User is not using kerberos to authenticateRESOLUTION: In order to access the Falcon UI after enabling kerberos, user needs to authenticate using SPNEGO to negotiate with kerberos not with a user name and password. Each browser supports SPNEGO, but configuration is different for each browser. Safari needs no further configuration.After configuring your browser to negotiate using SPNEGO, user must kinit and can try to access the Falcon UI again.