Community Articles

Error:%jdbc (Hive) java.util.ServiceConfigurationError: javax.xml.parsers.DocumentBuilderFactory: Provider org.apache.xerces.jaxp.DocumentBuilderFactoryImpl not found at java.util.ServiceLoader.fail(ServiceLoader.java:239) at java.util.ServiceLoader.access$300(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at javax.xml.parsers.FactoryFinder$1.run(FactoryFinder.java:294) at java.security.AccessController.doPrivileged(Native Method) at javax.xml.parsers.FactoryFinder.findServiceProvider(FactoryFinder.java:289) at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:267) at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:120) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2549) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:25

Error:at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:302)at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:120)at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)... 43 moreCaused by: KrbException: Identifier doesn't match expected value (906)at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60)at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)... 49 more16/11/29 13:13:12 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.Cause:The cause for this issue was that there were multiple accounts in the Active Directory that had a servicePrincipalName value containing the Zookeeper principal names - "zookeeper/<

PROBLEM :Currently in Ranger Audit UI (HDP 2.4) , we do not have a feature or a search filter so that one can pull out a report which answers the question "who made a change to a particular policy" without scrolling through all the pages of audit. A search by Policy Id / Policy Name should solve that.RESOLUTION :We have a Internal Feature request raised to track the same

Synopsis You are a marketing analyst tasked with developing a BI dashboard for management that reports on website activity in realtime throughout the day for Black Friday, the day after Thanksgiving. Unfortunately, all of the data engineers are busy with other projects and cannot immediately help you. Not to worry. With Hortonworks DataFlow, powered by Apache NiFi, you can easily create a data flow in minutes to process the data needed for the dashboard. Source Data: Fictitious online retailer Initech Corporation JSON Formatted Clickstream data Your company stores the raw Clickstream log files in HDFS. The JSON formatted log files are written to HDFS directories partitioned by day and hour. For example: /data/clickstream/2016-11-20/hour=00 /data/clickstream/2016-11-20/hour=01 /data/clickstream/2016-11-20/hour=... /data/clickstream/2016-11-20/hour=23 There could also be subdirectories under the hour This directory structure or a similar variant is commonly fo

Error:DEBUG [master:hbasrmaster01:60000.oldLogCleaner] master.ReplicationLogCleaner: Found log in ZK, keeping: hbasregion.abcd.com%2C60020%2C1446216523936.1475050795365 The duration for which WAL files are kept is controlled by hbase.master.logcleaner.ttl - default is 10 minutes. If the replication peer can be dropped (for longer than the above config value), WAL files would be cleaned. After that the replication peer can be added back. Issue: Hbase OldWAL's in /apps/hbase/data/oldWALs not getting deletedRoot Cause: Hbase had a peer that prevents the deletion of old files.In hbase shell: To List Peers: list_peers To Disable Peers: disable_peer("1") To Remove Peer: remove_peer ("1") List agin to verify removal: list_peers Next tail the Hbase Master log file to ensure the deletion was working. Also look at hbase replication folder in hdfs. Lastly we re-added the peer: add_peer '<n>', "slave.zookeeper.quorum:zookeeper.clientport.:zookeeper.znode.parent"

SummaryHDFS Rolling Upgrade facilitates software upgrade of independent individual components in an HDFS cluster. During the upgrade window, HDFS will not physically delete blocks. Normal block deletion resumes after the administrator finalizes the upgrade. A common source of operational problems is forgetting to finalize an upgrade. If left unaddressed, HDFS will run out of storage capacity. Attempts to delete files will not free space. To avoid this problem, always finalize HDFS rolling upgrades in a timely fashion.This information applies to both Ambari Rolling & Express Upgrade.Rolling Upgrade Block HandlingThe high-level workflow of a rolling upgrade for the administrator is: Initiate rolling upgrade.Perform software upgrade on individual nodes.Run typical workloads and validate new software works.If validation is successful, finalize the upgrade.If validation discovers a problem, revert to the prior software via one of 2 options: Rollback - Restore prior software and re

The following steps explain how to configure LDAP for Zeppelin1) Make sure you can do an ldapsearch with the System Username that has AD permissions to query your OU.Example:ldapsearch -h 10.1.1.10:389 -D adsystem@ABC.YOURCO.COM -w abc123 -b OU=users,DC=ABC,DC=YOURCO,DC=COM dn2) Using Ambari go into Zeppelin Configs and Advanced Zeppelin-env.3) Edit the shiro_ini_content by adding the following parameters (remove existing first and replace with new):[users] admin = yourpassword,admin [main] adRealm = org.apache.shiro.realm.activedirectory.ActiveDirectoryRealm adRealm.url = ldap://10.1.1.10 adRealm.searchBase = OU=users,DC=ABC,DC=YOURCO,DC=COM adRealm.systemUsername = adsystem@ABC.YOURCO.COM adRealm.systemPassword = abc123 adRealm.principalSuffix = @ABC.YOURCO.COM adRealm.authorizationCachingEnabled = true sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager securityManager.sessionManager = $sessionManager securityManager.sessionManager.globalSessionTimeout

This tutorial is a follow on to the Apache Spark Fine Grain Security with LLAP Test Drive tutorial. These two articles cover the entire range of security authroization capabilities available for Spark on the Hortonworks Data Platform.Getting StartedInstall an HDP 2.5.3 Cluster via Ambari. Make sure the following components are installed:HiveSparkSpark Thrift ServerHbaseAmbari InfraAtlasRangerEnable LLAPNavigate to the Hive Configuration Page and click Enable Interactive Query. Ambari will ask what host group to put the Hiveserver2 service into. Select the Host Group with the most available resources.With Interactive Query enabled, Ambari will display new configurations options. These options provide control of resource allocation for the LLAP service. LLAP is a set of long lived daemons that facilitate interactive query response times and fine grain security for Spark. Since the goal of this tutorial is to test out fine grain security for Spark, LLAP only needs a minimal allocation of

Running a hadoop client on Mac OS X and connect to a Kerberized cluster poses some extra challenges. I suggest to use brew, the Mac package manager to conveniently install the Hadoop package: $ brew search hadoop$ brew install hadoop This will install the latest (apache) Hadoop distro, (2.7.3 at the time of writing). Minor version differences to your HDP version will not matter. You may test the installation by running a quick 'hdfs dfs -ls / ' on HDFS. Without further configuration a local single node 'cluster' will be assumed. We now have to point the client to the real HDP cluster. In order to do so you need to copy the full contents of the config files below from any HDP node: Source: /etc/hadoop/{hdp-version}/0/hadoop-env.sh/etc/hadoop/{hdp-version}/0/core-site.xml/etc/hadoop/{hdp-version}/0/hdfs-site.xml /etc/hadoop/{hdp-version}/0/yarn-site.xml Target: /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/hadoop-env.sh/usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/core-site.xml/u

The table definition "LINES TERMINATED BY" only supports newline '\n' right now. This is a known issue and Jira Hive 11996 has already been raised for the issue: To handle the newline characters within the data, you can use the Omniture Data format which uses a EscapedLineReader which gets around Omniture's pesky escaped tabs and newlines. Please note that the data files need to include '\' characters before the newline character within the data and run the below command in sequence and required jars are attached along with data file:add jar /tmp/omnituredata-1.0.2-SNAPSHOT-jar-with-dependencies.jar;add jar /tmp/omnituredata-1.0.2-SNAPSHOT-javadoc.jar;add jar /tmp/omnituredata-1.0.2-SNAPSHOT-sources.jar;add jar /tmp/omnituredata-1.0.2-SNAPSHOT.jar; (Note: jars are available on the HDFS /tmp folder).CREATE TABLE test8(id string,desc string)ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'STORED AS INPUTFORMAT 'org.rassee.omniture.hadoop.mapred.OmnitureDataFileInputFormat'OUTPUTFORMAT '

1Nifi Custom Processor OverviewApache nifi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Its very simple to use product using which you can build "data flow" very easily. As of version 3.0 it has 90 prebuilt processors but then again you can extend them by adding your own custom processors.In this article I am going to talk about how you can build a custom nifi processor in your local machine and then move the final finished processor which is a nar file to nifi.This article is based on a video from youtube and here is the link for that https://www.youtube.com/watch?v=3ldmNFlelhw2Steps to build Customer processorHere are the steps involved to build the custom processor for nifi. I used my mac to build this processor 2.1Required Software Two software that you would need in you location machine are1.maven 2.JavaHere is how you can quickly check if you have them installed mvn -versionjava -versionHere are the results from my ma

SYMPTOM: Not able to login to Ambari UI. The Ambari ui was hung. When checked in logs we found below error -ERROR: +++++++ 16 Sep 201613:41:21,377 WARN [qtp-client-10408] ObjectGraphWalker:209 - The configuredlimit of 1,000 object references was reached while attempting to calculate thesize of the object graph. Severe performance degradation could occur if thesizing operation continues. This can be avoided by setting the CacheManger orCache <sizeOfPolicy> elements maxDepthExceededBehavior to"abort" or adding stop points with @IgnoreSizeOf annotations. Ifperformance degradation is NOT an issue at the configured limit, raise thelimit value using the CacheManager or Cache <sizeOfPolicy> elementsmaxDepth attribute. For more information, see the Ehcache configurationdocumentation. +++++++ ROOT CAUSE: As per above logs we found that the issue is with Ambari cache.This is related to JIRA - https://issues.apache.org/jira/browse/AMBARI-13517RESOLUTION: We have two wa

ObjectiveAccept a parameter -DfileType=[csv|tsv] from the Oozie command line. Use Oozie'sdecision node functionality to simulate an if-then-else conditional operation. If the value of the fileType variable equals tsv, execute a Hive 2 action which will execute theload_policestationstsv.ddl which will in-turn load a tab-separated-value filepolicestations.tsv into a Hive table named policestationstsv. Else, if the value of thefileType variable equals csv, execute a Hive 2 action which will execute theload_policestationscsv.ddl which will in-turn load a comma-separated-value filepolicestations.csv into a Hive table named policestationscsv. These Hive 2 actions will drop any pre-existing policestationstsv or policestationscsv tables from Hive as a preparatory step each time this workflow is run.ProcedureOn an edge node containing the oozie client, change users to the oozie user[root@jyoung-hdp234-1 ~]# su - oozieAuthenticate to the KDC using the oozie service account kerberos keytab[oozie@

ObjectiveLoop over a list of primary crime types ("THEFT", "STALKING", "GAMBLING", "DOMESTIC VIOLENCE"). For each crime type, query hive, get the sum of that primary crime type from the crime table, insert the sum into a new table (crimespertype).This demo uses the oozieloop project written by Jeremy Beard in order to emulate looping in Oozie via sub-workflows. Please see the oozieloop project home page for a detailed explanation of how it works: https://github.com/jeremybeard/oozieloopIMPORTANT!: This demo requires the crime database table existing in Hive as created in the first exercise: Hive-2-Action-in-a-Kerberized-clusterProcedureOn an edge node containing the oozie client, install git if it doesn't already exist[root@jyoung-hdp234-1 ~]# yum install git-allChange users to the oozie user[root@jyoung-hdp234-1 ~]# su - oozieAuthenticate to the KDC using the oozie service account kerberos keytab[oozie@jyoung-hdp234-1 ~]$ kinit -kt /etc/security/keytabs/oozie.service.keytab oozie/jyou

Here I have created a series of demos to highlight workflows that cover examples of using:Hive 2 Action in a Kerberized clusterJeremy Beard's oozieloop project to simulate looping with sub-workflowsDecision nodes to simulate conditional operators (if-then-else)These workflow demo hands-on wikis and source code are hosted on my GitHub repository at: https://github.com/jlyoung/advancedoozieworkflows

SYMPTOMS:HDP running has recently been upgraded to Isilon 8.0.1.0 and kerberized.When trying to start services through Ambari, user receives the following stderr:Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 289, in <module> Resourcemanager().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 124, in start self.wait_for_dfs_directories_created(params.entity_groupfs_store_dir, params.entity_groupfs_active_dir) File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 246, in wait_for_dfs_directories_created self.wait_for_dfs_directory_created(dir_path, ignored_dfs_dirs) File "/usr/lib/python2.6/site-packages/resource_management/libraries/

SYMPTOMS: Starting Falcon server from Ambari is successful, but when user tries to access Falcon web UI. Logs show one or more cluster entities trying to initialize: INFO - [main:] ~ Initializing FS: hdfs://fc1.support.com:8020 for cluster: falconcluster1INFO - [main:] ~ Initializing FS: hdfs://ft2.support.com:8020 for cluster: falcontest2 Start up is delayed with numerous attempts to contact cluster entity. 2016-12-05 13:49:17,240 INFO - [main:] ~ Retrying connect to server: ft2.support.com/xxx.xxx.xxx.xx:8020. Already tried 0 time(s); maxRetries=45 (Client:835)........2016-12-05 14:19:52,971 INFO - [main:] ~ Retrying connect to server: ft2.support.com/xxx.xxx.xxx.xx:8020. Already tried 44 time(s); maxRetries=45 (Client:835)2016-12-05 14:20:13,000 ERROR - [main:] ~ Failed to initialize FS for cluster : (SharedLibraryHostingService:200)org.apache.falcon.FalconException: Failed to initialize FS for cluster : falcontest2 at org.apache.falcon.service.SharedLibraryHostingService

HDP Environment:Secure cluster (kerberos enabled)Users managed by Active Directory (AD)SYMPTOMS:When user tries to access the Falcon UI with the following address, they are prompted to enter their username and password:http://<falcon server host>:15000/index.html?user.name=admin#/After entering the correct AD username and password, user gets this exception in the UI:Using curl to negotiate with Falcon UI URL has no issues.ROOT CAUSE: User is not using kerberos to authenticateRESOLUTION: In order to access the Falcon UI after enabling kerberos, user needs to authenticate using SPNEGO to negotiate with kerberos not with a user name and password. Each browser supports SPNEGO, but configuration is different for each browser. Safari needs no further configuration.After configuring your browser to negotiate using SPNEGO, user must kinit and can try to access the Falcon UI again.

SYMPTOM: Oozie workflow is failing with NoClassDefFoundError even though required jars are uploaded to the Oozie sharelib ROOT CAUSE: Oozie needs to know what jars are needed for a specific action. For example, if you are using the Hive action but it needs HBase jars, this needs to explicitly defined in the oozie-site.xmlRESOLUTION:Add oozie.action.sharelib.for.hive to the oozie-site.xml and list which sharelibs that are needed for this action. In the case where Hive action will use HBase:oozie.action.sharelib.for.hive = hive,hcatalog,hbase,oozie,hive2

SYMPTOMS: When using HDFS commands to change ownership of a directory on a cluster backed with Isilon, it returns the following error:[hdfs@hdp.test.cluster ~]$ hadoop fs -chown -R hduser1:hdfs /user/hduser1 chown: changing ownership of '/user/hduser1': Failed to get id rec: 2:hduser1Running "hdfs groups" command returns the following exception:[hdfs@hdp.test.cluster ~]$ hdfs groups hduser1 Exception in thread "main" org.apache.hadoop.ipc.RemoteException(java.io.IOException): bad protocol: org.apache.hadoop.tools.GetUserMappingsProtocol at org.apache.hadoop.ipc.Client.call(Client.java:1427) at org.apache.hadoop.ipc.Client.call(Client.java:1358) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy8.getGroupsForUser(Unknown Source) at org.apache.hadoop.tools.protocolPB.GetUserMappingsProtocolClientSideTranslatorPB.getGroupsForUser(GetUserMappingsProtocolClientSideTranslatorPB.java:57)at org.apache.hadoop.tools.GetGroupsBase.run(Ge

SYMPTOMS: Unable to submit Faclon due to XML validation error. In this case, the <timeout> property tag is causing the issue: ERROR: Bad Request;javax.xml.bind.UnmarshalException <br>- with linked exception: <br>[org.xml.sax.SAXParseException; lineNumber: 11; columnNumber: 14; cvc-complex-type.2.4.a: Invalid content was found starting with element 'timeout'. One of '{"uri:falcon:process:0.1":sla, "uri:falcon:process:0.1":timezone, "uri:falcon:process:0.1":inputs, "uri:falcon:process:0.1":outputs, "uri:falcon:process:0.1":properties, "uri:falcon:process:0.1":workflow}' is expected.] ROOT CAUSE: Property tags in Falcon process xml need to be in the correct order. RESOLUTION:The tag needs to follow the correct order in the xml schema. This order is the same as the listed properties in the Falcon documentation: Falcon Entity Specification For example, the <timeout> tag must be after order and before frequency in the xml

SYMPTOM: Ambari upgrade command gives error as Error Code: 1005. Can't create table 'ambaridb.#sql-4168_34d2' (errno: 150) ERROR:CREATE TABLE blueprint_setting (id BIGINT NOT NULL, blueprint_name VARCHAR(255) NOT NULL, setting_name VARCHAR(255) NOT NULL, setting_data LONGTEXT NOT NULL)03 Oct 2016 09:30:06,326 INFO [main] DBAccessorImpl:824 - Executing query: ALTER TABLE blueprint_setting ADD CONSTRAINT PK_blueprint_setting PRIMARY KEY (id) 03 Oct 2016 09:30:06,388 INFO [main] DBAccessorImpl:824 - Executing query: ALTER TABLE blueprint_setting ADD CONSTRAINT UQ_blueprint_setting_name UNIQUE (blueprint_name, setting_name) 03 Oct 2016 09:30:06,489 INFO [main] DBAccessorImpl:824 - Executing query: ALTER TABLE blueprint_setting ADD CONSTRAINT FK_blueprint_setting_name FOREIGN KEY (blueprint_name) REFERENCES blueprint (blueprint_name) 03 Oct 2016 09:30:06,545 ERROR [main] DBAccessorImpl:830 - Error executing query: ALTER TABLE blueprint_setting ADD CONSTRAINT FK_blueprint_setting_name FOREIG

ISSUE: While upgrading HDP, Ranger service failed. Below was the error -Failed to apply patch 020-datamask-policy.sql” with error “Not able to drop table ‘x_datamast_type_’def” Foreign constraints fails Error Code 1217 ERROR:ROOT CAUSE: Issue was with foreign_key_checksRESOLUTION:“SET foreign_key_checks = 0;” in mysql ranger DB and dropped the table manually and re-created and again “SET foreign_key_checks = 1;”

Issue: While performing HDP downgrade the last "Finalize Downgrade" step went successfully but 'Downgrade in Progress' is still struck on 99%. Please find the screenshot below -ERROR:ROOT CAUSE: There are few task which are in PENDING state from table host_role_commandBelow is sample output - SELECT task_id, status, event, host_id, role, role_command, command_detail, custom_command_name FROM host_role_command WHERE request_id = 858 AND status != 'COMPLETED' ORDER BY task_id DESC 8964, PENDING, 4, KAFKA_BROKER, CUSTOM_COMMAND, RESTART KAFKA/KAFKA_BROKER, RESTART 8897, PENDING, 4, KAFKA_BROKER, CUSTOM_COMMAND, STOP KAFKA/KAFKA_BROKER, STOP RESOLUTION: We need to manually move the task to COMPLETED state UPDATE host_role_command SET status = 'COMPLETED' WHERE request_id = 858 AND status = 'PENDING'

Issue: When user login no service tabs/action button were displayed in Ambari UI. The Ambari UI was displaying below notification on dashboard - "Move Master Wizard In Progress"ERROR:ROOT CAUSE: Seems user who has admin access to Ambari UI was doing some operation which was left open. The user is no more currently online.This was related to hortonworks Internal Jira -https://hortonworks.jira.com/browse/EAR-4843RESOLUTION:To get passed this problem we run, and set the UserName of the json to admin then when login as admin we were able to close the wizard and solved the problem. curl -u admin -i -H 'X-Requested-By: ambari' -X POST -d '{"wizard-data":" {\"userName\":\"admin\",\"controllerName\":\"<Controller_Name>\"} "}' http://<ambari-host>:8080/api/v1/persist Below was command I used - curl -u admin -i -H 'X-Requested-By: ambari' -X POST -d '{"wizard-data":" {\"userName\":\"admin\",\"controllerName\":\"moveMasterWizard\"} "}' http://l