Resources

SYMPTOM: Oozie workflow is failing with NoClassDefFoundError even though required jars are uploaded to the Oozie sharelib ROOT CAUSE: Oozie needs to know what jars are needed for a specific action. For example, if you are using the Hive action but it needs HBase jars, this needs to explicitly defined in the oozie-site.xmlRESOLUTION:Add oozie.action.sharelib.for.hive to the oozie-site.xml and list which sharelibs that are needed for this action. In the case where Hive action will use HBase:oozie.action.sharelib.for.hive = hive,hcatalog,hbase,oozie,hive2

SYMPTOMS: When using HDFS commands to change ownership of a directory on a cluster backed with Isilon, it returns the following error:[hdfs@hdp.test.cluster ~]$ hadoop fs -chown -R hduser1:hdfs /user/hduser1 chown: changing ownership of '/user/hduser1': Failed to get id rec: 2:hduser1Running "hdfs groups" command returns the following exception:[hdfs@hdp.test.cluster ~]$ hdfs groups hduser1 Exception in thread "main" org.apache.hadoop.ipc.RemoteException(java.io.IOException): bad protocol: org.apache.hadoop.tools.GetUserMappingsProtocol at org.apache.hadoop.ipc.Client.call(Client.java:1427) at org.apache.hadoop.ipc.Client.call(Client.java:1358) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy8.getGroupsForUser(Unknown Source) at org.apache.hadoop.tools.protocolPB.GetUserMappingsProtocolClientSideTranslatorPB.getGroupsForUser(GetUserMappingsProtocolClientSideTranslatorPB.java:57)at org.apache.hadoop.tools.GetGroupsBase.run(Ge

SYMPTOMS: Unable to submit Faclon due to XML validation error. In this case, the <timeout> property tag is causing the issue: ERROR: Bad Request;javax.xml.bind.UnmarshalException <br>- with linked exception: <br>[org.xml.sax.SAXParseException; lineNumber: 11; columnNumber: 14; cvc-complex-type.2.4.a: Invalid content was found starting with element 'timeout'. One of '{"uri:falcon:process:0.1":sla, "uri:falcon:process:0.1":timezone, "uri:falcon:process:0.1":inputs, "uri:falcon:process:0.1":outputs, "uri:falcon:process:0.1":properties, "uri:falcon:process:0.1":workflow}' is expected.] ROOT CAUSE: Property tags in Falcon process xml need to be in the correct order. RESOLUTION:The tag needs to follow the correct order in the xml schema. This order is the same as the listed properties in the Falcon documentation: Falcon Entity Specification For example, the <timeout> tag must be after order and before frequency in the xml

SYMPTOM: Ambari upgrade command gives error as Error Code: 1005. Can't create table 'ambaridb.#sql-4168_34d2' (errno: 150) ERROR:CREATE TABLE blueprint_setting (id BIGINT NOT NULL, blueprint_name VARCHAR(255) NOT NULL, setting_name VARCHAR(255) NOT NULL, setting_data LONGTEXT NOT NULL)03 Oct 2016 09:30:06,326 INFO [main] DBAccessorImpl:824 - Executing query: ALTER TABLE blueprint_setting ADD CONSTRAINT PK_blueprint_setting PRIMARY KEY (id) 03 Oct 2016 09:30:06,388 INFO [main] DBAccessorImpl:824 - Executing query: ALTER TABLE blueprint_setting ADD CONSTRAINT UQ_blueprint_setting_name UNIQUE (blueprint_name, setting_name) 03 Oct 2016 09:30:06,489 INFO [main] DBAccessorImpl:824 - Executing query: ALTER TABLE blueprint_setting ADD CONSTRAINT FK_blueprint_setting_name FOREIGN KEY (blueprint_name) REFERENCES blueprint (blueprint_name) 03 Oct 2016 09:30:06,545 ERROR [main] DBAccessorImpl:830 - Error executing query: ALTER TABLE blueprint_setting ADD CONSTRAINT FK_blueprint_setting_name FOREIG

ISSUE: While upgrading HDP, Ranger service failed. Below was the error -Failed to apply patch 020-datamask-policy.sql” with error “Not able to drop table ‘x_datamast_type_’def” Foreign constraints fails Error Code 1217 ERROR:ROOT CAUSE: Issue was with foreign_key_checksRESOLUTION:“SET foreign_key_checks = 0;” in mysql ranger DB and dropped the table manually and re-created and again “SET foreign_key_checks = 1;”

Issue: While performing HDP downgrade the last "Finalize Downgrade" step went successfully but 'Downgrade in Progress' is still struck on 99%. Please find the screenshot below -ERROR:ROOT CAUSE: There are few task which are in PENDING state from table host_role_commandBelow is sample output - SELECT task_id, status, event, host_id, role, role_command, command_detail, custom_command_name FROM host_role_command WHERE request_id = 858 AND status != 'COMPLETED' ORDER BY task_id DESC 8964, PENDING, 4, KAFKA_BROKER, CUSTOM_COMMAND, RESTART KAFKA/KAFKA_BROKER, RESTART 8897, PENDING, 4, KAFKA_BROKER, CUSTOM_COMMAND, STOP KAFKA/KAFKA_BROKER, STOP RESOLUTION: We need to manually move the task to COMPLETED state UPDATE host_role_command SET status = 'COMPLETED' WHERE request_id = 858 AND status = 'PENDING'

Issue: When user login no service tabs/action button were displayed in Ambari UI. The Ambari UI was displaying below notification on dashboard - "Move Master Wizard In Progress"ERROR:ROOT CAUSE: Seems user who has admin access to Ambari UI was doing some operation which was left open. The user is no more currently online.This was related to hortonworks Internal Jira -https://hortonworks.jira.com/browse/EAR-4843RESOLUTION:To get passed this problem we run, and set the UserName of the json to admin then when login as admin we were able to close the wizard and solved the problem. curl -u admin -i -H 'X-Requested-By: ambari' -X POST -d '{"wizard-data":" {\"userName\":\"admin\",\"controllerName\":\"<Controller_Name>\"} "}' http://<ambari-host>:8080/api/v1/persist Below was command I used - curl -u admin -i -H 'X-Requested-By: ambari' -X POST -d '{"wizard-data":" {\"userName\":\"admin\",\"controllerName\":\"moveMasterWizard\"} "}' http://l

Create a Box.com Applicationhttps://YourCompany.app.box.com/developers/services/Get your client api, client secret, developer token, use server authentication with OAuth 2.0 + JWT, Add a public key from your developer machine and server.This takes a few steps and you have to create a Private and Public key.openssl genrsa -aes256 -out private_key.pem 2048openssl rsa -pubout -in private_key.pem -out public_key.pemAnatomy of a Box.com Directoryhttps://myenterprise.app.box.com/files/0/f/26783331215/NIFITESTYou need the bolded # for accessing that directory, it is the Folder ID.Box.Com Java SDK<dependency> <groupId>com.box</groupId> <artifactId>box-java-sdk</artifactId> <version>2.1.1</version></dependency>Create a New Java Maven Applicationmvn archetype:generate -DgroupId=com.yourenterprise -DartifactId=boxapp -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=falseJava Codepackage com.dataflowdeveloper;import java.io

This first in a series of article lists 3 easy ways in which you can optimize your Spark code. This can be summed up as follows: Use ReduceByKey over GroupByKeyBe vary of ActionsGracefully Deal with Bad Quality DataUse ReduceByKey over GroupByKeyLet's look at two different ways to compute word counts, one using reduceByKey and the other using groupByKey:val words = Array("one", "two", "two", "three", "three", "three")val wordPairsRDD = sc.parallelize(words).map(word => (word, 1))val wordCountsWithReduce = wordPairsRDD .reduceByKey(_ + _) .collect()val wordCountsWithGroup = wordPairsRDD .groupByKey() .map(t => (t._1, t._2.sum)) .collect()While both of these functions will produce the correct answer, the reduceByKey example works much better on a large dataset. That's because Spark knows it can combine output with a common key on each partition before shuffling the data.With reduceByKey, the pairs on the same machine with the same key are combined (by using the lamdba function

Flume will not deliver partial events. The rollSize and rollInterval are approximate thresholds and are not hard limits. Once the threshold set by rollSize or rollInterval is met, an entire batch will be drained out to the file before it is rolled. Data should not be split up.

PROBLEM: 1. Create a source external hive table as below: CREATE EXTERNAL TABLE `casesclosed`(`number` int,`manager` string,`owner` string)ROW FORMAT DELIMITEDFIELDS TERMINATED BY ','LINES TERMINATED BY '\n'STORED AS INPUTFORMAT'org.apache.hadoop.mapred.TextInputFormat'OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'LOCATION'hdfs://sumeshhdp/tmp/casesclosed'TBLPROPERTIES ('COLUMN_STATS_ACCURATE'='true','numFiles'='1','totalSize'='3693','transient_lastDdlTime'='1478557456') 2. Create an ORC table with CTAS from the source table as below: CREATE TABLE casesclosed_modSTORED AS ORC tblproperties("orc.compress"="ZLIB", "orc.compress.size"="8192")ASSELECTcast(number as int) as number,cast(manager as varchar(40)) as manager,cast(owner as varchar(40)) as ownerFROM cases closed;3. On creating the Spark DataFrame against both non-orc table ( source ) and the orc table, we are unable to list out the column names in the ORC table :scala> val df = sqlContext.table("de

Check auth_to_local rules for Atlas.In many installation there exist invalid rule like below:RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*//After removing above rule Atlas UI will work fine.

Issue:After enabling Kerberos with option 'Manage Kerberos principals and keytabs manually', adding a new component will not respond. This is due to a bug in Ambari 2.2. Refer link. Below is the error message reported on the web consoleError: http://{ambari-server}:8080/api/v1/clusters/{cluster_name}/services/KERBEROS?fields=Services/attributes/kdc_validation_result,Services/attributes/kdc_validation_failure_details&_=1446149280683Workaround:Add KERBEROS service via API as below and try installing the component and then remove the KERBEROS service1. Run the below curl command on the ambari-server (Enable admin user if disabled) curl -H "X-Requested-By:ambari" -u admin:admin -i -X POST http://<ambari-server-host>:8080/api/v1/clusters/<cluster-name>/services/KERBEROS 2. Add the new component which was failing before with the above error using Add service wizard3. Once the installation is complete you can remove the service Kerberos using the below curl commandcurl -H "X-R

SYMPTOM: Knox can get LDAP user but can't find related groups. Our LDAP is an openldap (REDHAT). The membership attribute is defined in groups with "uniquemember" ERROR: 2016-05-09 14:42:01,229 INFO hadoop.gateway (KnoxLdapRealm.java:getUserDn(556)) - Computed userDn: uid=a196011,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: a196011 2016-05-09 14:42:01,230 INFO hadoop.gateway (KnoxLdapRealm.java:doGetAuthenticationInfo(180)) - Could not login: org.apache.shiro.authc.UsernamePasswordToken - a196xxx, rememberMe=false (10.xxx.xx.64) 2016-05-09 14:42:01,230 DEBUG hadoop.gateway (KnoxLdapRealm.java:doGetAuthenticationInfo(181)) - Failed to Authenticate with LDAP server: {1} org.apache.shiro.authc.AuthenticationException: LDAP naming error while attempting to authenticate user. at org.apache.shiro.realm.ldap.JndiLdapRealm.doGetAuthenticationInfo(JndiLdapRealm.java:303) The above initial error was wrt ldap misconfiguration. Correcting ldap configuration below was the

PROBLEM: Some users may be associated to many groups causing a very long list of groups to be passed through the Rest APIs headers in Ranger and KMS. ERROR: error log from /var/log/ranger/kms/kms.log2016-12-01 14:04:12,048 INFO Http11Processor - Error parsing HTTP request header Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level. java.lang.IllegalArgumentException: Request header is too large at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:515) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504) at org.apache.coyote.http11.InternalInputBuffer.parseHeader(InternalInputBuffer.java:396) at org.apache.coyote.http11.InternalInputBuffer.parseHeaders(InternalInputBuffer.java:271) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1007) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.n

SYMPTOMS: If MaxApplications=1000 set in configuration than jobs would start getting stuck in "accepted" state after 500+ submissions. The following error is seen in resource manager logs:caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1452018403088_0506 to YARN : org.apache.hadoop.security.AccessControlException: Queue root.hive1 already has 1000 applications, cannot accept submission of application: application_1452018403088_0506 at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:271) at org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:291) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:290) ... 18 moreROOT CAUSE: This a known issue reported in internal jira BUG-50642. As of date this issue is unresolved. However in Hadoop 2.8.0 this behavior is not seen.WORKAROUND: Please set yarn.scheduler.capacity.root.ordering-p

Static views or instances are created by default in Ambari. For smartsense the default view is non-editable, so you may end up creating additional views in such situations. Using the below curl command you can change the hostname of the hst server on the static instance. curl -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{ "ViewInstanceInfo" : { "properties" : { "hst.server.url" : "<hst host>:9000"} } }’ http://<host>:<port>api/v1/views/HORTONWORKS_SMARTSENSE/versions/<version no>/instances/SmartSense

Question: Is it ok to run purge scripts on WF_JOBS & COORD_JOBS tables that are in oozie Database configured in MySQL? Will the purge scripts remove the running workflows and coordinators?Below are the scripts we will be running to purge -DELETE FROM WF_ACTIONS where WF_ID IN (SELECT ID from WF_JOBS where end_time < timestamp('2016-06-01 00:00:00')); DELETE from wf_jobs where end_time < timestamp('2016-06-01 00:00:00'); DELETE from COORD_ACTIONS where JOB_ID in (select ID from COORD_JOBS where END_TIME < timestamp('2016-06-01 00:00:00')); DELETE from coord_jobs where END_TIME < timestamp('2016-06-01 00:00:00'); Reply: Oozie has feature to purge older jobs from database. by default it's 30 days. Actions related to long running coordinators do not purged until co-ordinator completes, ( example - if you have coordinator running for 6 months, then all the related workflows will be there in database for 6 months ) Will the purge scripts remove the running workflows and coord

Introduction h2o is a package for running H2O via its REST API from within R. This package allows the user to run basic H2O commands using R commands. No actual data is stored in the R workspace; and no actual work is carried out by R. R only saves the named objects, which uniquely identify the data set, model, etc. on the server. When the user makes a request, R queries the server via the REST API, which returns a JSON file with the relevant information that R then displays in the console. Scope I tested this installation guide on CentOS 7.2, but it should work on similar RedHat/Fedora/Centos… Steps 1. Install R sudo yum install R 2. Install Java https://www.java.com/en/download/help/linux_x64rpm_install.xml 3. Start R and install dependencies install.packages(RCurl)install.packages(bitops)install.packages(rjson)install.packages(statmod)install.packages(tools) 4. Install h20 package and load library for use install.packages("h2o").library(h2o) If this is your first time usin

PROBLEM STATEMENT: After finishing upgrade of all Hadoop components, I issued su - hdfs -c "hdfs dfsadmin -finalizeUpgrade" on active Namenode. However, NN UI shows all is good, until I failover to standby and standby becomes active - then again I got message that I should finalize my upgrade. ERROR: 2015-10-27 18:16:04,954 - Task. Type: EXECUTE, Script: scripts/namenode.py - Function: prepare_non_rolling_upgrade2015-10-27 18:16:05,044 - Preparing the NameNodes for a NonRolling (aka Express) Upgrade.2015-10-27 18:16:05,045 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://ambari.apache.org:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpKcaPrw 2>/tmp/tmpPjCQIr''] {'quiet': False}2015-10-27 18:16:05,068 - checked_call returned (0, '')2015-10-27 18:16:05,069 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://ambari.apache.org:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpPIEy

SYMPTOM: Knox logs are filling up disk spaceROOT CAUSE: Kerberos debug is turned on by default causing the gateway.out file to grow rapidly.RESOLUTION: To turn off kerberos debug logging. 1. Go to Ambari. KNOX -> Configs-> Advanced gateway-site 2. Change parameter sun.security.krb5.debug from true to false. 3. Restart Knox.

SYMPTOM: Hbase is giving an error KeyValue size too large when inserting large data sizes.ERROR: 0java.lang.IllegalArgumentException: KeyValue size too large at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:1521) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.validatePut(BufferedMutatorImpl.java:147) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.doMutate(BufferedMutatorImpl.java:134) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:105) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1050) at org.apache.hadoop.hbase.rest.RowResource.update(RowResource.java:229) at org.apache.hadoop.hbase.rest.RowResource.put(RowResource.java:318) at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInv

Problem: Time to live for collection data was never set on Solr Cloud server and caused the disk and nodes to fill up with too many documents due to ranger audit. Solution: 1. Delete the collection through Solr api because the ranger archive was stored in HDFS anyways. (The following cleared up disk space) http://<solr_host>:<solr_port>/solr/admin/collections?action=DELETE&name=collection 2. Set the configuration in solr to have a time to live. a. Download each of the following configs from zookeeper: schema.xml, solrconfig.xml, and managed-schema.xml /opt/hostname-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost <zookeeper host>:<zookeeper port> -cmd get /ranger_audits/configs/ranger_audits/solrconfig.xml >/tmp/solrconfig.xml /opt/hostname-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost <zookeeper host>:<zookeeper port> -cmd get /ranger_audits/configs/ranger_audits/schema.xml >/tmp/schema.xml /opt/hostname-hdpsear

SYMPTOM: After Upgrading ambari from 1.7.0 to ambari 2.2.1.1 there are lots of alerts with respect to HIVE ALERTS Example: ExecuteTimeoutException: Execution of 'ambari-sudo.sh su ambari-qa -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/sbin/:/usr/lib/hive/bin'"'"' ; export HIVE_CONF_DIR='"'"'/etc/hive/conf.server'"'"' ; hive --hiveconf hive.metastore.uris=thrift://host1:9083 --hiveconf hive.metastore.client.connect.retry.delay=1 --hiveconf hive.metastore.failure.retries=1 --hiveconf hive.metastore.connect.retries=1 --hiveconf hive.metastore.client.socket.timeout=14 --hiveconf hive.execution.engine=mr -e '"'"'show databases;'"'"''' was killed due timeout after 60 seconds ) 2016-05-11 03:25:04,779 [CRITICAL] [HIVE] [hive_server_process] (HiveServer2 Process) Connection failed on host host1:10000 (Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIV

Problem Statement: Customer has incorporated the use of ACL's within HDFS to control authorisation on the directories and files. They have also changed the fs.permissions.umask-mode = 007 in Ambari under Advanced settings (hdfs-site.xml) file.The ACL's seems to be working correctly when making directories using the hadoop fs -mkdir command. However, when making a directory using the Hue File Browser, the umask permissions are not being set properly according the the umask property set.Using the hadoop fs -mkdir command, folders are being created with a group mask:rwx, files with a group mask:rw-. Through the Hue file browser, folder group mask:r-x, file group: r-x.There seems to be a discrepancy between the mask properties set on folders and files when using hadoop fs mkdir command vs Hue file browser make directory and file command.Why does this discrepency exist? What do customer need to do in order to enforce Hue to follow the same umask and acl permissions set that the hadoop fs co