Community Articles

Create a Box.com Applicationhttps://YourCompany.app.box.com/developers/services/Get your client api, client secret, developer token, use server authentication with OAuth 2.0 + JWT, Add a public key from your developer machine and server.This takes a few steps and you have to create a Private and Public key.openssl genrsa -aes256 -out private_key.pem 2048openssl rsa -pubout -in private_key.pem -out public_key.pemAnatomy of a Box.com Directoryhttps://myenterprise.app.box.com/files/0/f/26783331215/NIFITESTYou need the bolded # for accessing that directory, it is the Folder ID.Box.Com Java SDK<dependency> <groupId>com.box</groupId> <artifactId>box-java-sdk</artifactId> <version>2.1.1</version></dependency>Create a New Java Maven Applicationmvn archetype:generate -DgroupId=com.yourenterprise -DartifactId=boxapp -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=falseJava Codepackage com.dataflowdeveloper;import java.io

This first in a series of article lists 3 easy ways in which you can optimize your Spark code. This can be summed up as follows: Use ReduceByKey over GroupByKeyBe vary of ActionsGracefully Deal with Bad Quality DataUse ReduceByKey over GroupByKeyLet's look at two different ways to compute word counts, one using reduceByKey and the other using groupByKey:val words = Array("one", "two", "two", "three", "three", "three")val wordPairsRDD = sc.parallelize(words).map(word => (word, 1))val wordCountsWithReduce = wordPairsRDD .reduceByKey(_ + _) .collect()val wordCountsWithGroup = wordPairsRDD .groupByKey() .map(t => (t._1, t._2.sum)) .collect()While both of these functions will produce the correct answer, the reduceByKey example works much better on a large dataset. That's because Spark knows it can combine output with a common key on each partition before shuffling the data.With reduceByKey, the pairs on the same machine with the same key are combined (by using the lamdba function

Flume will not deliver partial events. The rollSize and rollInterval are approximate thresholds and are not hard limits. Once the threshold set by rollSize or rollInterval is met, an entire batch will be drained out to the file before it is rolled. Data should not be split up.

PROBLEM: 1. Create a source external hive table as below: CREATE EXTERNAL TABLE `casesclosed`(`number` int,`manager` string,`owner` string)ROW FORMAT DELIMITEDFIELDS TERMINATED BY ','LINES TERMINATED BY '\n'STORED AS INPUTFORMAT'org.apache.hadoop.mapred.TextInputFormat'OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'LOCATION'hdfs://sumeshhdp/tmp/casesclosed'TBLPROPERTIES ('COLUMN_STATS_ACCURATE'='true','numFiles'='1','totalSize'='3693','transient_lastDdlTime'='1478557456') 2. Create an ORC table with CTAS from the source table as below: CREATE TABLE casesclosed_modSTORED AS ORC tblproperties("orc.compress"="ZLIB", "orc.compress.size"="8192")ASSELECTcast(number as int) as number,cast(manager as varchar(40)) as manager,cast(owner as varchar(40)) as ownerFROM cases closed;3. On creating the Spark DataFrame against both non-orc table ( source ) and the orc table, we are unable to list out the column names in the ORC table :scala> val df = sqlContext.table("de

Check auth_to_local rules for Atlas.In many installation there exist invalid rule like below:RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*//After removing above rule Atlas UI will work fine.

Issue:After enabling Kerberos with option 'Manage Kerberos principals and keytabs manually', adding a new component will not respond. This is due to a bug in Ambari 2.2. Refer link. Below is the error message reported on the web consoleError: http://{ambari-server}:8080/api/v1/clusters/{cluster_name}/services/KERBEROS?fields=Services/attributes/kdc_validation_result,Services/attributes/kdc_validation_failure_details&_=1446149280683Workaround:Add KERBEROS service via API as below and try installing the component and then remove the KERBEROS service1. Run the below curl command on the ambari-server (Enable admin user if disabled) curl -H "X-Requested-By:ambari" -u admin:admin -i -X POST http://<ambari-server-host>:8080/api/v1/clusters/<cluster-name>/services/KERBEROS 2. Add the new component which was failing before with the above error using Add service wizard3. Once the installation is complete you can remove the service Kerberos using the below curl commandcurl -H "X-R

SYMPTOM: Knox can get LDAP user but can't find related groups. Our LDAP is an openldap (REDHAT). The membership attribute is defined in groups with "uniquemember" ERROR: 2016-05-09 14:42:01,229 INFO hadoop.gateway (KnoxLdapRealm.java:getUserDn(556)) - Computed userDn: uid=a196011,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: a196011 2016-05-09 14:42:01,230 INFO hadoop.gateway (KnoxLdapRealm.java:doGetAuthenticationInfo(180)) - Could not login: org.apache.shiro.authc.UsernamePasswordToken - a196xxx, rememberMe=false (10.xxx.xx.64) 2016-05-09 14:42:01,230 DEBUG hadoop.gateway (KnoxLdapRealm.java:doGetAuthenticationInfo(181)) - Failed to Authenticate with LDAP server: {1} org.apache.shiro.authc.AuthenticationException: LDAP naming error while attempting to authenticate user. at org.apache.shiro.realm.ldap.JndiLdapRealm.doGetAuthenticationInfo(JndiLdapRealm.java:303) The above initial error was wrt ldap misconfiguration. Correcting ldap configuration below was the

PROBLEM: Some users may be associated to many groups causing a very long list of groups to be passed through the Rest APIs headers in Ranger and KMS. ERROR: error log from /var/log/ranger/kms/kms.log2016-12-01 14:04:12,048 INFO Http11Processor - Error parsing HTTP request header Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level. java.lang.IllegalArgumentException: Request header is too large at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:515) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504) at org.apache.coyote.http11.InternalInputBuffer.parseHeader(InternalInputBuffer.java:396) at org.apache.coyote.http11.InternalInputBuffer.parseHeaders(InternalInputBuffer.java:271) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1007) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.n

SYMPTOMS: If MaxApplications=1000 set in configuration than jobs would start getting stuck in "accepted" state after 500+ submissions. The following error is seen in resource manager logs:caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1452018403088_0506 to YARN : org.apache.hadoop.security.AccessControlException: Queue root.hive1 already has 1000 applications, cannot accept submission of application: application_1452018403088_0506 at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:271) at org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:291) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:290) ... 18 moreROOT CAUSE: This a known issue reported in internal jira BUG-50642. As of date this issue is unresolved. However in Hadoop 2.8.0 this behavior is not seen.WORKAROUND: Please set yarn.scheduler.capacity.root.ordering-p

Static views or instances are created by default in Ambari. For smartsense the default view is non-editable, so you may end up creating additional views in such situations. Using the below curl command you can change the hostname of the hst server on the static instance. curl -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{ "ViewInstanceInfo" : { "properties" : { "hst.server.url" : "<hst host>:9000"} } }’ http://<host>:<port>api/v1/views/HORTONWORKS_SMARTSENSE/versions/<version no>/instances/SmartSense

Question: Is it ok to run purge scripts on WF_JOBS & COORD_JOBS tables that are in oozie Database configured in MySQL? Will the purge scripts remove the running workflows and coordinators?Below are the scripts we will be running to purge -DELETE FROM WF_ACTIONS where WF_ID IN (SELECT ID from WF_JOBS where end_time < timestamp('2016-06-01 00:00:00')); DELETE from wf_jobs where end_time < timestamp('2016-06-01 00:00:00'); DELETE from COORD_ACTIONS where JOB_ID in (select ID from COORD_JOBS where END_TIME < timestamp('2016-06-01 00:00:00')); DELETE from coord_jobs where END_TIME < timestamp('2016-06-01 00:00:00'); Reply: Oozie has feature to purge older jobs from database. by default it's 30 days. Actions related to long running coordinators do not purged until co-ordinator completes, ( example - if you have coordinator running for 6 months, then all the related workflows will be there in database for 6 months ) Will the purge scripts remove the running workflows and coord

Introduction h2o is a package for running H2O via its REST API from within R. This package allows the user to run basic H2O commands using R commands. No actual data is stored in the R workspace; and no actual work is carried out by R. R only saves the named objects, which uniquely identify the data set, model, etc. on the server. When the user makes a request, R queries the server via the REST API, which returns a JSON file with the relevant information that R then displays in the console. Scope I tested this installation guide on CentOS 7.2, but it should work on similar RedHat/Fedora/Centos… Steps 1. Install R sudo yum install R 2. Install Java https://www.java.com/en/download/help/linux_x64rpm_install.xml 3. Start R and install dependencies install.packages(RCurl)install.packages(bitops)install.packages(rjson)install.packages(statmod)install.packages(tools) 4. Install h20 package and load library for use install.packages("h2o").library(h2o) If this is your first time usin

PROBLEM STATEMENT: After finishing upgrade of all Hadoop components, I issued su - hdfs -c "hdfs dfsadmin -finalizeUpgrade" on active Namenode. However, NN UI shows all is good, until I failover to standby and standby becomes active - then again I got message that I should finalize my upgrade. ERROR: 2015-10-27 18:16:04,954 - Task. Type: EXECUTE, Script: scripts/namenode.py - Function: prepare_non_rolling_upgrade2015-10-27 18:16:05,044 - Preparing the NameNodes for a NonRolling (aka Express) Upgrade.2015-10-27 18:16:05,045 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://ambari.apache.org:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpKcaPrw 2>/tmp/tmpPjCQIr''] {'quiet': False}2015-10-27 18:16:05,068 - checked_call returned (0, '')2015-10-27 18:16:05,069 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://ambari.apache.org:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpPIEy

SYMPTOM: Knox logs are filling up disk spaceROOT CAUSE: Kerberos debug is turned on by default causing the gateway.out file to grow rapidly.RESOLUTION: To turn off kerberos debug logging. 1. Go to Ambari. KNOX -> Configs-> Advanced gateway-site 2. Change parameter sun.security.krb5.debug from true to false. 3. Restart Knox.

SYMPTOM: Hbase is giving an error KeyValue size too large when inserting large data sizes.ERROR: 0java.lang.IllegalArgumentException: KeyValue size too large at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:1521) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.validatePut(BufferedMutatorImpl.java:147) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.doMutate(BufferedMutatorImpl.java:134) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:105) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1050) at org.apache.hadoop.hbase.rest.RowResource.update(RowResource.java:229) at org.apache.hadoop.hbase.rest.RowResource.put(RowResource.java:318) at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInv

Problem: Time to live for collection data was never set on Solr Cloud server and caused the disk and nodes to fill up with too many documents due to ranger audit. Solution: 1. Delete the collection through Solr api because the ranger archive was stored in HDFS anyways. (The following cleared up disk space) http://<solr_host>:<solr_port>/solr/admin/collections?action=DELETE&name=collection 2. Set the configuration in solr to have a time to live. a. Download each of the following configs from zookeeper: schema.xml, solrconfig.xml, and managed-schema.xml /opt/hostname-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost <zookeeper host>:<zookeeper port> -cmd get /ranger_audits/configs/ranger_audits/solrconfig.xml >/tmp/solrconfig.xml /opt/hostname-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost <zookeeper host>:<zookeeper port> -cmd get /ranger_audits/configs/ranger_audits/schema.xml >/tmp/schema.xml /opt/hostname-hdpsear

SYMPTOM: After Upgrading ambari from 1.7.0 to ambari 2.2.1.1 there are lots of alerts with respect to HIVE ALERTS Example: ExecuteTimeoutException: Execution of 'ambari-sudo.sh su ambari-qa -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/sbin/:/usr/lib/hive/bin'"'"' ; export HIVE_CONF_DIR='"'"'/etc/hive/conf.server'"'"' ; hive --hiveconf hive.metastore.uris=thrift://host1:9083 --hiveconf hive.metastore.client.connect.retry.delay=1 --hiveconf hive.metastore.failure.retries=1 --hiveconf hive.metastore.connect.retries=1 --hiveconf hive.metastore.client.socket.timeout=14 --hiveconf hive.execution.engine=mr -e '"'"'show databases;'"'"''' was killed due timeout after 60 seconds ) 2016-05-11 03:25:04,779 [CRITICAL] [HIVE] [hive_server_process] (HiveServer2 Process) Connection failed on host host1:10000 (Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIV

Problem Statement: Customer has incorporated the use of ACL's within HDFS to control authorisation on the directories and files. They have also changed the fs.permissions.umask-mode = 007 in Ambari under Advanced settings (hdfs-site.xml) file.The ACL's seems to be working correctly when making directories using the hadoop fs -mkdir command. However, when making a directory using the Hue File Browser, the umask permissions are not being set properly according the the umask property set.Using the hadoop fs -mkdir command, folders are being created with a group mask:rwx, files with a group mask:rw-. Through the Hue file browser, folder group mask:r-x, file group: r-x.There seems to be a discrepancy between the mask properties set on folders and files when using hadoop fs mkdir command vs Hue file browser make directory and file command.Why does this discrepency exist? What do customer need to do in order to enforce Hue to follow the same umask and acl permissions set that the hadoop fs co

SYMPTOM: Cluster was upgraded to 2.3 After the upgrade oozie has configuration issues. User has workflow defined to create job files in directory /tmp/hadoop-${user.name}/job_details but instead, the directory is getting created in / and permission denied error is thrown ERROR: Sample workflow:<workflow-app xmlns='uri:oozie:workflow:0.5' name='scisit_all_oozie_workflow'> <global> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <job-xml>${runtime}/runtime_params.xml</job-xml> <job-xml>scisit_all_tables_config.xml</job-xml> <job-xml>ColumnTransformationRules.xml</job-xml> <job-xml>HeadersAndTrailers.xml</job-xml> <configuration> <property> <name>oozie.use.system.libpath</name> <value>true</value> </property> <property> <name>oozie.action.sharelib.for.java</name> <value>hive</value&gt

SYMPTOM: A create table in a cluster enforcing authorization using Ranger, fails to create the table and post that HiveServer2 process crashes. Create table fails as follows 0: jdbc:hive2://xxxx.hk.example.com> CREATE EXTERNAL TABLE TMP_HIVE2PHOENIX_E32E8 (CUSTOMER_ID STRING, ACCOUNT_ID STRING, ROLE_ID STRING, ROLE_NAME STRING, START_DATE STRING, END_DATE STRING, PRIORITY STRING, ACTIVE_ACCOUNT_ROLE STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' STORED AS TEXTFILE LOCATION '/tmp/example' TBLPROPERTIES ('serialization.null.format'=''); Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0) Following errors are observed in hiveserver2.log: 2016-11-15 11:42:06,721 WARN [HiveServer2-Handler-Pool: Thread-32350]: thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(492)) - Error executing statement:org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: HiveAccessControlException Permission den

SYMPTOM: - Enabled Ranger Kafka plugin via Ambari and restarted kafka service.- kafka logs still populating with "Ranger Plugin returned null" error - check ranger logs and could not see any info about kafka policy download - checked /etc/ranger/test_kafka/policycache/ and the json file in that is empty! bash-4.1# cd /etc/ranger/test_kafka/ bash-4.1# cd policycache/ bash-4.1# ls -ltr total 0 -rw-r--r-- 1 kafka hadoop 0 Mar 2 16:00 kafka_test_kafka.json_old -rw-r--r-- 1 kafka hadoop 0 Mar 16 11:30 kafka_test_kafka.json_old1 -rw-r--r-- 1 kafka hadoop 0 Mar 16 12:27 kafka_test_kafka.json - checked Test Connection for Kafka repo in Ranger. It was successful. - The Ranger plugin audits, did not have info on the kafka plugin sync. - Thus the kafka plugin is not being synced in this case. Policy refresh not working. - Tried deleting the default kafka policy and created a new one however issue still exists. - Tried to use REST API to get the policy details however no output. ERROR:2016-0

SYMPTOM: All the services in the cluster are down and restarting the services fails with the following error: 2016-11-17 21:42:18,235 ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode. java.io.IOException: Login failure for nn/lxxx.examplet.ex.com@EXAMPLE.AD.EX.COM from keytab /etc/security/keytabs/nn.service.keytab: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) ...Caused by: KrbException: Client not found in Kerberos database (6) ...Caused by: KrbException: Identifier doesn't match expected value (906) Regeneration of Keytabs using Ambari too failed as follows: 17 Nov 2016 23:58:59,136 WARN [Server Action Executor Worker 12702] CreatePrincipalsServerAction:233 - Principal, HTTP/xxx.examplet.ex.com@EXAMPLE.AD.EX.COM, does not exist, creating new principal 17 Nov 2016 23:58:59,151 ERROR [Server Action Executor Worker 12702] CreatePrincipalsServerAction:284 - Failed to create or update principal, HTTP/xxx.examplet.ex.com@E

SYMPTOMS: Due to both Resource manager getting active simultaneously , all node managers crash. Errors visible in RM logs are as follows:2015-06-27 20:08:35,922 DEBUG [main] service.AbstractService (AbstractService.java:enterState(452)) - Service: Dispatcher entered state STOPPED2015-06-27 20:08:35,923 WARN [AsyncDispatcher event handler] event.AsyncDispatcher (AsyncDispatcher.java:handle(247)) - AsyncDispatcher thread interruptedjava.lang.InterruptedExceptionat java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)atjava.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)at org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:244)at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.updateApplicationAttemptState(RMStateStore.java:652)at org.apache.hadoop.yarn.server.res

SYMPTOMSometimes while performing any Ambari operation like for example, adding a Ranger KMS service or doing any other operation, the Ambari UI (or even curt response) might show this error:ERROR:Error 500 status code received on GET method for API: /api/v1/stacks/HDP/versions/2.3/recommendationsError message: Error occurred during stack advisor command invocation: Cannot create /var/run/ambari-server/stack-recommendationsROOT CAUSE This is most likely a permission issue with /var/run/ambari-server/ location for the user who is running the 'ambari-server' process.RESOLUTION To resolve this, the permission for /var/run/ambari/server should be setup correctly. On the good cluster, where the ambari-server is running as 'root' the permission looks like: [root@test ~]# ll -d /var/run/ambari-server/drwxr-xr-x 4 root root 4096 Dec 4 05:10 /var/run/ambari-server/[root@test ~]# ll /var/run/ambari-server/total 12-rw-r--r-- 1 root root 6 Dec 4 05:10 ambari-server.piddrwxr-xr-x 4 root root 4

PROBLEM: After upgrading from Ambari from v2.1.2.1 to v2.2.2.0 when attempting to "Re-install" Grafana nothing happened. No task was started and the Ambari UI seemed to hang.ERROR: The following error messages were found in the Ambari server log file08 Jul 2016 10:28:38,174 ERROR [qtp-ambari-client-2410] ClusterImpl:2347 - Config inconsistency exists: unknown configType=kerberos-env 08 Jul 2016 10:28:38,174 ERROR [qtp-ambari-client-2410] ClusterImpl:2347 - Config inconsistency exists: unknown configType=krb5-conf 08 Jul 2016 10:28:38,174 ERROR [qtp-ambari-client-2410] ClusterImpl:2347 - Config inconsistency exists: unknown configType=ranger-ugsync-site 08 Jul 2016 10:28:38,174 ERROR [qtp-ambari-client-2410] ClusterImpl:2347 - Config inconsistency exists: unknown configType=admin-properties 08 Jul 2016 10:28:38,174 ERROR [qtp-ambari-client-2410] ClusterImpl:2347 - Config inconsistency exists: unknown configType=usersync-properties 08 Jul 2016 10:28:38,175 ERROR [qtp-ambari-client-2410]