Community Articles

Issue:When you are trying to run queries from Tableau, SQuirrel etc.. connecting to Hiveserver2 via JDBC/ODBC driver in a Knox environment, you might intermittently come across the above error. This error pops-up especially when the query has more columns in the select statement. Interestingly, the same query works fine from hive cli and beeline cli.Resolution:This is a bug in Knox - https://issues.apache.org/jira/browse/KNOX-492Log into Ambari > Knox > Config > Add the below under "Advanced Topology" file section<service> <role>HIVE</role> <url>http://localhost:10001/cliservice</url> <param> <name>replayBufferSize</name> <value>16</value> </param></service>

SYMPTOM While adding new service to the HDP cluster, it fails with below error:Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/hook.py", line 37, in <module> AfterInstallHook().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/hook.py", line 31, in hook setup_hdp_symlinks() File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py", line 49, in setup_hdp_symlinks hdp_select.select_all(version) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/hdp_select.py", line 122, in select_all Execute(command, only_if = only_if_command) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/s

Problem Statement: Whenever HBase is running, the "hdfs fsck /" reports four hbase-related files in the path "hbase/data/WALs/" as CORRUPT. Even after letting the cluster sit idle for a couple hours, it is still in the corrupt state. If HBase is shut down, the problem goes away. If HBase is then restarted, the problem recurs.ERROR:hdfs@test1:>$ hdfs fsck /Connecting to namenode via http://test1.example.com:50070/fsck?ugi=hdfs&path=%2FFSCK started by hdfs (auth:SIMPLE) from /39.0.8.2 for path / at Wed Jun 24 20:40:17 GMT 2015.../apps/hbase/data/WALs/test2.example.com,16020,1435168292684/test2.example.com%2C16020%2C1435168292684.default.1435175500556: MISSING 1 blocks of total size 83 B./apps/hbase/data/WALs/test3.example.com,16020,1435168290466/test3.example.com%2C16020%2C1435168290466..meta.1435175562144.meta: MISSING 1 blocks of total size 83 B./apps/hbase/data/WALs/test3.example.com,16020,1435168290466/test3.example.com%2C16020%2C1435168290466.default.1435175498500: MISSING 1

[users] # To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISections admin = admin [main] ldapRealm = org.apache.zeppelin.server.LdapGroupRealm ldapRealm.contextFactory.environment[ldap.searchBase] = dc=mgmt,dc=example,dc=net ldapRealm.contextFactory.url = ldaps://me.abc.example.net:636 ldapRealm.userDnTemplate = uid={0},cn=group,dc=mgmt,dc=example,dc=netldapRealm.contextFactory.authenticationMechanism = SIMPLEldapRealm.contextFactory.systemUsername=uid=example,ou=example,dc=mgmt,dc=example,dc=netldapRealm.contextFactory.systemPassword=examplesessionManager=org.apache.shiro.web.session.mgt.DefaultWebSessionManagersecurityManager.sessionManager = $sessionManager securityManager.sessionManager.globalSessionTimeout = 86400000 shiro.loginUrl = /api/login[urls] /** = authc

hcc-58548.zipUntil recently, Hive could only read and write UTF-8 text files, and no other character sets were supported forcing people to convert their possibly huge and/or multiple input files to UTF-8 using "iconv" or other such utility which can be cumbersome (for example, iconv supports only files smaller than 16G), and time-consuming. However, since Hive-0.14, by way of LazySimpleSerDe, conversion from and to other character sets is supported. The SerDe can be specified by, for example hive> CREATE TABLE mytable(...) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES("serialization.encoding"='SJIS'); or by declaring the table in a usual way, omitting the SerDe, specifying the field delimiter, and then doing: hive> ALTER TABLE mytable SET SERDEPROPERTIES ('serialization.encoding'='SJIS'); where, instead of "SJIS" one can insert the encoding of the input file (note that char-set self-detection is not supported), like for example o

My first caveat would be that in my tests, the pre-trained models is missing a lot of names. If this is for a production work load, I would recommend training your own models using your own data. Maybe use all of your corporate directory, client list, Salesforce data, LinkedIn and social media. I would recommend full name, first names and any nicknames that are commonly used.The current version is 1.7.0 and there are pre-trained 1.5.0 models that work. They have a number of pre-trained models in a few human languages. I chose English (http://opennlp.sourceforge.net/models-1.5/en-ner-person.bin).Walk Through:Create TokenNameFinderModel from pre-built person model.Tokenize the input sentence.Find the identified people.Convert to JSON array.You can easily plug this into a custom NiFi processor, microservice, command line tool or routine in a larger Apache Storm or Apache Spark pipeline.Code (JavaBean)public class PersonName { private String name = ""; public String getName() { re

Was an interesting issue faced last week. Putting here for bigger audience, might be helpful to others too.PROBLEM On one of the node, datanode and nodemanager were not coming up. Below is the error after starting from ambari. resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start datanode'' returned 1. starting datanode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-datanode-ny-node3.hwxblr.com.outError: Could not find or load main class org.apache.hadoop.hdfs.server.datanode.DataNodeAs datanode process itself wasn't loaded, so nothing was printed in datanode logs. Only thing we see in .out file is Error: Could not find or load main class org.apache.hadoop.hdfs.server.datanode.DataNodeVerified in the jar, DataNode class was present./usr/jdk64/jdk1.8.0_77/bin/jar -tvf /usr/hdp/2.5.0.0-1245/hadoop-hdfs/hadoop-hdfs-2.7.3

Using SaltStack to run commands on HDCloud and CloudbreakHDCloud and Cloudbreak make it easy to provision, configure and elastically grow HDP clusters on cloud infrastructure. During provisioning time Cloudbreak can execute recipes on the nodes participating in the cluster. Recipes are simple script written in bash or python or any other scripting language available on the nodes.It is common that users would like to execute ad-hoc commands (e.g. collecting logs, installing extra packages, executing scripts) on nodes not only during provisioning time, but when the cluster is used.Infrastructure management of CloudbreakUnder the hood Cloudbreak uses SaltStack to manage nodes of the cluster, install packages, change configuration files and execute recipes. After provisioning phase, users can take advantage of this infrastructure management tool and execute their own scripts and create their own Salt States. Connectivity checkBy default Salt master is installed on one of the master nodes,

Below article will guide you on how to access Ranger KMS policies via rest api -1. I have Cluster with Ranger and Ranger KMS installed.2. From the documentation it is clearly given on "how to access Ranger policies using rest api". Please check the link below -https://cwiki.apache.org/confluence/display/RANGER/REST+APIs+for+Policy+Management3. Below is the example for Range rest api GET method -a.Sample Ranger Policy for HDFS repository in UI is as below -Below is the rest api GET method call which we can use to get the policy as displayed above - curl -iv -u <username>:<password> -H "Content-type:application/json" -X GET http://localhost:6080/service/public/api/repository/{id}Eg. In above screenshot my policy id is "2"curl -iv -u admin:admin -H "Content-type:application/json" -X GET http://localhost:6080/service/public/api/repository/24. Below is my Ranger KMS UI with policy -But if you try the same steps for Ranger KMS it will fail -curl -iv -u <username>:<passwo

User tries to decommission/recommission nodes from Ambari UI, nothing happens on the UI and it seems like the operation did not go through.Ambari-server Logs:WARN [C3P0PooledConnectionPoolManager[identityToken->2s8bny9j1mxgjkn9oj5d8|79679221]-HelperThread-#0] StatementUtils:223 - Statement close FAILED.com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1 at sun.reflect.GeneratedConstructorAccessor198.newInstance(Unknown Source)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.Util.getInstance(Util.java:386) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1052)ROOT CAUSE The default JDBC driver

USE CASEACCESS MODEL [Let's say in this case TERADATA] User U1 can only read all tables in Database D1 & D2User U2 can Read Database D1 and INSERT , UPDATE , DELETE , SELECT all tables in Database D2. User U2 Cannot DROP or Create Table in Database D2.User U3 can SELECT,INSERT,UPDATE,DELETE,DROP,CREATE (ALL ACCESS) on Database D1 and D2OBJECTIVE :- Want to have same model on Hadoop with one improvement.We will have Storage Groups and ACLs – grouping the tables of same subject area. One Database may have more than one storage Group. Say SG11 , SG12 and SG21 and SG22 (SG11 and SG12 are associated with database D1 and SG21 & SG22 with D2)User U1 should read all Tables in D1 and D2 .User U2 should only INSERT,UPDATE,DELETE and SELECT Tables covered by SG11 ( in Database D1) - U2 will not be able to Update tables in SG12 ( in Database D1) but can readUser U3 can do all operations on SG11,SG12 ( D1) and SG21,SG22 (D2) and is OWNER of all the objects in D1 and D2 .OU

Additional Use of Color Color has been added throughout the UI for important messaging and to highlight changes in state/status, such as whether processors are running, stopped, or invalid and whether controller services are disabled or enabled. Here are some sample screenshots: Status Bar Processors Process Groups Remote Process Groups Summary Settings Customization of Processor Color To further highlight when a processor’s color has been customized, the processor icon now appears on a square background of the selected color. Additionally, the processor itself is outlined by the custom color. If users wish to return the processor to the default color, they can select the top most left corner of the color palette: Back Pressure Indicators Processor connections now visually indicate when back pressure is enabled. Progress bars appear on the connection label for “Back Pressure Object Threshold" and "Back Pressure Data Size Threshold.”The progress bars change color

Property: tez.am.view-aclsLocation: tez-site.xml Navigation Option: Ambari > Tez > Advanced tez-site > tez-site.xmlDefault Value: Empty stringAM view will only be displayed to users who submitted the job.Description: The Property "tez.am.view-acls" allows you to control users or groups that can view Tez AM jobs. "No default value" or Empty Value indicate that only the users who submitted the job can view their job status.AM view ACLs. This setting enables the specified users or groups to view the status of the AM and all DAGs that run within the AM. Format: a comma-separated list of users, a white space, and then a comma-separated list of groups. For example, "lhale,msmith administrators,users"If you want to allow all users have visibility of all Tez jobs, set tez.am.view-acls=*Services to be restarted to apply configuration: Tez, Hive, Oozie.This Article assumes the following:1. Environment is Ambari Managed Cluster2. Tez View is enabled.

When enabling Kerberos, where Ambari is managing the Kerberos identities (principals and keytab files), the KDC administrator's credentials are needed in order to create the needed accounts in the KDC. During the process of enabling Kerberos via Ambari's web-based UI, the user is prompted for this information and has the option to store the credentials in either the temporary or persisted credential store. The temporary credential store is a keystore in memory where each entry is removed after 90 minutes (from initial creation), when Ambari is restarted, or by user request. The persisted credential store is a keystore stored on disk where each entry is removed only by user request. The option to store a credential in the persisted store is only available if Ambari's credential store has been setup. Creating Ambari's Credential StoreTo set up Ambari's credential store, the following commands must be invoked from the Ambari server host's command line:ambari-server setup-securityOpt

Apache Zeppelin is a flexible, multi-purpose notebook used for code editing and data visualization. It supports a variety of languages (using interpreters) and enables users to quickly prototype, write code, share code, and visualize the results using standard charts.But as you move towards data apps and front-end development, you may need to create custom visualizations. Fortunately, Zeppelin comes with built-in support for Angular through both a front-end API and back-end API. I recently worked on a project to identify networking issues. These issues could range from bad/weak network signals, wiring issues, down nodes, etc. To replicate this, I created an example project where I mapped out the path that a packet takes from my home wifi connection to the google.com servers (using traceroute). The complete project is here, but for this post I want to focus on the steps I took to display the Spark output within Google Maps, all within Zeppelin. Here's a couple screenshots showing the e

PROBLEM Zookeeper transaction logs and snapshot files are created very frequently (multiple files in every minute) and that fills up the FileSystem in a very short time. ROOT CAUSE One or more application are creating or modifying the znodes too frequently, causing too many transactions in a short duration. This leads to the creation of too many transactional log files and snapshot files since they get rolled over after 100,000 entries by default (as defined by zookeeper property 'snapCount') RESOLUTION The resolution for such cases involves reviewing the zookeeper transaction logs to find the znodes that are updated/created most frequently using the following command on one of the zookeeper servers: # cd /usr/hdp/current/zookeeper-server# java -cp zookeeper.jar:lib/* org.apache.zookeeper.server.LogFormatter /hadoop/zookeeper/version-2/logxxx(where 'dataDir' is set to '/hadoop/zookeeper' within zookeeper configuration)Once the frequently updating znodes are identified using the abov

SYMPTOM Namenode crash might be observed when jni based unix group mappings is enabled. The crash will usually generate the "hs_err" log file which will have the stack as below: ## A fatal error has been detected by the Java Runtime Environment:## SIGSEGV (0xb) at pc=0x00007fbc814dd2a0, pid=380582, tid=140448021370624## JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)# Problematic frame:# C [libnss_uxauth.so.2+0x4e2a0] sqlite3ExprCodeTarget+0xcc3#......Stack: [0x00007fbc9a5c5000,0x00007fbc9a6c6000], sp=0x00007fbc9a6c2860, free space=1014kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)C [libnss_uxauth.so.2+0x4e2a0] sqlite3ExprCodeTarget+0xcc3C [libnss_uxauth.so.2+0x4e8db] evalConstExpr+0xf7C [libnss_uxauth.so.2+0x47ae2] sqlite3WalkExpr+0x34C [libnss_uxauth.so.2+0x47bdd] sqlite3WalkExprList+0x42C [libnss_uxauth.so.2+0x4

SYMPTOM: While adding a new host via the Ambari , results in an exception as below:#####11 Apr 2016 10:52:24,629 ERROR [qtp-client-81] AbstractResourceProvider:279 - Caught AmbariException when creating a res ource org.apache.ambari.server.HostNotFoundException: Host not found, hostname=hostname_123.abc.xyz.com at org.apache.ambari.server.state.cluster.ClustersImpl.getHost(ClustersImpl.java:343) at org.apache.ambari.server.state.ConfigHelper.getEffectiveDesiredTags(ConfigHelper.java:108) at org.apache.ambari.server.controller.AmbariManagementControllerImpl.findConfigurationTagsWithOverrides(AmbariM anagementControllerImpl.java:1820) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.google.inject.internal.DelegatingInvocationHandler.invoke(Delegatin

The Apache Spark - Apache HBase Connector (SHC) is a library to support Spark accessing HBase table as external data source or sink. It provides high performance HBase access via SparkSQL and DataFrames. SHC implements the standard Spark data source APIs, and leverages the Spark catalyst engine for query optimization. It bridges the gap between the simple HBase Key Value store and complex relational SQL queries and enables users to perform complex data analytics on top of HBase using Spark. With the data frame support, SHC leverages all the optimization techniques in catalyst, and achieves data locality, partition pruning, predicate pushdown, Scanning and BulkGet, etc. For the detailed information, please refer the README in SHC github, which is kept up-to-date.

How to enable SSL for Storm UI on an unsecured cluster: 1. Generate keystore and certificate:root@beautiful-storm2 ~]# /usr/jdk64/jdk1.8.0_77/bin/keytool -genkeypair -alias certificatekey -keyalg RSA -validity 7 -keystore keystore.jksEnter keystore password:Re-enter new password:What is your first and last name? [Unknown]: stormWhat is the name of your organizational unit? [Unknown]: stormWhat is the name of your organization? [Unknown]: stormWhat is the name of your City or Locality? [Unknown]: stormWhat is the name of your State or Province? [Unknown]: stormWhat is the two-letter country code for this unit? [Unknown]: stormIs CN=storm, OU=storm, O=storm, L=storm, ST=storm, C=storm correct? [no]: yesEnter key password for <certificatekey> (RETURN if same as keystore password):Re-enter new password:2. Add below properties via Ambari to custom storm-site ui.https.key.password=bigdataui.https.keystore.password=bigdataui.https.keystore.path=/keystore.jks (This is path to your keysto

SYMPTOM: On invoking a mapreduce job or invoking a hive shell (incase where we use tez execution engine, which is default) which involves spinning up a mapreduce container on the mapreduce queue), we get the below error : [hive@sumeshhdpn2 root]$ hadoop jar /usr/hdp/2.2.4.2-2/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 1 10Number of Maps = 1Samples per Map = 10Wrote input for Map #0Starting Job16/08/15 04:12:47 INFO impl.TimelineClientImpl: Timeline service address: http://sumeshhdpn2:8188/ws/v1/timeline/16/08/15 04:12:47 INFO client.RMProxy: Connecting to ResourceManager at sumeshhdpn2/172.25.16.48:805016/08/15 04:12:48 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 129 for hive on ha-hdfs:sumeshhdp16/08/15 04:12:48 INFO security.TokenCache: Got dt for hdfs://sumeshhdp; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:sumeshhdp, Ident: (HDFS_DELEGATION_TOKEN token 129 for hive)16/08/15 04:12:53 INFO input.FileInputFormat: Total input paths to process : 116/08/15 04:12:

Setting up Log Search SSL and HTTPS A keystore and a truststore are required for this setup. These instructions assume that you have already created .jks for the keystore and truststore 1. Create keystore location a. Keystore Setup Place the keystore in /etc/security/certs/ and else by using a symlink you can point to another location of your keystore.jks b. Ensure log search user can read the keystore chown logsearch:hadoop *.keyStore.jks2. Create a truststore for logsearch: a. Cert signed by CA: i. Copy the keystore into <host>.trustStore.jks ii. Create a symlink to this similar to the keystore /etc/security/certs/truststore.jks -> /etc/security/certs/<host>.trustStore.jks b. Ensure log search user can read the trust store chown logsearch:hadoop *.trustStore.jks3. Update Ambari configuration a. Update logsearch UI Protocol to https b. Update Trust store location ( logsearch_t

Error:Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '115' for key 'PRIMARY'ROOT CAUSE: corrupted ambari db-- seeing following errors in Hive View Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '115' for key 'PRIMARY' SOLUTION: we truncated the following tables from amabri db DS_FILERESOURCEITEM_1 | | DS_JOBIMPL_3 | | DS_SAVEDQUERY_6 | | DS_STOREDOPERATIONHANDLE_5 | | DS_TESTBEAN_4 | | DS_UDF_2 Restart ambari server and we are able to access views and run query

Issue:Solr service check fails from ambari when Namenode HA is enabled with error:Unable to create core [collection1_shard2_replica1] Caused by: Connection refusedSolution 1:add this line "SOLR_HDFS_CONFIG=/etc/hadoop/conf" to solr-config-env content at the end of fileRestart Solr and now the service checks should pass.Solution 2:Another workaround is to edit the following file on your Ambari Server: /var/lib/ambari-server/resources/mpacks/solr-ambari-mpack-5.5.2.2.5/common-services/SOLR/5.5.2.2.5/package/scripts/solr.py, and then make this change: https://github.com/lucidworks/solr-stack/commit/7b79894b37b862b86d80c64b34230bc9fed6e54a. Then restart the Ambari Server. With this change in place the Solr instances will be started and will work fine with NN HA.This will be fixed in HDP 2.6

Issue:If there is a custom PID location configured for services and a non standard service account other than user "ranger", before finalizing upgrade, Ambari shows Ranger service as stopped.Solution:Confirm that the Ranger process is running:ps -ef | grep rangerThe pid and the chown values are hard coded in the /usr/bin/ranger*.From this location, start and stop are called.The pidf and the chown values are hard coded in the /usr/bin/ranger-admin from where start and stop are called if you invoke the same from ambari UI.after changing those values to custom parameters ambari reported the service as running from UI.Previous value:cd /usr/bincat ranger-admin | grep -i pid pidf=/var/run/ranger/rangeradmin.pidNew value:vi ranger-adminpidf=<custom location>Save and quitAfter changing these values restart ranger service from ambari