Resources

Many of us in Hortonworks Community Connection feel most at home when we are talking about technologies and tools and the "animals in the zoo". However if we want to grow the data lake and gain support from the business we have to learn to think a little differently and use a new vocabulary to communicate. Start by meeting with the business to identifying possible use cases. Talk to the analysts about the highest priorities and pain points for the business. Before thinking about anything remotely Hadoop animal like, summarize "what" needs to be done. This may take several interviews with different business analysts to gain a full understanding of the problem. Then determine if Big Data can solve the problem. Are data silos preventing the organization from getting a complete view of the customer or logistics? Is the volume of data required to solve the problem too much or too expensive for existing systems to handle? Are the unstructured or semi-structured data required to s

Connecting to HBase throws the below exceptions: [root@hl1 hbase]# cd /usr/hdp/current/phoenix-client/bin/[root@hl1 bin]# python sqlline.py localhost:2181:/hbase-unsecuresun.misc.SignalHandler not found in gnu.gcj.runtime.SystemClassLoader{urls=[file:/etc/hbase/conf/,file:/usr/hdp/2.5.3.0-37/phoenix/bin/../phoenix-4.7.0.2.5.3.0-37-client.jar,file:./,file:/etc/hadoop/conf/,file:/usr/hdp/2.5.3.0-37/hadoop/conf/,file:/usr/hdp/2.5.3.0-37/hadoop-hdfs/./], parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}Setting property: [incremental, false]Setting property: [isolation, TRANSACTION_READ_COMMITTED]issuing: !connect jdbc:phoenix:localhost:2181:/hbase-unsecure none none org.apache.phoenix.jdbc.PhoenixDriverConnecting to jdbc:phoenix:localhost:2181:/hbase-unsecurejava.lang.ClassFormatError: org.apache.phoenix.jdbc.PhoenixDriver (unrecognized class file version) at java.lang.VMClassLoader.defineClass(libgcj.so.10) at java.lang.ClassLoader.defineClass(libgcj.so.10) at java

Storage is Fundamental to Big DataStorages can be chiefly evaluated on three classes of performance metrics: Cost per GigabyteDurability - this is the measure of the permanence of data once it has been successfully written to the medium. Modern hard disks are highly durable, however given a large enough collection of disks, regular disk failures are a statistical certainty. Annual Failure Rates of disks vary between 2 – 8% for 1 – 2 year old disks as observed in a large-scale study of disk failure rates.[1]Performance - there are two key measures of storage performance: Throughput - this is the maximum raw read/write rate that the storage can support and is typically measured in MegaBits/second (MBps). This is the primary metric that batch processing applications care about.IO operations per second (IOPS) - the number of IO operations per second is affected by the workload and IO size. The rotational latency of spinning disks limits the maximum IOPS for a random IO workload which can

By default the GC logs are not enabled for Hive components. It is good to enable them to troubleshoot GC pauses on hiveserver2 instances.---------------------------------Hiveserver2 / Metastore:---------------------------------In Ambari navigate to the following pathServices -- > Hive -- > Configs -- > Advanced --> Advanced hive-env --> hive-env templateAdd the following lines at the beginning,if [[ "$SERVICE" == "hiveserver2" || "$SERVICE" == "metastore" ]]; thenHIVE_SERVERS_GC_LOG_OPTS="-Xloggc:{{hive_log_dir}}/gc.log-$SERVICE-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps"export HADOOP_OPTS="$HADOOP_OPTS $HIVE_SERVERS_GC_LOG_OPTS"fi---------------------------------Webhact :---------------------------------In Ambari navigate to the following pathServices -- > Hive -- > Configs -- > Advanced --> Advanced webhcat-env --> webhcat-env templateAdd the following lines at the bottom,WEBHACAT_GC_LOG_OPTS="-Xlog

Error:ERROR [2016-12-13 00:48:04,166] ({pool-2-thread-2} Job.java[run]:189) - Job failedjava.lang.NoClassDefFoundError: org/apache/hadoop/security/UserGroupInformation$AuthenticationMethod at org.apache.zeppelin.jdbc.security.JDBCSecurityImpl.getAuthtype(JDBCSecurityImpl.java:66)....Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.security.UserGroupInformation$AuthenticationMethodat java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 15 more Solution:For systems in hdp cluster with no internet access local-repo folder doesnt get created when running zeppelin interpreters which hold the jar files needed for the interpreters.To circumvent this we would need to copy the following jars to a temporary location and then specify these in the interpreter setting from zeppelin UIcreate and copy

When using kerberos with HDP it's not uncommon to find the odd strange encryption type floating around, possibly from a badly configured AD server. By adding the following to the Ambari -> Kerberos config section under supported encryption types its possible to isolate this issue for diagnostic.While its probably not a wise idea to run with all these enabled in production having a full list of supported types can be useful for diagnostic or reference. des-cbc-crc des-cbc-md4 des-cbc-md5 des3-cbc-sha1 arcfour-hmac arcfour-hmac-exp aes128-cts-hmac-sha1-96 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha256-128 aes256-cts-hmac-sha384-192 camellia128-cts-cmac camellia256-cts-cmac

A quick easy way to export users from LDAP to a CSV for use in importing to Ambari.ldapsearch -h ldapserver.mycompany.com -p 389 -D bind-user -w bind-password -b OU=London\ Users,OU=my\ dept,DC=mycompany,DC=com -s sub "(objectClass=user)" | grep sAMAccountName | sed 's/sAMAccountName://g' | awk -vORS=, '{ print $1 }' | sed 's/,$/\n/'

NOTE: This is for information and testing purposes only. Pleaes exercise caution prior to using it on business critical environments.STEPS:To completely remove HDP stack from the Operating System, do the following:1. Stop all the services in Ambari UI.2. Shutdown ambari-server and ambari-agent on Ambari server host using the following commands:$ ambari-server stop$ ambari-agent stop3. Shutdown ambari-agent on all other nodes using the following commands:$ ambari-agent stop4. Run the following command to check the installed services to delete the related configs and directories:$ hdp-select | grep -v 'None'5. Follow this URLto uninstall each services that is installed as per the above step or from the Ambari UI.6. Remove all the files and directory related with Hadoop and HDP using the following commands. This step will delete all the files and directories name with hadoop under root partition. Please be extra caution to run this step and make sure you have nothing important in these di

Note: This is for information and testing purposes only. Please exercise caution prior to using it on business critical environments.Steps to migrate the ranger and ranger_audit database to the new MySQL host without re-installing Ranger Service via Ambari UI - 1. Stop all the services on the cluster via Ambari UI.2. Make sure Ranger Admin and Ranger usersync is down.3. Take dump of ranger and ranger_audit databases.$ mysqldump ranger > /path/to/ranger_dump.sql$ mysqldump ranger_audit > /path/to/ranger_audit_dump.sql 4. Login to new database host and to the database.$ mysql -u root -p 5. Create user rangerdba and provide them all privileges.CREATE USER 'rangerdba'@'localhost' IDENTIFIED BY '<rangerdba_password>'; GRANT ALL PRIVILEGES ON *.* TO 'rangerdba'@'localhost'; CREATE USER 'rangerdba'@'%' IDENTIFIED BY '<rangerdba_password>'; GRANT ALL PRIVILEGES ON *.* TO 'rangerdba'@'%'; GRANT ALL PRIVILEGES ON *.* TO 'rangerdba'@'<new_database_fqdn>';GRANT ALL PRIVILEG

In the current version of Kafka, when Kafka cluster is enabled with ranger authorizer(authorizer.class.name=org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer) it is not possible to auto create topics as a non super user even if the auto create topic flag is set to true. In other words, Kafka create topic authorization can not be done at a topic level.For example, create a ranger policy as below, Topic AutoCreateTopic_Test* with all permissions to a non super user. Run the command line Kafka producer script to create a non existing topic, /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list < > --topic AutoCreateTopic_Test01--security-protocol PLAINTEXTSASL[2017-02-24 19:10:30,232] WARN Error while fetching metadata [{TopicMetadata for topic test4 -> No partition metadata for topic test4 due to kafka.common.TopicAuthorizationException}] for topic [test4]: class kafka.common.TopicAuthorizationException (kafka.producer.BrokerPartitionIn

PROBLEM : Hive view errors out with null pointer exception from Stand alone ambari view server. ERROR :java.lang.NullPointerExceptionat org.apache.ambari.view.hive2.resources.jobs.atsJobs.ATSParser.getHiveQueryIdFromJson(ATSParser.java:113)at org.apache.ambari.view.hive2.resources.jobs.atsJobs.ATSParser.getHiveQueryIdByOperationId(ATSParser.java:107)at org.apache.ambari.view.hive2.resources.jobs.Aggregator.readATSJob(Aggregator.java:260)at org.apache.ambari.view.hive2.resources.jobs.JobService.jsonObjectFromJob(JobService.java:161)at org.apache.ambari.view.hive2.resources.jobs.JobService.getOne(JobService.java:145)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ROOT CAUSE : Stand alone ambari host name missing in yarn.timeline-service.http-authentication.proxyuser.ambari-server. WORKAROUND / RESOLUTION : Update below property in YARN configs and make the value to *:yarn.timeline-service.http-authentication.proxyuser.ambari-server-<CLUSTER_NAME>.hosts=*yarn.timeline

Assuming you start with a kerberized HDP cluster with Hbase installed.First check what your hbase service principal is i.e.klist -kt /etc/security/keytabs/hbase.service.keytabKeytab name: FILE:hbase.service.keytabKVNO Timestamp Principal---- ----------------- -------------------------------------------------------- 2 12/20/16 13:51:21 hbase/hdp253k1.hdp@HDP.COM 2 12/20/16 13:51:21 hbase/hdp253k1.hdp@HDP.COM 2 12/20/16 13:51:21 hbase/hdp253k1.hdp@HDP.COM 2 12/20/16 13:51:21 hbase/hdp253k1.hdp@HDP.COM 2 12/20/16 13:51:21 hbase/hdp253k1.hdp@HDP.COM In Ambari head to Hbase -> Configs -> Advanced -> Custom Hbase-Site.xml and add the following new parameters with the keytab / principal substituted for your environment. If they already exist for your cluster set the values as indicated:hbase.rest.authentication.type=kerberoshbase.master.kerberos.principal=hbase/_HOST@HDP.COMhbase.master.keytab.file=/etc/security/keytabs/hbase.service.keytabhadoop.proxyuser.HTTP.grou

Introduction The Hortonworks Sandbox running on Azure requires opening ports a bit differently than when the sandbox is running locally on Virtualbox or Docker. We’ll walk through how to open a port in Azure so that outside connections make their way into the sandbox, which is a Docker container inside an Azure virtual machine. Note: There are multiple ways to open ports to a Docker container (i.e. the sandbox). This tutorial will cover the simplest method, which reinitializes the sandbox to its original state. In other words, you will lose all changes made to the sandbox. Prerequisites Deploying Hortonworks Sandbox on Microsoft Azure Outline SSH Into The Azure VM Add Ports to the Docker Script Remove the Current Sandbox Container) Restart the Azure VM (Optional) Add New Ports to the SSH Config SSH Into the Azure VM If you followed the previous tutorial, Deploying Hortonworks Sandbox on Microsoft Azure, this step is as easy as running: ssh azureSandbox

PROBLEM : When the property skip.header.line.count is enabled on tables with large text files (~100G), we can see that the tez.grouping* parameters are ineffective, spinning one container per file.EXAMPLE: We have a table with 10 files, 117.0 G A.txt 29.0 G B.txt 30.0 G C.txt 1.9 G D.txt 11.0 G E.txt 18.9 G F.txt 5.7 G G.txt 159.4 H.txt 2.4 G I.txt 214.4 M J.txt Created table on top of this data, using CREATE TABLE `tab1`( `opr_glbl_sku_id` string, `opr_loc_nbr` string, `wk_start_dt` string, `base_fcst` float) ... TBLPROPERTIES ( 'skip.header.line.count'='1') Set these params (below), after which expectation is at least 200G/0.5G= 400 containers should be launched. SET tez.grouping.min-size=53421772; SET tez.grouping.max-size=534217728; With tez as the execution engine an attempt to perform select count(*) on the table spins only 10 containers. Once we remove the skip.header.line.count', it launched about 540 containers, which is expected behavior.ROOT CAUSE : hive.tez.i

1. Determine the version of the source hive database (from old cluster): mysql> select * from VERSION; +--------+----------------+----------------------------+ | VER_ID | SCHEMA_VERSION | VERSION_COMMENT | +--------+----------------+----------------------------+ | 1 | 1.2.0 | Hive release version 1.2.0 | +--------+----------------+----------------------------+ 2. On the old clusters metastore Mysql Database, take a dump of the hive database mysqldump -u root hive > /data/hive_20161122.sql 3. Do a find and replace in the dump file for any host name from the old cluster and change them to the new cluster (i.e. namenode address). Be cautious if the target cluster is HA enabled, when replacing hostname/hdfs url.4. On the new cluster, stop the Hive Metastore service 5. Within MySQL, perform drop/create of the new hive database. mysql> drop database hive; create database hive; 7. Run the schematool to initialize the schema to the exact version of the hive schema on the ol

PROBLEM : Query fails with the below mentioned exception.ERROR :java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:192)at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.<init>(TezGroupedSplitsInputFormat.java:131)ROOT CAUSE : Query fails on a field of datatype timestamp within an ORC table having null values, provided Predicate push down is enabled. This is a known issue. Reference: https://hortonworks.jira.com/browse/BUG-47472WORKAROUND / RESOLUTION : This setting is whether to enable automatic

In this article, we will see how to perform the mirroring of Hive data and metadata using Falcon from source cluster to destination cluster. This article is based on HDP 2.5. Configure HiveConfigure source and target Hive by clicking “Hive” from the Ambari Services menu, then click “configs” to add following custom properties on Ambari UI by scroll down to “Custom hive-site”, click it and then click “Add Property”.Add following property name with value:hive.metastore.event.listeners = org.apache.hive.hcatalog.listener.DbNotificationListenerhive.metastore.dml.events = truePress OK to save the changes, then click Restart all the impacted services. Bootstrap Table and DBBefore creating Hive DR mirroring job to replicate Hive data/metadata for a DB or Table, it is required to perform initial bootstrap of Table and DB from source to target cluster. Table BootstrapFor bootstrapping table replication, do an EXPORT of the table in question at source cluster, distcp the export directory t

There are many ways to validate a json file against a avro schema to verify all is kosher. Sharing a practice I have been using for few years.Objective - Validate avro schema well bound to the json fileFirst you must have a avro schema and json file. From there download the latest a avro-tools jar. At the moment 1.8.1 is the latest avro-tools version jar available.Store the avro schema and json file in the same directory. Issue a wget to fetch the avro-tools jarwget http://www.us.apache.org/dist/avro/avro-1.8.1/java/avro-tools-1.8.1.jarHere is what the directory looks likeObjective Details - Validate avro schema student.avsc binds to student.jsonHow - Issue the followingjava -jar ./avro-tools-1.8.1.jar fromjson --schema-file YourSchemaFile.avsc YourJsonFile.json > AnyNameForYourBinaryAvro.avroUsing the student files example:java -jar ./avro-tools-1.8.1.jar fromjson --schema-file student.avsc student.json > student.avroValidation passed, a avro binary was created.Now as a la

After defining cluster roles for the users and groups, LogSearch UI is not accessible with login failure message for individual users and groups.Below was the error in logs - logsearch-audit.json{"level":"WARN","file":"LogsearchAuthenticationProvider.java","thread_name":"qtp1464642111-13","line_number":124,"log_message":"{\"principal\":\"sysair\",\"result\":\"denied\",\"reason\":\"Wrong password\",\"remote_ip\":\"192.168.0.201\",\"session\":\"1a52ertsmv7l31qpo3gl3hdsyt\",\"auth_class\":\"org.springframework.security.authentication.UsernamePasswordAuthenticationToken\",\"user\":\"sysair\"}","logger_name":"org.apache.ambari.logsearch.audit","logtime":"1483536859578"} logsearch.json{"level":"INFO","file":"LogsearchAuthenticationProvider.java","thread_name":"qtp1464642111-14","line_number":66,"log_message":"Authenticating user:sysair, userDetail\u003dorg.springframework.security.authentication.UsernamePasswordAuthenticationToken@b48163fb: Principal: sysair; Credentials: [PROTECTED]; Authen

OverviewI recently encountered a question where someone asked to see how you can do preemption across YARN queues when a spark job is beyond it's queue's min guarantee. They had seen this before with the Fair Scheduler and Map Reduce, but wanted to apply the same experience here but with Spark and the Capacity Scheduler. This how-to article describes how to setup this experience. Goal: Run large Spark jobs in two separate capacity queues to produce an equal share of resources for both jobs. Hardware: 5 Nodes of AWS EC2 r3.xlarge Cluster Configuration: HDP: 2.4.2, Spark: 1.6.1, 5 Node Managers, 20GB (20480MB) Yarn Containersyarn.scheduler.maximum-allocation-mb = 20480yarn.scheduler.minimum-allocation-mb = 2560 High Level Setup:1. Add preemption properties as per documentation2. Create Two YARN Queues with Fair OrderingChild queue “test1” with a min capacity of 50% and a max of 100%Child queue “test2” with a min capacity of 50% and a max of 100%Root queue with a fair ordering policy3.

If you upgrade to or install HDP 2.5.0 or later without first installing the Berkeley DB file, you will get the error "Unable to initialize Falcon Client object. Cause : Could not authenticate, Authentication failed". The information below has also been added to the Data Movement and Integration guide and other Hortonworks documentation.Prerequisite to Installing or Upgrading Falcon Starting with the HDP 2.5.0 release, customers must provide the free Berkeley DB JAR file, under the Oracle license, as part of a Falcon installation or upgrade. This applies to Ambari express or rolling upgrades, to manual upgrades, and to all installations of HDP 2.5.0 or later, if you are using the Falcon service. Following are both Ambari and manual instructions for downloading and installing the JAR.Ambari Install or Upgrade of Falcon If you are using the Falcon service, the following steps must be performed on all Ambari server hosts on the cluster, prior to installing or upgrading to HDP 2.5.0

SYMPTOM: While integrating Storm 1.0.1 to Elastic Search 5.0.0, the following error is observed:Exception in thread "main" java.lang.NoClassDefFoundError: org/elasticsearch/common/base/Preconditions at org.apache.storm.elasticsearch.common.EsConfig.<init>(EsConfig.java:62) at org.apache.storm.elasticsearch.common.EsConfig.<init>(EsConfig.java:49) at com.mz.pipeline.StreamToES5_1.main(StreamToES5_1.java:72) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) Caused by: java.lang.ClassNotFoundException: org.elasticsearch.common.base.Preconditions Following were the maven dependencies:<dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-cor

SYMPTOM: For a Capacity Scheduler queue that specifies some groups in its acl_submit_applications property, a user who is not a member of any of those groups is still able to submit jobs to the queue. ROOT CAUSE: By default the root queue is allow-all, which results in all child queues defaulting to allow-all.The acl_submit_applications property is described as: The ACL which controls who can submit applications to the given queue. If the given user/group has necessary ACLs on the given queue or one of the parent queues in the hierarchy they can submit applications. ACLs for this property are inherited from the parent queue if not specified.SOLUTION: Set the root queue to deny-all, by entering a "space" for the value. Then set who to allow in the ACL for each child queue. For example:yarn.scheduler.capacity.root.acl_submit_applications= yarn.scheduler.capacity.root.default.acl_administer_jobs=appdevyarn.scheduler.capacity.root.default.acl_submit_applications=appdevyarn.scheduler.capac

PROBLEM DESCRIPTIONThe Oozie service check fails and the following error message is displayed in Ambari:stderr: /var/lib/ambari-agent/data/errors-12523.txtPython script has been killed due to timeout after waiting 300 secs There is no error in the stdout. The service check is terminated because the timeout (300 secs by default) is reached. CAUSEThis issue occurs when the time taken by Ambari to upload jar and workflow files to hdfs for Oozie service check takes longer than the timeout mentioned in the server settings.Example:Nodes in a cluster are configured on ipv4 proxy, which causes network slowness among nodes. Ambari uploads jar files and workflow files to the hdfs for Oozie service check, depending on network performance time required to upload these files exceeds 300 secs timeout.WORKAROUNDTo increase the timeout, find and update the timeout set in the metainfo.xml file located in Edit /var/lib/ambari-server/resources/common-services/OOZIE/your_version_number/metainfo.xml, RES

To configure Oozie workflow after enabling the ResourceManager HA, do the following: Find the YARN Resource Manager cluster id. To find the cluster id, Go to Ambari > YARN > configs Search for property yarn.resourcemanager.cluster-idAfter finding the cluster id, at the Job Property file, provide the YARN Resource Manager cluster id to the job _Tracker like the following:jobTracker= yarn-haSubmit the Oozie job using the modified job property file.