Community Articles

No Valid Credentials Provided ErrorUpon Enabling Kerberos in Ambari, some components started (Name Node), but other components that are are failing, such as MapReduce and Hive. Here is an example of the error output when we try to start these services. Fail: Execution of 'hadoop fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` /app-logs /mapred /mapred/system /mr-history/tmp /mr-history/done && hadoop fs -chmod -R 777 /app-logs && hadoop fs -chmod 777 /mr-history/tmp && hadoop fs -chmod 1777 /mr-history/done && hadoop fs -chown mapred /mapred && hadoop fs -chown hdfs /mapred/system && hadoop fs -chownyarn:hadoop/app-logs && hadoop fs -chownmapred:hadoop/mr-history/tmp /mr-history/done' returned 1. mesg: ttyname: Invalid argument 15/04/28 16:12:33 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials pr

It is very easy to connect various tools to Hive.Apache Zeppelin is my favorite tool to query Hive via HiveQL, but you have a ton of options.First, DBVisualizer, I am using the Free Edition for MacOSx.For both tools, you will need to copy (or acquire) a copy of two Hive JARS:hadoop-common.jarhive-jdbc-1.2.1000.2.4.0.0-169-standalone.jarOn HDP 2.4:/usr/hdp/current/hive-client/lib/hive-jdbc-1.2.1000.2.4.0.0-169-standalone.jar/usr/hdp/current/hadoop-client/hadoop-common.jarAnother tool, DBeaver which has a unique twist (it finds and downloads the JARS itself via Maven, though it takes a bit for the initial driver install). It also supports Phoenix, Drill and GemfireXD.Also I did SquirrelSQL. For SquirrelSQL setup details, check out the official Hive Docs.jdbc:hive2://localhost:10000/defaultorg.apache.hive.jdbc.HiveDriverFor Hive URL, see the Hive Docs.I also connected from SQLWorkBench/J as well. This is a free SQL Tool.

Grafana is a popular, open-source graph and dashboard builder ( licensed under Apache License, Version 2.0).Grafana is feature rich; not only does it let users create customizable and shareable dashboards, it also offers templated/scripted dashboards, LDAP integration, multiple data sources, and more. Out of the box, Grafana supports data sources such as Graphite, OpenTSDB, ElasticSearch, InfluxDB, CloudWatch, and etc. Luckily, the data source part is pluggable; it was not too difficult to create an Ambari data source for it: https://github.com/u39kun/ambari-grafanaIt works for all service metrics currently exposed via Ambari: HDFS, YARN, HBase, Storm, Kafka, Flume, Accumulo, and Ambari Metrics.It supports typeahead of component names and their associated metrics, as well as aggregation (“avg”, “sum”, “min”, and “max” of metrics across all instances of the specified component).Host Metrics are not supported yet, but it shouldn't be too difficult to add.Please visit https://github.com/

Setup your environment GistDemoSource

Hive: Apache Hive is a data warehouse infrastructure built on top of Hadoop for providing data summarization, query, and analysis. HBase: Apache HBase™ is the Hadoop database, a distributed, scalable, big data store Hawq: http://hawq.incubator.apache.org/ PXF: PXF is an extensible framework that allows HAWQ to query external system data Let's learn Query federation This topic describes how to access Hive data using PXF. Link Previously, in order to query Hive tables using HAWQ and PXF, you needed to create an external table in PXF that described the target table's Hive metadata. Since HAWQ is now integrated with HCatalog, HAWQ can use metadata stored in HCatalog instead of external tables created for PXF. HCatalog is built on top of the Hive metastore and incorporates Hive's DDL. This provides several advantages: You do not need to know the table schema of your Hive tables You do not need to manually enter information about Hive table location or format If Hive table metadata

Deploy HDB (HAWQ) on HDP 2.4.2 and use Zeppelin for VisualizationGoals:Install a 4 node cluster running HDP 2.4.2 using Ambari 2.2.2.0 (including Zeppelin and HDB) using Ambari bootstrap via blueprints or Ambari install wizardConfigure HAWQ for ZeppelinConfigure Zeppelin for HAWQRun HAWQ queries via ZeppelinAccess Hive tables from HAWQNotes:HDB managed via Ambari is only supported from Ambri 2.2.2.0 onwards. Do not attempt using older versions of AmbariSingle node setup: The same steps can be used to install a single node setup - just make sure to change the HAWQ master port from default (5432) - for example: to 10432. Sample script to automate bringing up a single node HDP cluster with HDB/Zeppelin installed is available herePrebuilt single node VM for VMWare built with HDP 2.4.2, Ambari 2.2.2.0, HDB 2.0.0.0 and Zeppelin is available for download here (size: 4.84 GB). It was setup using below so once services are up, you can access Zeppelin (on port 9995) and use the existing 'HAWQ'

Problem Statement: Ranger HDFS repository test connection getting failed while the policies are working fine. When tried enabling debug more for ranger admin service found below error in log -(RangerAuthenticationProvider.java:335) - Unix Authentication Failed:org.springframework.security.authentication.AuthenticationServiceException: FAILED: unable to authenticate to AuthenticationService: node.example.com:5151 at org.springframework.security.authentication.jaas.DefaultLoginExceptionResolver.resolveException(DefaultLoginExceptionResolver.java:33) at org.springframework.security.authentication.jaas.AbstractJaasAuthenticationProvider.authenticate(AbstractJaasAuthenticationProvider.java:181) at org.apache.ranger.security.handler.RangerAuthenticationProvider.getUnixAuthentication(RangerAuthenticationProvider.java:327) at org.apache.ranger.security.handler.RangerAuthenticationProvider.authenticate(RangerAuthenticationProvider.java:114) at org.springframew

Having run a bunch of Spark jobs locally, in Spark Standalone clusters and in HDP Yarn Clusters; I have found a few JVM settings that helped with debugging non-production jobs and assist with better Garbage Collection. This is important even with off-heap storage and bare metal optimizations.spark-submit --driver-java-options "-XX:+PrintGCDetails -XX:+UseG1GC -XX:MaxGCPauseMillis=400" You can also set options extra options in the runtime environment (see Spark Documentation).For HDP / Spark, you can add this from Ambari.In your Scala Spark Program:sparkConf.set("spark.cores.max", "4")sparkConf.set("spark.serializer", classOf[KryoSerializer].getName)sparkConf.set("spark.sql.tungsten.enabled", "true")sparkConf.set("spark.eventLog.enabled", "true")sparkConf.set("spark.app.id", "MyAppIWantToFind")sparkConf.set("spark.io.compression.codec", "snappy")sparkConf.set("spark.rdd.compress", "false")sparkConf.set("spark.suffle.compress", "true")Make sure you have Tungsten on, the KryoSerializer

After a rolling or express upgrade of HDP from 2.2.x to 2.4 (and I'm told also to 2.3.2 and 2.3.4) you may face some issues with your Kafka as I did. In my case HDP was upgraded from HDP-2.2.6 (Kafka-0.8.1) to HDP-2.4 (Kafka-0.9.0) using Ambari-2.2.1.1. Here are 4 items to check and fix if needed. After the upgrade and Kafka restart, brokers IDs are set to 1001, 1002, ... However, topics created before the upgrade point to brokers numbered 0, 1, ... For example (k81c was created before the upgrade, k9a after):$ ./kafka-topics.sh --zookeeper zk1.example.com:2181 --describe --topic k81c Topic:k81c PartitionCount:2 ReplicationFactor:2 Configs: Topic: k81c Partition: 0 Leader: 0 Replicas: 0,1 Isr: 0 Topic: k81c Partition: 1 Leader: 0 Replicas: 1,0 Isr: 0$ ./kafka-topics.sh --zookeeper zk1.example.com:2181 --describe --topic k9aTopic:k9a PartitionCount:2 ReplicationFactor:1 Configs: Topic: k9a

SmartSense is an excellent tool for keeping your cluster running at optimal efficiency while maintaining operational best practices. We’ve combined knowledge from the greatest minds in the industry, and use it to analyze metadata about your cluster from the bundles you submit.Have you ever wondered exactly what data you’re sending to SmartSense? The SmartSense Admin Guide contains a high-level description (see What’s Included in a Bundle), but for the greatest understanding you should extract a bundle and explore it with your own eyes!Obtain a BundleThere are two types of bundles...Analysis Bundle: configs and metrics for all services on all hostsTroubleshooting Bundle: Analysis Bundle + logs for selected service(s)To begin, let’s capture an Analysis Bundle:...and download an unencrypted copy to our local machine:The bundle is a gzipped tar file that contains a gzipped tar file from each host running the HST Agent. In the following examples, notice the bundle variable excludes the .tgz

Chronos is a replacement for cron. A fault tolerant job scheduler for Mesos which handles dependencies and ISO8601 based schedules Marathon is a framework for Mesos that is designed to launch long-running applications, and, in Mesosphere, serves as a replacement for a traditional system In Mesosphere, Chronos compliments Marathon as it provides another way to run applications, according to a schedule or other conditions, such as the completion of another job. It is also capable of scheduling jobs on multiple Mesos slave nodes, and provides statistics about job failures and successes. Source Install https://mesos.github.io/chronos/docs/ and gist

This article is a continuation of Monitoring Kafka with Burrow - Part 1.Before diving into evaluation rules, HTTP endpoint API and notifiers, I would like to point-out a few other tools that are utilizing Burrow. Burrower (http://github.com/splee/burrower), a tool for gathering consumer lag information from Burrow and sending it into influxDBansible-burrow (https://github.com/slb350/ansible-burrow) provides an Ansible role for installing Burrow.Consumer Lag EvaluationStatusThe status of a consumer group in Burrow is determined based on several rules evaluated against the offsets for each partition the group consumes. Thus, there is no need for setting a discrete threshold for the number of messages a consumer is allowed to be behind before alerts go off. By evaluating against every partition the group consumes, the entire consumer group health status is evaluated, and not just the topics that are being monitored. This is very important for wildcard consumers, such as Kafka Mirror Maker

How to Index PDF File with Flume and MorphlineSolrSink The flow is as follows: Spooling Directory Source > File Channel > MorphlineSolrSinkThe reason I wanted to complete this exercise was to provide a less complex solution; that is, fewer moving parts, less configuration, and no coding compared to kafka / storm or spark. Also, the example is easy to setup and demonstrate quickly.Flume compared to Kafka/Storm is limited by its declarative nature, but that is what makes it easy to use. However, the morphline does even provide a java command (with some potential performance side effects), so you can get pretty explicit.I’ve read that Flume can handle at 50,000 events per second on a single server, so while the pipe may not be as fat as a Kafka/Storm pipe, it may be well suited for many use cases. Step-by-step guide1. Take care of dependencies. I am running HDP 2.2.4 Sandbox and the Solr that came with it. To get started, you will need to add a lot of dependencies to

QUESTION:How are the Ambari-2.2.1.1 local accounts protected, such as "admin"? What about all the various component's configuration data managed by Ambari-2.2.1.1?ANSWER:Ambari local account credentialsThese are stored in the Ambari database as the SHA256 hash of the (randomly salted) password.Service configuration password propertiesThese are stored in the Ambari database in blobs of JSON-formatted data in plaintext.When returned via API calls, the properties marked as passwords are masked and not displayed as plaintext.When sent to the agents, they are stored in plaintext in the command.json files stored in /var/lib/ambari-agent/data (readable only by root and the user that executes ambari-agent).Ambari-specific database and ldap credentialsThese are stored in plaintext in the ambari.properities file by default but can be encrypted via ambari-server setup-security.If encrypted, they are stored in a Java Keystore implementation (JCEKS) which uses 3DES in CBC mode with PKCS #5 padding

Virtual Memory swapping can have a large impact on the performance of a Hadoop system. Because of the memory requirements of YARN containers and processes running on the nodes in a cluster, swapping process out of memory to disk can cause serious performance limitations. As such, the historical recommendations for setting the swappiness, or propensity to swap out a process, on a Hadoop system has been to disable swap altogether. With newer versions of the Linux kernel, Out Of Memory (OOM) situations can be more likely to indiscriminately kill important processes to reclaim valuable physical memory on the system with a swappiness of 0.In order to prevent the system from swapping processes too frequently, but still allow for emergency swapping (instead of killing processes), the recommendation is now to set swappiness to 1 on Linux systems. This will still allow swapping, but with the least possible aggressiveness (for comparison, the default value for swappiness is 60).To change the sw

Original Post DC/OS - a new kind of operating system that spans all of the servers in a physical or cloud-based datacenter, and runs on top of any Linux distribution. Source Projects More details https://docs.mesosphere.com/overview/components/ Let's cover Mesos in this post Frameworks (Application running on mesos) http://mesos.apache.org/documentation/latest/frameworks/ I used http://mesos.apache.org/gettingstarted/ to install Mesos in my local machine. I am launching c++, java and python framework in this demo. Slide Share http://www.slideshare.net/tomasbart/introduction-to-apache-mesos

IntroductionMirrorMaker can be used to replicate messages of a defined list of topics across clusters.ConfigurationWe need to configure both the producer and the consumer through config files. The config file for the producer will define the destination cluster Kafka brokers, whereas the consumer one will point to the Zookeeper servers from the source cluster. The following configuration also assumes that security is enabled (Kerberos) so the PLAINTEXTSASL protocol will be used:mm_producer-1.properties:bootstrap.servers=dest-kafkabrk1:6667,dest-kafkabrk2:6667,dest-kafkabrk3:6667producer.type=asyncqueue.time=1000000queue.enqueueTimeout.ms=-1security.protocol=PLAINTEXTSASL#sasl.kerberos.service.name=kafkamm_consumer-1.properties:zookeeper.connect=src-zkhost1:2181,src-zkhost2:2181,src-zkhost3:2181zk.connectiontimeout.ms=1000000bootstrap.servers=src-kafkabrk1:6667,src-kafkabrk2:6667,src-kafkabrk3:6667consumer.timeout.ms=-1security.protocol=PLAINTEXTSASLgroup.id=kafka-mirror#sasl.kerberos.s

1.Login to Ambari UI using admin credentials [admin/admin]2.Check the alert definition using below command http://<ambari_fqdn>:8080/api/v1/clusters/<cluster-name>/alert_definitions/3.Get the respective alert definition you wan to modify from the above output. Here for example lets say we will modify "Hive Metastore".Here we will change 'check.command.timeout' value from default 60 to 120 4.Copy the output of point2 to a file say "test.json"5.First - Modify the test.json and remove the 2nd line of the file which starts with "href" eg. is below -"href" : "http://<ambari-fqdn>:8080/api/v1/clusters/sandbox/alert_definitions/51",6.Second - edit the value of 'check.command.timeout' in test.json from 60.0 to 120.0{ "name" : "check.command.timeout", "display_name" : "Check command timeout", "units" : "seconds", "value" : 120.0, "description" : "The maximum time before check command will be killed by timeout", "type" :

Introduction: Here is a small demo how NiFi can let you retrieve Stock quotes on voice commands and respond back with the help of Mac Dictation.- Here you can view the screen recording session that Demonstrates how it works! NiFi + Mac Dictation DemoPrerequisite: 1) Assuming you already have latest version of NiFi [0.5.x or later] downloaded on your Mac. 2) Make sure you have Mac dictation feature, and its working! Steps: 1) Use GetHttp processor to Retrieve any Stock price from “Google Finance”, you can use the below URL to fetch YAHOO stock price: http://finance.google.com/finance/info?client=ig&q=YHOO - Processor Properties would look like: 2) Use ReplaceText and EvaluateJsonPath processors to trim json result and extract stock value and variation in stock price today. Use UpdateAttribute processor to save values to attributes. 3) Now you can use a RouteOnAttribute to decide when to trigger mail or/and give voice feedback. 4) Finally Use PutEmail process

Using Sqoop (Sql from/to Hadoop), you can use a password file instead of a plaintext password, which is more secure :<arg>--password-file</arg> <arg>hdfs://NAMENODE/teradata.password</arg> You may end with error like 3737 [main] ERROR org.apache.sqoop.teradata.TeradataSqoopExportHelper - Exception running Teradata export jobcom.teradata.connector.common.exception.ConnectorException: java.sql.SQLException: [Teradata Database] [TeraJDBC 15.00.00.20] [Error 8017] [SQLState 28000] The UserId, Password or Account is invalid. If you set your “Password” pass with vi you’ll end with a line feed control character, making the password invalid. To find if there’s a LF ending the password file, use od (display file in octal format): [root@localhost ~]# od -c teradata.password 0000000 P a s s w o r d \n 0000011You’ll have to delete your newline control character using tr : [root@localhost ~]# tr -d '\n' < teradata.password > teradata.password.new [r

You can now visualize any Zeppelin notebook using Zeppelinhub viewer. https://www.zeppelinhub.com/viewerpersonal likes:1. No need to sign up or register just paste a link2. I've been posting my zeppelin notebooks to github but everyone that wants to visualize them or interact with them needs to download, move to environment, import into their instance of zeppelin. Not anymore just paste the link.3. Less of a need to take screenshots and create a powerpoint just send the hyperlinkexamples: Stock Variance Notebook github - https://github.com/kirkhas/zeppelin-notebooks/blob/master/stock-variance/note.jsonvsStock Variance Notebook zephub - https://www.zeppelinhub.com/viewer/notebooks/aHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL2tpcmtoYXMvemVwcGVsaW4tbm90ZWJvb2tzL21hc3Rlci9zdG9jay12YXJpYW5jZS9ub3RlLmpzb24Credit Card Fraud Transactions git - https://github.com/vakshorton/CreditCardTransactionMonitor/blob/master/Zeppelin/notebook/2BGDWYZV9/note.jsonvshttps://www.zeppelinhub.com/viewer/

A question encountered by most organizations when attempting to take on new Big Data Initiatives or adapt current operations to keep up with the pace of innovation is What approach to take in architecting their streaming workflows and applications. Because Twitter data is free and easily accessible I chose to examine twitter events in this article; however, twitter events are simply json messages and can be conceptualized as machine generated json packets for a more generalized streaming architecture. Lets move forward with an evaluation of NiFi vs Python ingesting streaming events from twitter. When approaching a problem there are often several tools and methods that could ultimately yield similar results. In this article I will look at the problem of ingesting streaming twitter events and show how HDF and python can be used to create an application which achieves the goal. I will first lay out a solution using both platforms, then I will point out different aspects and co

Modern data driven applications require a "Connected Platform" capable of bringing data in and out to/from the Internet of Things, mobile users, and social media in real time. In order to monetize all of that real time data the platform must have the ability to process Petabytes of data to create adaptive learning algorithms and apply those algorithms in real time as the data streams in and out of the platform. However, the modern data application cannot be effectively utilized or operated without an application tier that allows the business to visualize, interact, and act on the massive volumes of data and insight coming in and out in real time and accumulating in storage. The Hortonworks Connected Platform "HDP+HDF" has the capability to act as a PaaS that can host the application tier of the modern data application along side of all of the data processing. It is possible to use Slider to run a dockerized application managed by Yarn inside of the Hadoop cluster similar to an applicat

I ran into some issues using the latest iODBC 3.52.10 version from www.iodbc.org (mxkozzzz.dmg)Instead of~/.odbc.ini ~/.odbcinst.ini ~/.hortonworks.hiveodbc.iniit uses (without leading dots):~/Library/ODBC/odbc.ini ~/Library/ODBC/odbcinst.ini ~/Library/ODBC/hortonworks.hiveodbc.iniNote: iODBCwill link ~/.odbc.ini and ~/.odbcinst.ini to the ~/Library/ODBC/ versionsMy setup:~/Library/ODBC/odbcinst.ini - edit at least host, port and user ~/Library/ODBC/odbcinst.ini - keep as is ~/Library/ODBC/hortonworks.hiveodbc.ini - Change ODBCInstLib to the fully qualified path of iODBC's libiodbcinst.dylib under /usr/local/iODBC/lib/[Driver]## - Note that this default DriverManagerEncoding of UTF-32 is for iODBC.DriverManagerEncoding=UTF-32ErrorMessagesPath=/usr/lib/hive/lib/native/hiveodbc/ErrorMessages/LogLevel=0LogPath=SwapFilePath=/tmp# iODBCODBCInstLib=/usr/local/iODBC/lib/libiodbcinst.dylib{ignore the "5." in the output above - no idea how to get rid of it}

When Storm developers want to share data across multiple bolts or cache a large amount of state information in a single bolt, one of the common choices is to use HBase. In a unsecured HDP cluster, the related code in the HBase Bolt is very intuitive:public class myHBaseBolt implements IRichBolt { ... private OutputCollector collector; private Connection connection; private Table myTable; ... public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) { this.collector = collector; try { this.connection = ConnectionFactory.createConnection(HBaseConfiguration.create()); this.myTable = connection.getTable(TableName.valueOf(MY_TABLE_NAME)); ... } catch (Exception e) { ... } }}However, in a Kerberized HDP cluster, we need to configure lots of information in our code to enable secured connection. First, we need to configure storm keytab and principal for the hbase client in storm bolt. For example, in the myTopology.java code:Map<String, Object>