Resources

Log Search is a log analysis / monitoring tool which is shipped with Ambari. Log Search has 2 components: Log Search Portal (server + web) and Log Feeder. The second one is responsible to monitor specific log files and ship the parsed log lines into Solr. To define which files should be monitored and how the parsing should work for those files, you will need input config descriptors (list of the input config descriptors are located in /etc/ambari-logsearch-logfeeder/conf/logfeeder.properties, all of them are defined in logfeeder.config.files configuration property, on ambari managed Log Search service, those can be found in logfeeder-properties/logfeeder.config.files configuration entry)If you have a specific custom service (see: https://cwiki.apache.org/confluence/display/AMBARI/Custom+Services), to support that inside Log Search application you will need a *-logsearch-conf.xml (* can be a custom name, Ambari will generate input.config-*.json file based on the name inside /etc/ambari-

Understanding scale issues in AMS (Why does it hapen)The Metrics Collector component is the central daemon that receives metrics from ALL the service sinks and monitors that sends metrics. The collector uses HBase as its store and phoenix as the data accessor layer.In a high level, the metrics collector performs 2 operations related to scale in a continuos basis. Handle raw writes - A raw write is a bunch of metric data points received from services written onto HBase through phoenix. There is no read or aggregation involved. Periodically aggregate data - AMS aggregates data across cluster and across time. Cluster Aggregator - Computing the min,max,avg and sum of memory across all hosts is done by a cluster aggregator. This is called a 'TimelineClusterAggregatorSecond' which runs every 2 mins. In every run it reads the entire last 2 mins data and calculates aggregates and writes back. The read is expensive since it has to read non-aggregated data, while the write volume is smaller s

Working with airbnb's SupersetThis is a very cool open source Analytics platform based on some cool Python.I installed this on a CentOS 7 edge node.sudo yum upgrade python-setuptoolssudo yum install gcc libffi-devel python-devel python-pippython-wheel openssl-devel libsasl2-devel openldap-develpip install virtualenvvirtualenv venv. ./venv/bin/activatepip install --upgrade setuptools pippip install mysqlclient pip install pyhivepip install supersetfabmanager create-admin --app superset2017-01-27 18:15:37,864:INFO:flask_appbuilder.security.sqla.manager:Created PermissionView: menu access on Query Search2017-01-2718:15:37,885:INFO:flask_appbuilder.security.sqla.manager:Added Permission menuaccess on Query Search to role AdminRecognized Database Authentications.2017-01-2718:15:37,907:INFO:flask_appbuilder.security.sqla.manager:Added user adminAdmin User admin created.superset db upgradesuperset load_examplessuperset initsuperset runserver -p 8088The main things you will need are PythonBrow

Installing the Alarm Fatigue Demo via Cloudbreak: There are multiple ways to deploy the Alarm Fatigue Demo via Cloudbreak. Below are four options: 1) Deploy via the Cloudbreak UI a) Login to https://cbdtest.field.hortonworks.comb) Select your credentials – if you credentials don’t exist, create them under “Manage Credentials”c) Once your credentials are selected, click “Create Cluster”d) Make-up a cluster name and choose the Availability Zone (SE) and then click “Setup Network and Security"e) “fieldcloud-openstack-network” should be selected and click “Choose Blueprint”f) Select the Blueprint called “alarm_fatigue_v2”Host Group 1 (Select Ambari Server, alarm-fatigue-demo and pre-install-java8)Host Group 2 (select pre-install-java8)Host Group 3 (select pre-install-java8)g) Click on “Review and Launch”e) Click on “Create and start cluster” (After clicking, the deployment via Cloudbreak will likely take 30-50 minutes, go get a coffee) 2) Deploy via Bash Script (specifying configur

A colleague recently asked me how to create a custom function for Hive using Python. You can pretty much create a function in any language and plug it into your Hive query using the Hive TRANSFORM clause. TRANSFORM lets you add your own mappers and/or reducers to process the data. The example in this article is working code that I wrote a few years ago using an early version of Hive to demonstrate how to add a custom function. In earlier versions of Hive we had to implement our own functions to hash sensitive data for PII compliance. Beginning with Hive 1.3 the SHA2 UDF was added to calculate a hash using SHA-224, SHA-256, SHA-384, or SHA-512. In my example below I create a custom UDF using Python to calculate the SHA-256 hash for social security number. Keep in mind that when I did this there were no out of the box Hive UDF’s available. This example is to only demonstrate how to write your own custom functions for Hive using Python. First, we need to write some python code that will r

If the cluster has only one queue at root level named 'default' and is consuming 100% of the capacity, Ambari will create a queue named 'llap' when HSI is enabled for the 1st time, which is set to (depends on which value is smaller) either : - the minimum required %age for LLAP to work, or - at 20% of cluster's capacity. --------------------------------------------------------------------------------------------------------------------------------------------------------------------If this is not the case, where there are more than one queue in cluster, user will have to create/set the queue capacity %age in order to be used for LLAP app. Starting with minimum required for queue capacity (shown below), one can increase the queue %age size in order to add up the LLAP nodes in the cluster, as queue size is one of the primary drivers of how many Node Managers nodes will be running LLAP.Reference code for calculating minimum queue size.Following calculations can be a good reference in orde

Objective This tutorial demonstrates how to use the QueryDatabaseTable and PutKudu processors to read data from a MySQL database and put into Kudu. Thanks to @Cam Mach for his assistance with this article. Note: The PutKudu processor was introduced in NiFi 1.4.0.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.4.0 Apache Kudu 1.5.0 MySQL 5.7.13PutKudu (AvroReader)Demo ConfigurationMySQL Setup In your MySQL instance, choose a database ("nifi_db" in my instance) and create the table "users": unix> mysql -u root -punix> Enter password:<enter>mysql> use nifi_db;mysql>CREATE TABLE `users` ( `id` mediumint(9) NOT NULL AUTO_INCREMENT, `title` text, `first_name` text, `last_name` text, `street` text, `city` text, `state` text, `zip` text, `gender` text, `email` text, `username` text, `password` text, `phone` text, `cell` text, `ssn` text, PRIMARY KEY (`id`)) ENGINE=InnoDB AUTO_INCRE

I've seen several systems with 400+ million objects represented in the Namenode without issues. In my opinion, that's not the "right" question though. Certainly, the classic answer to small files has been the pressure it put's on the Namenode but that's only a part of the equation. And with hardware / cpu and increase memory thresholds, that number has certainly climbed over the years since the small file problem was documented.The better question is: How do small files "impact" cluster performance? Everything is a trade-off when dealing with data at scale. The impact of small files, beyond the Namenode pressures, is more specifically related to "job" performance.Under classic MR, the number of small files controls the number of mappers required to perform a job. Of course, there are tricks to "combine" inputs and reduce this, but that leads to a lot of data back planing and increased cluster I/O chatter. A mapper in the classic sense, is a costly resource to allocate. If the actual ta

By definition, Atlas provides a scalable and core foundational services for Data Governance - enabling enterprises to efficiently and effectively meet their compliance requirements with their Hadoop eco system. However, it is a complex application, which is built using the integration of various components in the Hadoop eco system. Below are the components involved in the Architecture: AtlasRangerHiveHBaseAmbari InfraKafkaThe intention of this article is to provide troubleshooting tips if Ambari install is not functioning correctly.Install validation and troubleshooting tips: Make sure Atlas Metadata Server and Atlas Metadata clients are installed from Ambari.Install Ranger Tagsync component if you wish to do authorization control using Atlas Tags in Ranger.Make sure the below Kafka topics are created in Kafka: ATLAS_HOOKATLAS_ENTITIES You can check this by using the following command on any of the kafka brokers: (In Kerberized cluster you need kafka key tab to run this command)/usr/hd

Update Feb 2018 - Updated article for HDF 3.1: https://community.hortonworks.com/articles/173816/automate-deployment-of-hdf-31-clusters-using-ambar.htmlSummary:Ambari blueprints can be used to automate setting up clusters. With Ambari support being added to HDF 2.0, the same can be done for HDF clusters running Nifi, Storm, Kafka. This article shows how you can use ambari-bootstrap project to easily generate a blueprint and deploy HDF clusters to both either single node or development/demo environments in 5 easy steps. If you prefer, a script is also provided at the bottom of the article that automates these steps, so you can deploy the cluster in a few commands. Tested with HDF 2.x and 3.0There is also a single node HDF 2.1 demo cluster available on AWS as an AMI which can be brought up in 10 min. Details herePrerequisite: A number of freshly installed hosts running CentOS/RHEL 6 or 7 where HDF is to be installedReminder: Do not try to install HDF on a env where Ambari or HDP are alre

Apache Ranger let's you control the users who can submit the jobs to Yarn queues. Demo 1 ) Default queue User demouser can submit jobs to default queue and can be controlled from Ranger UI Demo 2) Non-Default queue User demouser can submit jobs to spark queue and can be controlled from Ranger UI Happy Hadooping!! More information

Scenario: The cluster is using both Hive and Atlas components.Sometimes a simple query like 'show databases' fails with the error stack shown below:beeline> show databases;Getting log thread is interrupted, since query is done!Error: Error while processing statement: FAILED: Hive Internal Error: java.util.concurrent.RejectedExecutionException(Task java.util.concurrent.FutureTask@e871c01 rejected from java.util.concurrent.ThreadPoolExecutor@5b868755[Running, pool size = 1, active threads = 1, queued tasks = 10000, completed tasks = 14807]) (state=08S01,code=12)java.sql.SQLException: Error while processing statement: FAILED: Hive Internal Error: java.util.concurrent.RejectedExecutionException(Task java.util.concurrent.FutureTask@e871c01 rejected from java.util.concurrent.ThreadPoolExecutor@5b868755[Running, pool size = 1, active threads = 1, queued tasks = 10000, completed tasks = 14807])at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:282)at org.apache.hive.beeline.Co

When starting spark-shell, it tries to bind to port 4040 for the SparkUI.If that port is already taken because of another spark-shell session active, it tries then to bind on 4041, then 4042, etc.Each time the binding didn't suceed, there's a huge WARN stack trace which could be filtered[user@serv hive]$ SPARK_MAJOR_VERSION=2 spark-shell SPARK_MAJOR_VERSION is set to 2, using Spark2 Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 17/09/20 11:49:43 WARN AbstractLifeCycle: FAILED ServerConnector@2d258eff{HTTP/1.1}{0.0.0.0:4040}: java.net.BindException: Address already in use java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.spark_project.jetty.server

Please follow below steps for running SparkR script via Oozie.1. Install R packages on all the node managersyum -y install R R-devel libcurl-devel openssl-devel.2. Keep your R script readyHere is the sample scriptlibrary(SparkR) sc <- sparkR.init(appName="SparkR-sample") sqlContext <- sparkRSQL.init(sc) localDF <- data.frame(name=c("ABC", "blah", "blah"), age=c(39, 32, 81)) df <- createDataFrame(sqlContext, localDF) printSchema(df) sparkR.stop() .3. Create workflow.xmlHere is the working example:<workflow-app xmlns='uri:oozie:workflow:0.5' name='SparkFileCopy'> <global> <configuration> <property> <name>oozie.launcher.yarn.app.mapreduce.am.env</name> <value>SPARK_HOME=/usr/hdp/2.5.3.0-37/spark</value> </property> <property> <name>oozie.launcher.mapred.child.env</name> <value>SPARK_HOME=/usr/hdp/2.5.3.0-37/spark</value> </property> </configuration> </global> <start to='spa

When the FSImage file is large (like 30 GB or more), sometimes due to other contributing factors like RPC bandwidth, network congestion, request queue length etc, it can take a long time to upload/download. This in turn can leads the Zookeeper to believe that the NameNode is not responding. It displays the SERVICE_NOT_RESPONDING status. Thereafter, it triggers a failover transition.The logs display the following statements:2017-09-04 05:02:26,017 INFO namenode.TransferFsImage (TransferFsImage.java:receiveFile(575)) - "Combined time for fsimage download and fsync to all disks took 237.14s. The fsimage download took 237.14s at 141130.21 KB/s. Synchronous (fsync) write to disk of /opt/hadoop/hdfs/namenode/image/current/fsimage.ckpt_0000000012106114957 took 0.00s. Synchronous (fsync) write to disk of /var/hadoop/hdfs/namenode/image/current/fsimage.ckpt_0000000012106114957 took 0.00s..2017-09-04 05:02:26,018 WARN client.QuorumJournalManager (IPCLoggerChannel.java:call(388)) - Remote journal

Two YARN Queues have been setup - default, llap Whichever queue is marked as Hive LLAP Queue (Interactive Query Queue), that queue does not process any queries/jobThe following are the steps to reproduce the issye: There are two YARN Queues setup.In Ambari, default queue is selected for LLAP from the Interactive Query Queue drop down (Ambari > Hive > Configs > Interactive Query Queue dropdown).Any job/query submitted to default queue does not run. However, all queries submitted to LLAP queue runs successfully. Using the same process listed above, change the Interactive Query Queue to LLAP.Any query/job submitted to LLAP queue does not run. However, all queries submitted to default queue runs successfully.Basically, at any time, the LLAP queue does not run any job/query.Root Cause:This issue occurs when there is an issue with queue prioritization, where both queues have priority set to 0 which means both are of equal priority. This can be viewed from Ambari > Yarn Queue Man

Scenario 1: Only one instance of Spark Thrift Server is neededApproach:If you are installing the Spark Thrift Server on a Kerberos-secured cluster, the following instructions apply: The Spark Thrift Server must run in the same host as HiveServer2, so that it can access the hiveserver2 keytab.Edit permissions in /var/run/spark and /var/log/spark to specify read/write permissions to the Hive service account. /var/run/spark and /var/log/spark should be able read/write to hive. So, just seeing contents as user hive is not enough, you need to be able to write to those folders. One way is to give 77x permissions on these folders. Since spark:hadoop is owner:group and hive belongs to group hadoop, it will have write access with this setup.Use the Hive service account to start the thriftserver process.It is recommend that you run the Spark Thrift Server as user hive instead of user spark. This ensures that the Spark Thrift Server can access Hive keytabs, the Hive metastore, and data in HDFS th

1) Download the Ambari Integration module and HDFS transparency connector. You can get it here.2) Stop all the services from Ambari. Ambari -> Actions -> Stop All3) After successfully stopping all the services, Go to Spectrum Scale service and Unintegrate transparency. SpectrumScale -> Service Actions -> Unintegrate TransparencyThis step will replace the HDFS transparency modules with native HDFS and add back Secondary Namenode.4) Delete the Spectrum Scale service. Type "delete" to confirm deletion. SpectrumScale -> Service Actions -> Delete Service5) Extract the tar file downloaded in Step 1 in your Ambari Server node and run the mPack uninstaller../SpectrumScaleMPackUninstaller.pyThe uninstaller prompts for few values such as Ambari IP, username etc. Enter them.6) Performing above steps will delete the service from Ambari. Restart Ambari server and Start all services.Ambari -> Actions -> Start AllIf you were using a existing ESS clusters, the nodes would be st

Hadoop Auth [1] is a Java library which enables Kerberos SPNEGO authentication for HTTP requests. It enforces authentication on protected resources, after successful authentication Hadoop Auth creates a signed HTTP Cookie with an authentication token, username, user principal, authentication type and expiration time. This cookie is used for all subsequent HTTP client requests to access a protected resource until the cookie expires. Given Apache Knox's pluggable authentication providers it is easy to setup Hadoop Auth with Apache Knox with only few configuration changes. The purpose of this article to describe this process in detail and with examples. Assumptions: Here we are assuming that we have a working Hadoop cluster with Apache Knox ( version 0.7.0 and up [2] ) moreover the cluster is Kerberized. Kerberizing the cluster is beyond the scope of this article. Setup: To use Hadoop Auth in Apache Knox we need to update the Knox topology. Hadoop Auth is configured as a provider so

This article shows a simple NiFi data flow from the web to HDFS that demonstrates several fundamental capabilities of NiFi, including:iterating the flow from an external configuration filegroovy scripting to transform contentgroovy scripting that imports an external jar libraryNiFi expression languageThe article may be particularly useful for newcomers to NiFi.OverviewThe NiFi flow is shown below. At a very high-level, an InvokeHTTP processor will retrieve in sequence the RSS feeds from a list of URLs that are configured locally. RSS feeds return XML content and one feed URL returns multiple XML item elements, each representing one news story and structured by child elements like title, description, etc. A groovy script will transform each XML-formatted feed response to produce a set or records. Each record is one news story (item element) with title, description etc extracted from the XML as tab-separated values. Additionally, HTML formatting is stripped from those values. The t

Step 1: Take a backup of original configuration: [~]$ cd /var/lib/knox/data-2.4.0.0-169/security/keystores[~]$ mkdir backup[~]$ mv __gateway-credentials.jceks gateway.jks backup/ Step 2: Create a keystore in PKCS12 format from your private key file, certificate and root public certificate [~]$ openssl pkcs12 -export -out corp_cert_chain.pfx -inkey <private-key>.key-in <cert.cer> -certfile <root_intermediate>.cer Step 3: Generate knox keystore [~]$ cp corp_cert_chain.pfx /var/lib/knox/data-2.4.0.0-169/security/keystores/[~]$ cd /var/lib/knox/data-2.4.0.0-169/security/keystores/[~]$ keytool -importkeystore -srckeystore corp_cert_chain.pfx-srcstoretype pkcs12 -destkeystore gateway.jks -deststoretype jks -srcstorepass<src-keystore-password> -deststorepass <knox-master-secret>-srcalias <src-alias> -destalias gateway-identity -destkeypass <knox-master-secret> Step 4: Store the keystore password in jceks file [~]$ /usr/hdp/current/knox-server/bi

1. Introduction NiFi is a powerful and easy to use technology to build dataflows from diverse sources to diverse targets while transforming and dynamically routing in between. NiFi is packaged in HDF 2.0 which (in addition to bundling Kafka and Storm for a complete data movement platform) pushes NiFi to enterprise readiness with Ambari management and Ranger security and multitenancy. One of the hallmarks of NiFi is its awesome drag-and-drop UI which makes building and configuring dataflows drop dead easy. However, when using the same parts of a flow repeatedly across projects within a team or across the organization ... the UI can slow development down by forcing the same manual steps to make the same pieces from scratch. You can overcome this problem by using two features of NiFi -- templates and configurations using Expression Language references -- to build a library of reusable components that can be used as pre-built components to new flows. Doing so provides the following a

The Balancer runs in iterations for balancing a cluster. An iteration consists of four steps. We discuss each step in detail below.Step 1, Storage Group ClassificationBalancer first invokes the getLiveDatanodeStorageReport() rpc to the Namenode in order to get the storage report for all the storage devices in all Datanodes. The storage report contains storage utilization information such as capacity, DFS used space, remaining space, etc. for each storage device in each Datanode.A Datanode may contain multiple storage devices and the storage devices may have different storage types. A storage group G(i,T) is defined to be the group of all the storage devices with the same storage type T in Datanode i. For example, G(i,DISK) is the storage group of all the disk storage devices in Datanode i.For each storage type T in each Datanode i, Balancer computes Storage Group Utilization (%)U(i,T) = 100% * (storage group used space)/(storage group capacity),and Average Utilization (%)U(avg,T)

1 : Check if hbase-master is runningsudo /etc/init.d/hbase-master statusif not, then start it sudo /etc/init.d/hbase-master start2 : Check if hbase-regionserver is runningsudo /etc/init.d/hbase-regionserver statusif not, then start it sudo /etc/init.d/hbase-regionserver start3 : Check if zookeeper-server is runningsudo /etc/init.d/zookeeper-server statusif not, then start it sudo /etc/init.d/zookeeper-server start4: Grepping for open portnetstat -apn |grep <port to look for>5: Process memory usageps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -5orps -A --sort -rss -o comm,pmem | head -n 116: Free memory on system: CentOSfree -g7: Yarn application log checkingyarn logs -applicationId <application ID> dig deeper via suing the container id along with application idyarn logs -applicationId <application ID> -containerId <container id> 8: starting zkCli cli connection to servercd /grid/0/hdp/current/zookeeper-client/binAnd./zkCli.sh -server <zookeeper server

Manual install cluster:::Create VM’s:OS = RHEL6/RHEL7 (depending on ur HDP version)Java = OpenJDK8Set java home on all the nodes of cluster (make sure you have the correct java path first before setting it. The path value might differ based on what java subversion gets installed) export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk.x86_64Do the pass-word less ssh pick one node as amber host, create pub-pri key pair using ssh-keygen, then copy the public key into "/home/root/.ssh/authorized_keys" file on all the nodesDo test is password less ssh works (via doing ssh from the ambari node to all other nodes), if this fails, you need to resolve it first as the entire installation depends on it.NTP set-up (on all nodes) Yum install -y ntpStart ntp:"/etc/init.d/ntpd start" (RHEL6)"systemctl enable ntpd" and "systemctl start ntpd" (RHEL7)Check "/etc/hosts" file on all hosts: should have all node entries in it vi /etc/hostsEdit network file (on all nodes) vi /etc/sysconfig/networkNETWORKING=yesHO