Community Articles

For a UI showing the biggest consumers of space in HDFS install and configure Twitter's HDFS-DU. For a quick visual representation of HDFS disk usage with no extra tools required, use this script:#!/usr/bin/env bashmax_depth=5largest_root_dirs=$(hdfs dfs -du -s '/*' | sort -nr | perl -ane 'print "$F[1] "')printf "%15s %s\n" "bytes" "directory"for ld in $largest_root_dirs; do printf "%15.0f %s\n" $(hdfs dfs -du -s $ld| cut -d' ' -f1) $ld all_dirs=$(hdfs dfs -ls -R $ld | egrep '^dr........' | perl -ane "scalar(split('/',\$_)) <= $max_depth && print \"\$F[7]\n\"" ) for d in $all_dirs; do line=$(hdfs dfs -du -s $d) size=$(echo $line | cut -d' ' -f1) parent_dir=${d%/*} child=${d##*/} if [ -n "$parent_dir" ]; then leading_dirs=$(echo $parent_dir | perl -pe 's/./-/g; s/^.(.+)$/\|$1/') d=${leading_dirs}/$child fi printf "%15.0f %s\n" $size $d donedoneSample output: bytes directory 480376973

This article will help to configure the ranger audit logs to be written into flat file system.some users don't want to use solr to reduce the hardware and software footprints, such cases it will help to write and debug, at the same time this can coexists with solrNiFi is log consolidation done by Logback, hence we need to make the following changes to logback configuration.To enable the ranger audits :in Advanced-nifi-ranger-audit section make the flowing parameters values to,xasecure.audit.destination.log4j=truexasecure.audit.destination.log4j.logger=ranger.audit To capture the logs generated by the logger, configure the logback (same as nifi-app module logger).In Advanced nifi-node-logback-env at add the following content logback.xml template<appender name="RANGER_AUDIT" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>${org.apache.nifi.bootstrap.config.log.dir}/ranger_nifi_audit.log</file> <rollingPolicy class="ch.qos.logback.core.ro

IntroductionThe objective of this article is to describe how use “DataImport” tool inside Apache Solr to index an Oracle DB table.Assumptions & DesignIn this exercise, I’ve used two virtual machines: 1-Oracle Database App Development VM – 2GB RAM 2-Linux Centos VM – 3GB RAM (Solr node) Please note that: Hortonworks HDP Sandbox comes with out-of-the-box Solr service that can be easily provisioned or enabled and used as well for this exercise through Ambari UI, instead of installing Solr service on a standalone node.Oracle side- Create a dummy table with the following structure:[1]- Insert some sample data into the created table.On Solr Node#yum install java-1.8.0-openjdk.x86_64#java -version#wget http://apache.org/dist/lucene/solr/6.6.0/solr-6.6.0.tgz#tar xzf solr-6.6.0.tgz solr-6.6.0/bin/install_solr_service.sh --strip-components=2#sudo bash ./install_solr_service.sh solr-6.6.0.tgz#sudo service solr restart you should see something like followingStarted Solr server on port 8983 (p

Ambari Backup and Restore for PostgresSummary:Ambari server stores cluster configurations in a database. The default database for the Ambari install is Postgres. This document goes through a step by step process for backing up and restoring the Postgres database.Background:A backup copy of your Ambari database is good insurance to keep your Ambari server running. Anytime you make changes to Ambari you should create a backup of the Ambari database before you make the changes. You will want to do this if you need to restore your cluster to the original state. Examples of what you want to take backups are before cluster upgrasdes, changing service configurations, adding new servers, etc.Backing up the server configuration: 1.Login to the server running the Ambari Server process. You will need to have the privileges to start/stop the Ambari server and read the Postgres database.2.Stop the Ambari server using the following command:ambari-server stop 3.Create a directory to hold the databas

GoalAccess Hive through R shell with RhiveAuthentication mechanism: Kerberos AssumptionsHave a HDP cluster up and runningHave configured and installed RCluster is configured on CentOS7Need to have openssl-devel installed on the server where R is installedIf using openjdk, ensure that the JAVA_HOME is set to the path where javac is availableStepsDownload Rservewget https://rforge.net/Rserve/snapshot/Rserve_1.8-5.tar.gzUse R to install RserveR CMD INSTALL Rserve_1.8-5.tar.gzDownload rhive librarygit clone https://github.com/nexr/RHive.gitaSet the appropriate path for HIVE_HOME and HADOOP_HOMEexport HADOOP_HOME=/usr/hdp/2.5.0.0-1245/hadoop; export HIVE_HOME=/usr/hdp/2.5.0.0-1245/hiveaChange directory to RHivecd RHiveUse ant to buildant buildUse R to build RHive, this will build a file within the same directoryR CMD build RHiveUse the newly created file name to install RHive libraryR CMD INSTALL RHive_2.0-0.10.tar.gzDemoLaunch R using "R" at the unix promptLoad RHive library> library(RH

Ambari 2.4 Kerberos with FreeIPAThis tutorial describes how to enable Kerberos using a FreeIPA server for LDAP and KDC functions on HDP 2.5. The following assumptions are made: An existing HDP 2.5 clusterNo existing IPA serverThere are sufficient resources to create an m3.medium VM to house the FreeIPA serverDNS is already taken care of in the environmentFreeIPA will run on RHEL/CentOS 7Step 1: Setup FreeIPA ServerInstall Entropy Tools Certain operations like generating encryption keys host entropy for creating random data. A fresh system with no processes running and no real device drivers can have issues generating enough random data for these types of operations. Install the rng-tools package and start rngd to help with this issue:yum -y install rng-toolssystemctl start rngdsystemctl enable rngdInstall FreeIPA ServerInstall NTP and the FreeIPA software and start the NTP service:yum -y install ntp ipa-server ipa-server-dnssystemctl enable ntpdsystemctl start ntpdIn order to use FreeI

If you're not using Tez UI to debug your Hive queries you need to start today. Ambari installs Tez UI by default, but some people need a standalone option. Tez UI is easy to install and can go anywhere, meaning there's no reason for you to wonder why that Hive query won't run faster. Why You Should Use Tez UI:Whether in Ambari or Standalone, here's some of the things you're missing if you don't use Tez UI:1. Inventory of all queries executed, with search and filter. 2. Drill into a query and see the actual query text. Great if a BI tool is generating long-running queries. 3. Vertex Swimlane. This one is new in HDP 2.5 but it helps you quickly pinpoint the long-running parts of your query and optimize them.How To Install Tez UI Standalone:The Tez UI Installation Instructions give the details on installing Tez UI, but it can be a bit hard to put the pieces together. In this How-To we will cover installing Tez UI into Apache httpd on an HDP 2.5 installation. The installation instructions

I noticed that using Ambari experimental functionality we can do lot of interesting stuff. For e.g. rolling/express upgrade is in progress and some service checked failed, to recover from failure we need to update some configuration parameters and restart that particular service, during upgrade Ambari will not allow us to modify any configuration property, so either we can use configs.sh script or another option is to use ambari experimental functionality and enable "opsDuringRollingUpgrade" option 🙂Please feel free add more information about Ambari experimental functionality here!

SymptomsA Spark job fails with INTERNAL_FAILURE. In the WA (Workload Analytics) page of the job that failed, the following message is reported: org.apache.spark.SparkException: Application application_1503474791091_0002 finished with failed status  DiagnosisAs the Telemetry Publisher didn't retrieve the application log due to a known bug, we have to diagnose the application logs (application_1503474791091_0002) directly, which are stored in the user's S3 bucket. If the following exception is found, it indicates that the application failed to resolve a dependency in the Hadoop class path: 17/08/24 13:13:33 INFO ApplicationMaster: Preparing Local resourcesException in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.tracing.TraceUtils.wrapHadoopConf(Ljava/lang/String;Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/htrace/core/HTraceConfiguration; at org.apache.hadoop.fs.FsTracer.get(FsTracer.java:42) at org.apache.hadoop.hdfs.DFSClient.<init>

This article aims to show how to planning a Nifi Cluster following the best practices.1) Hardware Provisioning2) Hardware Considerations for HDF- General HardwareA key design point of NiFi is to use typical enterprise class application servers.Hardware failure: Nodes are typically configured with RAID 5/10 to handle hardware failures through replication, redundancyMore nodes means less impact from failure. More nodes provide increased throughput.- Machine ClassA NiFi cluster consists of a single class of machineBalanced NiFi Node: 8 CPU cores per node minimum. 6 Hard disks per node minimum, Spinning or SSD base on throughput requirements. 8 GB of RAM per node minimum. Designed for availability. Typical enterprise class application server. Resilience built into the server itself (RAID) Cost reduced where possible to strike proper price/performance ratio owing to volume.- NetworkingNetwork decisions play a role due to the clustered nature of data processing.In-rack backplane/Top-of-rack

Adding TTL on Solr: cd to this directoryStep1:Step2:Step3:vi managed-schema: add these 3 lines<field name="_timestamp_" type="date" indexed="true" stored="true" multiValued="false" /> <field name="_ttl_" type="string" indexed="true" multiValued="false" stored="true" /> <field name="_expire_at_" type="date" multiValued="false" indexed="true" stored="true" />Step4:vi solrconfig.xmlReplace the below 3 lines with the lines after it: <updateRequestProcessorChain name="add-unknown-fields-to-the-schema"> <!-- UUIDUpdateProcessorFactory will generate an id if none is present in the incoming document --> <processor class="solr.UUIDUpdateProcessorFactory"/>as<updateRequestProcessorChain name="add-unknown-fields-to-the-schema"> <processor class="solr.TimestampUpdateProcessorFactory"> <str name="fieldName">_timestamp_</str> </processor> <processor class="solr.DefaultValueUpdateProcessorFactory"> <str name=

When importing data from Oracle using Sqoop, it fails with the following error:Error: java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: ORA-09817: Write to audit file failed.Linux-x86_64 Error: 28: No space left on deviceAdditional information: 12ORA-02002: error while writing to audit trailCause:This issue occurs when there is no sufficient space in the /var/log/audit directory on the Oracle server.Solution:To resolve the issue, clear the space on the /var/log/audit directory of the Oracle server.

This article helps with configuration of HDF processors to integrate with HDP components Integration for puthivestreaming processor: Pre-requisites: In HDP ambari, enable the below properties for hive. hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager hive.compactor.initiator.on = truehive.compactor.worker.threads > 0 BUG: HiveStreaming processor not picking the updated hive-meta store principal from hive-site. xml Resolution: 1. Copy hive-site.xml, core-site.xml, hdfs-site.xml to the conf directory of NiFi2. Clear the Hive Configuration Resources property3. Create an ExecuteStream processor on the canvas scheduled to run as often as the ticket needs to be refreshed (every hour should do for most setups) with the following groovy script, replacing nifi@HDF.COM with your principal and /etc/nifi.headless.keytab with your keytab Steps: Create /etc/hdp/tgt.groovy and copy the code below into the file on all nifi nodes. Note: Change the kerberos principal and keytab locat

PAM authentication for HiveIn some situations an organization does not have an LDAP server and does not want to use Kerberos for the authentication, since this complicates the integration with third-party software. A quick alternative is PAM authentication which is in Hive since HDP2.2. It uses the OS users and passwords of the host running the hiveserver2 for authentication. The examples below are done using HDP 2.2/2.3 and Redhat. 1. Install JPam - Download JPam library and unzip on the hive host: JPam Libraries- Copy the .so file into the library path of the hive server. An easy way to find this is to use ps -ef | grep hiveserver2 and find the -Djava.library.path variable. For example copy the .so file to the following folder: /usr/hdp/2.2.4.2-2/hadoop/lib/native/Linux-amd64-64 2. Make shadow file accessible to hive user /etc/shadow needs to be readable by hive:chgrp hive /etc/shadow chmod 550 /etc/shadow The wiki entry also says to make /etc/login.defs accessible but this doesn't s

Part 1: https://community.hortonworks.com/articles/82964/getting-started-with-apache-ambari-workflow-design.htmlPart 2: https://community.hortonworks.com/articles/82967/apache-ambari-workflow-designer-view-for-apache-oo.htmlPart 3: https://community.hortonworks.com/articles/82988/apache-ambari-workflow-designer-view-for-apache-oo-1.htmlPart 4: https://community.hortonworks.com/articles/83051/apache-ambari-workflow-designer-view-for-apache-oo-2.htmlPart 5: https://community.hortonworks.com/articles/83361/apache-ambari-workflow-manager-view-for-apache-ooz.htmlPart 6: https://community.hortonworks.com/articles/83787/apache-ambari-workflow-manager-view-for-apache-ooz-1.htmlPart 7: https://community.hortonworks.com/articles/84071/apache-ambari-workflow-manager-view-for-apache-ooz-2.htmlPart 8: https://community.hortonworks.com/articles/84394/apache-ambari-workflow-manager-view-for-apache-ooz-3.htmlPart 9: https://community.hortonworks.com/articles/85091/apache-ambari-workflow-manager-view-f

While accessing psql for HAWQ, the command fails with the following error:[etl_user@hdpmaster2 ~]$ source /usr/local/hawq/greenplum_path.sh [etl_user@hdpmaster2 ~]$ psql psql: FATAL: no pg_hba.conf entry for host "[local]", user "etl_user", database "etl_user", SSL offThis issue occurs when the etl_user is missing under pg_hba.conf and Postgres database.To resolve the issue, create a new user from postgres by performing the following steps: Login as gpadmin, sudo su - gpadmin.Create new role and database:CREATE USER <user_name> SUPERUSER;CREATE DATABASE <user_name> WITH OWNER <database_name>;Edit pg_hba.conf under Master data directory and make the following changes:local all <user_name> trust host all <user_name> 0.0.0.0/0 trust host all <user_name> ::/0 trustRestart HAWQ cluster.

SymptomsAdding a cluster may fail with various error messages:Cluster creation failed.An internal error occurred while creating the cluster.Your quota allows for 0 more running instance(s). You requested at least 3. Your quota allows for 0 more running instance(s). You requested at least 1. Applies ToAdding a cluster in Cloudera Altus  CauseCertain error messages are straighforward and self-explains the cause of the issue.SSH Private Key information may be invalid during the process adding a cluster.  Troubleshooting StepsRegarding the quota error, you need to check with your Amazon EC2 admin to find out if the account has reached its limit. Please check below reference on how to check the limit.Make sure the SSH Private Key used is valid when adding the cluster. Please check below reference on how to create the key pair.Contact Cloudera Support for further help. ReferencesHow to create key pairI need more Amazon EC2 instances, but I've reached my limit

Many times we see some repeated logging inside our log files. For example in case of ambari-server.log we see the following kind of repeated logging inside the log. WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.ambari.server.api.services.StacksService.getStackArtifacts(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String), should not consume any entity.We might see the above kind of warning messages repeated many times.# grep 'public javax.ws.rs.core.Response org.apache.ambari.server.api.services.RequestService.getRequests' /var/log/ambari-server/ambari-server.log150- These are actually harmless WARNING messages, but many times it is desired to make sure that they are not logged, That way we can save some disk space issues and have a clean log.- Every time it is not possible to change the rootLogger to "ERROR" like following to avoid printing some INFO/WARNING messages, Because it will cause suppressing othe

IntroductionThe purpose of this article is to compare the upload time between three different methods for uploading same structured datasets into Hadoop (two methods) and MariaDB (one method). Assumptions & Designa small environment is used to deploy three node Hadoop cluster (one master node, two worker nodes). The exercise will be run from my laptop which has the following specs:Processor Name: Intel Core i7Processor Speed: 2.5 GHzNumber of Processors: 1Total Number of Cores: 4Memory: 16 GBThe Hadoop cluster will be virtualized on top of my Mac machine by “Oracle VM VirtualBox Manager”. The virtual Hadoop nodes running will have the following specs:Table 1: Hadoop Cluster –nodes’ specifications Specification Namenode (Master node) Datanode#1 (Worker node #1) Datanode #2 (Worker node #2) Hostname hdpnn.lab1.com hdpdn1.lab1.com hdpdn2.lab1.com Memory 4646 MB 3072 MB 3072 MB CPU Number 3 2 2 Hard disk size 20 GB 20 GB 20 GB

I have created the ER diagram for the complete list of Ambari database tables.It was taken from the Ambari 2.4.2.0 version. It might be helpful when troubleshooting ambari config issues.ambari-database-er-diagram.pdf

Objective This tutorial walks you through a NiFI flow that utilizes the PartitionRecord processor with GrokReader/JSONWriter controller services to parse the NiFi app log in Grok format, convert to JSON and then group the output by log level (INFO, WARN, ERROR). Note: The record-oriented processors and controller services were introduced in NiFi 1.2.0. As such, the tutorial needs to be done running Version 1.2.0 or later.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.3.0PartitionRecord FlowTemplate Import the template: partitionrecord-groktojson.xml The flow should appear as follows on your NiFi canvas: Enable Controller Services Select the gear icon from the Operate Palette: This opens the NiFi Flow Configuration window. Select the Controller Services tab: Enable AvroSchemaRegistry by selecting the lightning bolt icon/button. This will then allow you to enable the GrokReader and JSONRecordSetWr

Ambari database cleanup - Speed upambari-server db-cleanup -d 2016-09-30 --cluster-name=TESTHDPI have ran this on a ambari database used by on one of the 500 node cluster HDP cluster. It ran for more than 15 hours but without any success. I analyzed the ambari server logs to check where it is taking most of the time, it seems to be spending more time on batch deletes on ambari.alert_notice, ambari.alert_current ,ambari.alert_history table.To improve the performance of the db cleanup, I have created the index on the ambari.alert_notice table..-bash-4.1$ psql -U ambari -d ambari Password for user ambari: psql (8.4.20)Type "help" for help.ambari=> CREATE INDEX alert_notice_idx ON ambari.alert_notice(history_id);After this i ran re-loaded my ambari database from the backup and ran the db-cleanup, it took only less than 2 min to complete the cleanup.Also to reclaim the disk space and reindex after the cleanup, i ran the following commands as super user "postgres"Vacuum full;reindex da

This article explains on how to build a spark application in eclipse that can run in local mode on a kerberized hadoop cluster.Environment:Eclipse : Scala IDERequisites:krb5.conf Copy /etc/krb5.conf from any one of the node of your cluster where keberos client is installed to local machinecore-site.xmlCopy core-site.xml from /usr/hdp/current/conf/core-site.xml to local machinehdfs-site.xmlCopy hdfs-site.xml from /usr/hdp/current/conf/core-site.xml to local machineyarn-site.xmlCopy yarn-site.xml from /usr/hdp/current/conf/core-site.xml to local machinekeytabCopy keytab file of the user from cluster to local machine. This keytab file should be of the corresponding user principal.Steps:1. Add namenode(s) hostname to host file on local machine(In case of HA, add both namenodes to host file).2. Add hostname of KDC server to host file on local machine.3. Right click on your project in Eclipse and add source folder, rename it to resources.4. Copy core-site.xml, hdfs-site.xml and yarn-site.xml

Copy, paste and run the following code: val data = Array(1, 2, 3, 4, 5) // create Array of Integersval dataRDD = sc.parallelize(data) // create an RDDval dataDF = dataRDD.toDF() // convert RDD to DataFramedataDF.write.parquet("data.parquet") // write to parquetval newDataDF = sqlContext. read.parquet("data.parquet") // read back parquet to DFnewDataDF.show() // show contents If you run this code in a Zeppelin notebook you will see the following output data: data: Array[Int] = Array(1, 2, 3, 4, 5)dataRDD: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:31dataDF: org.apache.spark.sql.DataFrame = [_1: int]newDataDF: org.apache.spark.sql.DataFrame = [_1: int]+---+| _1|+---+| 1|| 2|| 3|| 4|| 5|+---+

If you need to install the specific version of HDP using Blue prints use the Following steps. Install ambari-server and ambari-agent and register it.Use the appropriate os version. Here I have used the os version as centos/redhat 7 Make sure you do it on a clean host with no existing hadooppackages etc.Create 4 files under say /root [root@vijaybluetest1 ~]# lsblueprint.json cluster.json repo.json utils.jsonblueprint.json :- Contains cluster definition.1) cat blueprint.json{ "host_groups" : [ { "name":"host_group_1", "components":[ { "name" : "NODEMANAGER" }, { "name" : "DATANODE" }, { "name" : "ZOOKEEPER_SERVER" }, { "name" : "HISTORYSERVER" }, { "name" : "JOURNALNODE" }, { "name" : "APP_TIMELINE_SERVER" }, { "name" : "RESOURCEMANAGER" }, { "name" : "MAPREDUCE2_CLIENT" }, { "name" : "YARN_CLIENT" }, { "name" : "HDFS_CLIENT" }, { "name" : "ZOOKEEPER_CLIENT" }, { "name" : "NAMENODE" }, { "name" : "SECONDARY_NAMENODE"} ], "cardinality":"1" } ], "Blueprints" : {