Resources

Kerberos is a widely-used authentication system and is used throughout the Hadoop ecosystem, in particular, for strong authentication. The KDC, or key distribution center, is the name for the Kerberos server application that exposes the Authentication Service and Ticket Granting Service, as well as hosting the Kerberos principal database.Active Directory is used in many enterprises for Identity and Access Management. A common architecture in enterprise deployment scenarios is to make use of a local cluster KDC (often using the MIT-KDC packaging) to host the service and host principals associated with the cluster, and to configure a one-way trust between the AD domain and the cluster realm. This has the advantage of offloading Kerberos traffic from the domain controller(s) and not all enterprises do not necessarily want to host cluster principals within their AD domain. So how does a one-way trust work? The first thing to note is that the trust is instantiated by the existence of a spec

(I think) this occurred after upgrading from Ambari 2.4.2 to 2.5.0.3 (only looked for this when on Ambari 2.5.1.0)Here are the messages in SmartSense ViewFound these messages in /var/log/ambari-server/ambari-server.log when clicking on launch SmartSense View14 Jul 2017 11:14:05,523 ERROR [ambari-client-thread-467377] ServerProxy:162 - Failed to execute GET /recommendations. Reason: Connection refused14 Jul 2017 11:14:13,695 ERROR [ambari-client-thread-467507] ServerProxy:162 - Failed to execute GET /context. Reason: Connection refused14 Jul 2017 11:14:14,900 ERROR [ambari-client-thread-467507] ServerProxy:162 - Failed to execute GET /checkconfig. Reason: Connection refusedDiff'ing current hst-server.ini versus a previous (think it was from Ambari 2.4.2) yields# diff conf/hst-server.ini conf_13_04_17_04_17.save/2,4d1< enable.flex.subscription = false< smartsense.id=X-00000000-X-00000000< flex.subscription.id =6a4> smartsense.id=X-00000000-X-0000000027,28d24< url = http://

MiNiFi has http configuration Listener feature which can pull flow file configuration from the remote command control server, validate and apply the new flow configuration. The new file configuration should be a valid flow configuration yaml file.The way to configure the URL endpoint, pull interval and related security parameters.in minifi.properties nifi.configuration.listener.type=http nifi.configuration.listener.http.url=https://localhost:8080 nifi.configuration.listener.pull.interval=1 sec if you want to enable client certificate nifi.https.need.ClientAuth=true nifi.https.client.certificate=./conf/client.pem nifi.https.client.private.key=./conf/client.key nifi.https.client.pass.phrase=./conf/password nifi.https.client.ca.certificate=./conf/nifi-cert.pem

Apache NiFi 1.1.0 is now available and once again I want to test it in a isolated environment. Docker!The steps to do this are extremely similar to what has been detailed here (https://community.hortonworks.com/articles/69043/launching-a-nifi-docker-instance.html)Pull the imagedocker pull sunileman/nifi1.1.0You may find the mirror site is not optimal based on your location. Go here and grab your mirror site. Update MIRROR_SITE parameter in the Dockerfile with your mirror site. The Dockerfile is available here. If you update the Dockerfile you will have to build an image. Do this by runningdocker build --no-cache -t sunileman/nifi1.1.0 .Whether you pulled the image or built a new one, run this to launch Apache NiFi 1.1.0docker run -it --rm -p 8080-8081:8080-8081 sunileman/nifi1.1.0NiFi UI should be available herehttp://localhost:8080/nifi/Have fun!

For a UI showing the biggest consumers of space in HDFS install and configure Twitter's HDFS-DU. For a quick visual representation of HDFS disk usage with no extra tools required, use this script:#!/usr/bin/env bashmax_depth=5largest_root_dirs=$(hdfs dfs -du -s '/*' | sort -nr | perl -ane 'print "$F[1] "')printf "%15s %s\n" "bytes" "directory"for ld in $largest_root_dirs; do printf "%15.0f %s\n" $(hdfs dfs -du -s $ld| cut -d' ' -f1) $ld all_dirs=$(hdfs dfs -ls -R $ld | egrep '^dr........' | perl -ane "scalar(split('/',\$_)) <= $max_depth && print \"\$F[7]\n\"" ) for d in $all_dirs; do line=$(hdfs dfs -du -s $d) size=$(echo $line | cut -d' ' -f1) parent_dir=${d%/*} child=${d##*/} if [ -n "$parent_dir" ]; then leading_dirs=$(echo $parent_dir | perl -pe 's/./-/g; s/^.(.+)$/\|$1/') d=${leading_dirs}/$child fi printf "%15.0f %s\n" $size $d donedoneSample output: bytes directory 480376973

This article will help to configure the ranger audit logs to be written into flat file system.some users don't want to use solr to reduce the hardware and software footprints, such cases it will help to write and debug, at the same time this can coexists with solrNiFi is log consolidation done by Logback, hence we need to make the following changes to logback configuration.To enable the ranger audits :in Advanced-nifi-ranger-audit section make the flowing parameters values to,xasecure.audit.destination.log4j=truexasecure.audit.destination.log4j.logger=ranger.audit To capture the logs generated by the logger, configure the logback (same as nifi-app module logger).In Advanced nifi-node-logback-env at add the following content logback.xml template<appender name="RANGER_AUDIT" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>${org.apache.nifi.bootstrap.config.log.dir}/ranger_nifi_audit.log</file> <rollingPolicy class="ch.qos.logback.core.ro

IntroductionThe objective of this article is to describe how use “DataImport” tool inside Apache Solr to index an Oracle DB table.Assumptions & DesignIn this exercise, I’ve used two virtual machines: 1-Oracle Database App Development VM – 2GB RAM 2-Linux Centos VM – 3GB RAM (Solr node) Please note that: Hortonworks HDP Sandbox comes with out-of-the-box Solr service that can be easily provisioned or enabled and used as well for this exercise through Ambari UI, instead of installing Solr service on a standalone node.Oracle side- Create a dummy table with the following structure:[1]- Insert some sample data into the created table.On Solr Node#yum install java-1.8.0-openjdk.x86_64#java -version#wget http://apache.org/dist/lucene/solr/6.6.0/solr-6.6.0.tgz#tar xzf solr-6.6.0.tgz solr-6.6.0/bin/install_solr_service.sh --strip-components=2#sudo bash ./install_solr_service.sh solr-6.6.0.tgz#sudo service solr restart you should see something like followingStarted Solr server on port 8983 (p

Ambari Backup and Restore for PostgresSummary:Ambari server stores cluster configurations in a database. The default database for the Ambari install is Postgres. This document goes through a step by step process for backing up and restoring the Postgres database.Background:A backup copy of your Ambari database is good insurance to keep your Ambari server running. Anytime you make changes to Ambari you should create a backup of the Ambari database before you make the changes. You will want to do this if you need to restore your cluster to the original state. Examples of what you want to take backups are before cluster upgrasdes, changing service configurations, adding new servers, etc.Backing up the server configuration: 1.Login to the server running the Ambari Server process. You will need to have the privileges to start/stop the Ambari server and read the Postgres database.2.Stop the Ambari server using the following command:ambari-server stop 3.Create a directory to hold the databas

GoalAccess Hive through R shell with RhiveAuthentication mechanism: Kerberos AssumptionsHave a HDP cluster up and runningHave configured and installed RCluster is configured on CentOS7Need to have openssl-devel installed on the server where R is installedIf using openjdk, ensure that the JAVA_HOME is set to the path where javac is availableStepsDownload Rservewget https://rforge.net/Rserve/snapshot/Rserve_1.8-5.tar.gzUse R to install RserveR CMD INSTALL Rserve_1.8-5.tar.gzDownload rhive librarygit clone https://github.com/nexr/RHive.gitaSet the appropriate path for HIVE_HOME and HADOOP_HOMEexport HADOOP_HOME=/usr/hdp/2.5.0.0-1245/hadoop; export HIVE_HOME=/usr/hdp/2.5.0.0-1245/hiveaChange directory to RHivecd RHiveUse ant to buildant buildUse R to build RHive, this will build a file within the same directoryR CMD build RHiveUse the newly created file name to install RHive libraryR CMD INSTALL RHive_2.0-0.10.tar.gzDemoLaunch R using "R" at the unix promptLoad RHive library> library(RH

Ambari 2.4 Kerberos with FreeIPAThis tutorial describes how to enable Kerberos using a FreeIPA server for LDAP and KDC functions on HDP 2.5. The following assumptions are made: An existing HDP 2.5 clusterNo existing IPA serverThere are sufficient resources to create an m3.medium VM to house the FreeIPA serverDNS is already taken care of in the environmentFreeIPA will run on RHEL/CentOS 7Step 1: Setup FreeIPA ServerInstall Entropy Tools Certain operations like generating encryption keys host entropy for creating random data. A fresh system with no processes running and no real device drivers can have issues generating enough random data for these types of operations. Install the rng-tools package and start rngd to help with this issue:yum -y install rng-toolssystemctl start rngdsystemctl enable rngdInstall FreeIPA ServerInstall NTP and the FreeIPA software and start the NTP service:yum -y install ntp ipa-server ipa-server-dnssystemctl enable ntpdsystemctl start ntpdIn order to use FreeI

If you're not using Tez UI to debug your Hive queries you need to start today. Ambari installs Tez UI by default, but some people need a standalone option. Tez UI is easy to install and can go anywhere, meaning there's no reason for you to wonder why that Hive query won't run faster. Why You Should Use Tez UI:Whether in Ambari or Standalone, here's some of the things you're missing if you don't use Tez UI:1. Inventory of all queries executed, with search and filter. 2. Drill into a query and see the actual query text. Great if a BI tool is generating long-running queries. 3. Vertex Swimlane. This one is new in HDP 2.5 but it helps you quickly pinpoint the long-running parts of your query and optimize them.How To Install Tez UI Standalone:The Tez UI Installation Instructions give the details on installing Tez UI, but it can be a bit hard to put the pieces together. In this How-To we will cover installing Tez UI into Apache httpd on an HDP 2.5 installation. The installation instructions

I noticed that using Ambari experimental functionality we can do lot of interesting stuff. For e.g. rolling/express upgrade is in progress and some service checked failed, to recover from failure we need to update some configuration parameters and restart that particular service, during upgrade Ambari will not allow us to modify any configuration property, so either we can use configs.sh script or another option is to use ambari experimental functionality and enable "opsDuringRollingUpgrade" option 🙂Please feel free add more information about Ambari experimental functionality here!

SymptomsA Spark job fails with INTERNAL_FAILURE. In the WA (Workload Analytics) page of the job that failed, the following message is reported: org.apache.spark.SparkException: Application application_1503474791091_0002 finished with failed status  DiagnosisAs the Telemetry Publisher didn't retrieve the application log due to a known bug, we have to diagnose the application logs (application_1503474791091_0002) directly, which are stored in the user's S3 bucket. If the following exception is found, it indicates that the application failed to resolve a dependency in the Hadoop class path: 17/08/24 13:13:33 INFO ApplicationMaster: Preparing Local resourcesException in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.tracing.TraceUtils.wrapHadoopConf(Ljava/lang/String;Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/htrace/core/HTraceConfiguration; at org.apache.hadoop.fs.FsTracer.get(FsTracer.java:42) at org.apache.hadoop.hdfs.DFSClient.<init>

This article aims to show how to planning a Nifi Cluster following the best practices.1) Hardware Provisioning2) Hardware Considerations for HDF- General HardwareA key design point of NiFi is to use typical enterprise class application servers.Hardware failure: Nodes are typically configured with RAID 5/10 to handle hardware failures through replication, redundancyMore nodes means less impact from failure. More nodes provide increased throughput.- Machine ClassA NiFi cluster consists of a single class of machineBalanced NiFi Node: 8 CPU cores per node minimum. 6 Hard disks per node minimum, Spinning or SSD base on throughput requirements. 8 GB of RAM per node minimum. Designed for availability. Typical enterprise class application server. Resilience built into the server itself (RAID) Cost reduced where possible to strike proper price/performance ratio owing to volume.- NetworkingNetwork decisions play a role due to the clustered nature of data processing.In-rack backplane/Top-of-rack

Adding TTL on Solr: cd to this directoryStep1:Step2:Step3:vi managed-schema: add these 3 lines<field name="_timestamp_" type="date" indexed="true" stored="true" multiValued="false" /> <field name="_ttl_" type="string" indexed="true" multiValued="false" stored="true" /> <field name="_expire_at_" type="date" multiValued="false" indexed="true" stored="true" />Step4:vi solrconfig.xmlReplace the below 3 lines with the lines after it: <updateRequestProcessorChain name="add-unknown-fields-to-the-schema"> <!-- UUIDUpdateProcessorFactory will generate an id if none is present in the incoming document --> <processor class="solr.UUIDUpdateProcessorFactory"/>as<updateRequestProcessorChain name="add-unknown-fields-to-the-schema"> <processor class="solr.TimestampUpdateProcessorFactory"> <str name="fieldName">_timestamp_</str> </processor> <processor class="solr.DefaultValueUpdateProcessorFactory"> <str name=

When importing data from Oracle using Sqoop, it fails with the following error:Error: java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: ORA-09817: Write to audit file failed.Linux-x86_64 Error: 28: No space left on deviceAdditional information: 12ORA-02002: error while writing to audit trailCause:This issue occurs when there is no sufficient space in the /var/log/audit directory on the Oracle server.Solution:To resolve the issue, clear the space on the /var/log/audit directory of the Oracle server.

This article helps with configuration of HDF processors to integrate with HDP components Integration for puthivestreaming processor: Pre-requisites: In HDP ambari, enable the below properties for hive. hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager hive.compactor.initiator.on = truehive.compactor.worker.threads > 0 BUG: HiveStreaming processor not picking the updated hive-meta store principal from hive-site. xml Resolution: 1. Copy hive-site.xml, core-site.xml, hdfs-site.xml to the conf directory of NiFi2. Clear the Hive Configuration Resources property3. Create an ExecuteStream processor on the canvas scheduled to run as often as the ticket needs to be refreshed (every hour should do for most setups) with the following groovy script, replacing nifi@HDF.COM with your principal and /etc/nifi.headless.keytab with your keytab Steps: Create /etc/hdp/tgt.groovy and copy the code below into the file on all nifi nodes. Note: Change the kerberos principal and keytab locat

PAM authentication for HiveIn some situations an organization does not have an LDAP server and does not want to use Kerberos for the authentication, since this complicates the integration with third-party software. A quick alternative is PAM authentication which is in Hive since HDP2.2. It uses the OS users and passwords of the host running the hiveserver2 for authentication. The examples below are done using HDP 2.2/2.3 and Redhat. 1. Install JPam - Download JPam library and unzip on the hive host: JPam Libraries- Copy the .so file into the library path of the hive server. An easy way to find this is to use ps -ef | grep hiveserver2 and find the -Djava.library.path variable. For example copy the .so file to the following folder: /usr/hdp/2.2.4.2-2/hadoop/lib/native/Linux-amd64-64 2. Make shadow file accessible to hive user /etc/shadow needs to be readable by hive:chgrp hive /etc/shadow chmod 550 /etc/shadow The wiki entry also says to make /etc/login.defs accessible but this doesn't s

Part 1: https://community.hortonworks.com/articles/82964/getting-started-with-apache-ambari-workflow-design.htmlPart 2: https://community.hortonworks.com/articles/82967/apache-ambari-workflow-designer-view-for-apache-oo.htmlPart 3: https://community.hortonworks.com/articles/82988/apache-ambari-workflow-designer-view-for-apache-oo-1.htmlPart 4: https://community.hortonworks.com/articles/83051/apache-ambari-workflow-designer-view-for-apache-oo-2.htmlPart 5: https://community.hortonworks.com/articles/83361/apache-ambari-workflow-manager-view-for-apache-ooz.htmlPart 6: https://community.hortonworks.com/articles/83787/apache-ambari-workflow-manager-view-for-apache-ooz-1.htmlPart 7: https://community.hortonworks.com/articles/84071/apache-ambari-workflow-manager-view-for-apache-ooz-2.htmlPart 8: https://community.hortonworks.com/articles/84394/apache-ambari-workflow-manager-view-for-apache-ooz-3.htmlPart 9: https://community.hortonworks.com/articles/85091/apache-ambari-workflow-manager-view-f

While accessing psql for HAWQ, the command fails with the following error:[etl_user@hdpmaster2 ~]$ source /usr/local/hawq/greenplum_path.sh [etl_user@hdpmaster2 ~]$ psql psql: FATAL: no pg_hba.conf entry for host "[local]", user "etl_user", database "etl_user", SSL offThis issue occurs when the etl_user is missing under pg_hba.conf and Postgres database.To resolve the issue, create a new user from postgres by performing the following steps: Login as gpadmin, sudo su - gpadmin.Create new role and database:CREATE USER <user_name> SUPERUSER;CREATE DATABASE <user_name> WITH OWNER <database_name>;Edit pg_hba.conf under Master data directory and make the following changes:local all <user_name> trust host all <user_name> 0.0.0.0/0 trust host all <user_name> ::/0 trustRestart HAWQ cluster.

SymptomsAdding a cluster may fail with various error messages:Cluster creation failed.An internal error occurred while creating the cluster.Your quota allows for 0 more running instance(s). You requested at least 3. Your quota allows for 0 more running instance(s). You requested at least 1. Applies ToAdding a cluster in Cloudera Altus  CauseCertain error messages are straighforward and self-explains the cause of the issue.SSH Private Key information may be invalid during the process adding a cluster.  Troubleshooting StepsRegarding the quota error, you need to check with your Amazon EC2 admin to find out if the account has reached its limit. Please check below reference on how to check the limit.Make sure the SSH Private Key used is valid when adding the cluster. Please check below reference on how to create the key pair.Contact Cloudera Support for further help. ReferencesHow to create key pairI need more Amazon EC2 instances, but I've reached my limit

Many times we see some repeated logging inside our log files. For example in case of ambari-server.log we see the following kind of repeated logging inside the log. WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.ambari.server.api.services.StacksService.getStackArtifacts(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String), should not consume any entity.We might see the above kind of warning messages repeated many times.# grep 'public javax.ws.rs.core.Response org.apache.ambari.server.api.services.RequestService.getRequests' /var/log/ambari-server/ambari-server.log150- These are actually harmless WARNING messages, but many times it is desired to make sure that they are not logged, That way we can save some disk space issues and have a clean log.- Every time it is not possible to change the rootLogger to "ERROR" like following to avoid printing some INFO/WARNING messages, Because it will cause suppressing othe

IntroductionThe purpose of this article is to compare the upload time between three different methods for uploading same structured datasets into Hadoop (two methods) and MariaDB (one method). Assumptions & Designa small environment is used to deploy three node Hadoop cluster (one master node, two worker nodes). The exercise will be run from my laptop which has the following specs:Processor Name: Intel Core i7Processor Speed: 2.5 GHzNumber of Processors: 1Total Number of Cores: 4Memory: 16 GBThe Hadoop cluster will be virtualized on top of my Mac machine by “Oracle VM VirtualBox Manager”. The virtual Hadoop nodes running will have the following specs:Table 1: Hadoop Cluster –nodes’ specifications Specification Namenode (Master node) Datanode#1 (Worker node #1) Datanode #2 (Worker node #2) Hostname hdpnn.lab1.com hdpdn1.lab1.com hdpdn2.lab1.com Memory 4646 MB 3072 MB 3072 MB CPU Number 3 2 2 Hard disk size 20 GB 20 GB 20 GB

I have created the ER diagram for the complete list of Ambari database tables.It was taken from the Ambari 2.4.2.0 version. It might be helpful when troubleshooting ambari config issues.ambari-database-er-diagram.pdf

Objective This tutorial walks you through a NiFI flow that utilizes the PartitionRecord processor with GrokReader/JSONWriter controller services to parse the NiFi app log in Grok format, convert to JSON and then group the output by log level (INFO, WARN, ERROR). Note: The record-oriented processors and controller services were introduced in NiFi 1.2.0. As such, the tutorial needs to be done running Version 1.2.0 or later.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.3.0PartitionRecord FlowTemplate Import the template: partitionrecord-groktojson.xml The flow should appear as follows on your NiFi canvas: Enable Controller Services Select the gear icon from the Operate Palette: This opens the NiFi Flow Configuration window. Select the Controller Services tab: Enable AvroSchemaRegistry by selecting the lightning bolt icon/button. This will then allow you to enable the GrokReader and JSONRecordSetWr