Resources

How you can easily create a Kibana dashboard for monitoring your Metron deploymentSuppose you have all your Metron topologies up-and-running and error free. Your work is not finished then. Metron is meant to be a (near) real time alerting system, so how do you guarantee that events flow through the Metron chain of Kafka topics and Storm topologies in a timely manner? What if some events belonging to some threat are hung up in one of the Kafka topics? How to keep track of the topology performance when the throughput of Metron increases over time? The way to check this is to constantly monitor the consumer lags of the Kafka topics, like this:**/usr/hdp/2.5.3.0-37/kafka/bin/kafka-consumer-offset-checker.sh --zookeeper $ZOOKEEPER_HOST:2181 --security-protocol SASL_PLAINTEXT --topic indexing --group indexing[2017-10-09 21:10:20,278] WARN WARNING: ConsumerOffsetChecker is deprecated and will be dropped in releases following 0.9.0. Use ConsumerGroupCommand instead. (kafka.tools.ConsumerOffset

This article describes the steps to add Spectrum Scale service to HDP cluster. We will be using an existing ESS cluster.Pre-Requisites:1) Download the Ambari Integration module and HDFS transparency connector. You can get it here.2) Collect the details of the existing ESS cluster. (IPs /hostnames/ public key)3) Install required packages yum -y install kernel-devel cpp gcc gcc-c++ binutils ksh libstdc++ libstd++-devel compact-libstdc++ imake make ncNote: Make sure that the kernel and kernel-devel version are the same.4) Ensure that you setup password-less ssh from ambari-server to the ESS cluster nodes and from ESS node to the all the nodes in the cluster.5) On the ambari server node, create a file called shared_gpfs_node.cfg under "/var/lib/ambari-server/resources/" directory and add the FQDN of any node in the ESS cluster. Make sure you add only one FQDN and password-less ssh is setup to this node from ambari server node.Note: Add the mapping in /etc/hosts for the FQDN aboveInstalling

IntroductionThis is part 3 of a series of articles on Data Enrichment with NiFi: Part 1: Data flow enrichment with LookupRecord and SimpleKV Lookup Service is available herePart 2: Data flow enrichment with LookupAttribute and SimpleKV Lookup Service is available herePart 3: Data flow enrichment with LookupRecord and MongoDB Lookup Service is available hereEnrichment is a common use case when working on data ingestion or flow management. Enrichment is getting data from external source (database, file, API, etc) to add more details, context or information to data being ingested. In Part 1 I showed how to use LookupRecord to enrich the content of a flow file. This is a powerful feature of NiFi based on the record based paradigm. For some scenarios we want to enrich the flow file by adding the result of the lookup as an attribute and not to the content of the flow file. For this, we can use LookupAttribute with a LookupService.ScenarioWe will be using the same retail scenario of the previ

Please follow below steps to modify quicklinks for Oozie service in Ambari Note - This tutorial has been successfully tried and tested on Ambari 2.4.2.0 and Ambari 2.5.2.01. Please make sure that your /var/lib/ambari-server/resources/stacks/HDP/2.0.6/services/OOZIE/metainfo.xml looks like below.<?xml version="1.0"?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for

IntroductionThis is a continuation of an article I wrote about 1 year ago: https://community.hortonworks.com/articles/60580/jmeter-setup-for-hive-load-testing-draft.htmlhttps://www.blazemeter.com/blog/windows-authentication-apache-jmeterSteps1) Enable Kerberos on your clusterPerform all steps specified here: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_security/content/configuring_amb_hdp_for_kerberos.html and connect successfully to hive service via command line using your user keytab. That implies a valid ticket.2) Install JMeterSee previous article mentioned in Introduction.3) Set Hive User keytab in jaas.confJMETER_HOME/bin/jaas.confYour jaas.conf should look something like this:JMeter {com.sun.security.auth.module.Krb5LoginModule requireduseTicketCache=falsedoNotPrompt=trueuseKeyTab=truekeyTab="/etc/security/keytabs/hive.service.keytab"principal="hive/server.example.com@EXAMPLE.COM"debug=true;};4) JMeter SetupThere are 2 files under /bin folder of the JMeter install

In preparing for my talk at DataWorksSummit in Australia, I wanted to try yet another way to integrate Apache NiFi with TensorFlow. The common ways being: calling a Python TensorFlow script from Execute Command, calling a TensorFlow Serving server via gRPC, calling a C++ TensorFlow executable via Execute Command, running TensorFlow on the edge and having MiniFi send it to NiFi or calling a TensorFlowOnSpark job via Kafka, Site-to-Site.TensorFlow has released a Java API (https://www.tensorflow.org/install/install_java), so I decided to write a quick custom processor to run TensorFlow Inception v3.It's a simple set of dependencies for Maven: <dependencies> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-api</artifactId> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-processor-utils</artifactId>

IntroductionThe term “Load Testing” has evolved over the years, however the core meaning still comes down to making sure that your system can handle a predefined amount of users at the same time. The load test is usually run over an extended period of time in a staggered manner, slowly ramping up the amount of users until you hit a predefined maximum that’s usually based on projected usage levels extrapolated from access logs with a buffer for surges. The goal of load testing is to determine the number of users that the system can typically cope with, this is called the systems “concurrency level” and give you a hard number to work with when dealing with capacity planning, performance and optimization, stability, SLAs etc.JMeter was originally designed for testing Web Applications but has since expanded to other test functions, including database via JDBC. As such, it has the ability to execute SQL queries against a given JDBC Driver. JMeter allows to define queries to execute against

This is part 1 of a miniseries. Part 2 can be found here.How do you count the number of lines in a FlowFile in NiFi? The naïve answer would be to run wc -l inside an ExecuteStreamCommand processor. This is unsatisfying because it requires firing up a shell command in a separate process. This is a scalability issue and also a security risk. But, there is a better way!The Record processors of NiFi 1.2+ offer the possibility to run real time, in-stream SQL against FlowFiles. Why not use this capability? Here's the approach:I am going to define a minimalistic schema that allows me to read each line of a CSV file as a record with one single field. Here is my example flow:Let's go through the steps. First, I use a GetFile processor to pick up files in my home directory. The actual configuration does not matter - use whatever your data source is. These data are passed to a QueryRecord processor, which is where the magic happens.My Query Record processor is configured like this:For reading the

Part 1: https://community.hortonworks.com/articles/82964/getting-started-with-apache-ambari-workflow-design.htmlPart 2: https://community.hortonworks.com/articles/82967/apache-ambari-workflow-designer-view-for-apache-oo.htmlPart 3: https://community.hortonworks.com/articles/82988/apache-ambari-workflow-designer-view-for-apache-oo-1.htmlPart 4: https://community.hortonworks.com/articles/83051/apache-ambari-workflow-designer-view-for-apache-oo-2.htmlPart 5: https://community.hortonworks.com/articles/83361/apache-ambari-workflow-manager-view-for-apache-ooz.htmlPart 6: https://community.hortonworks.com/articles/83787/apache-ambari-workflow-manager-view-for-apache-ooz-1.htmlPart 8: https://community.hortonworks.com/articles/84394/apache-ambari-workflow-manager-view-for-apache-ooz-3.htmlPart 9: https://community.hortonworks.com/articles/85091/apache-ambari-workflow-manager-view-for-apache-ooz-4.htmlPart 10: https://community.hortonworks.com/articles/85354/apache-ambari-workflow-manager-view-

Cloudbreak contains mini KNOX which is not managed by Ambari.Below are the steps to replace Self Signed Certificate with CA Signed CertificatesStep 1: Remove below two entries from /usr/hdp/current/knox-server/conf/gateway-site.xml and save it.<property> <name>gateway.signing.keystore.name</name> <value>signing.jks</value> </property> <property> <name>gateway.signing.key.alias</name> <value>signing-identity</value> </property> Step 2: Take a backup of original configuration:[~]$ cd /usr/hdp/current/knox-server/data/security/keystores/[~]$ mkdir backup[~]$ mv __gateway-credentials.jceks gateway.jks backup/Step 3: Create a keystore in PKCS12 format from your private key file, certificate, Intermediate certificate and root certificate[~]$ openssl pkcs12 -export -out corp_cert_chain.pfx -inkey <private-key>.key -in <cert.cer> -certfile <root_intermediate>.cer -certfile <root_ca>.cerStep 4: Regen

Objective This tutorial is the second article of a two part series. We will walk-through in detail a flow which: Uses the LookupRecord processor to parse NiFi provenance events in JSON format and add geolocation data Uses the PartitionRecord processor to group like records by State Publishes records originating from California to Kafka The first article can be found here. Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.3.0 Apache Kafka 0.10.2.1 NiFi LookupRecord Flow Walk Through Prerequisites The first article in this tutorial series has been completed. The file GeoLite2-City.mmdb has been downloaded locally into a directory named "enrichment-db" within your NiFi installation Provenance data has been generated and received by the "Provenance In" input port. All necessary controller services and reporting tasks are enabled/running. Kafka is running and a topic "California" has been created.

Objective This tutorial walks you through a NiFi flow that: Uses the LookupRecord processor to parse NiFi provenance events in JSON format and add geolocation data Uses the PartitionRecord processor to group like records by State Publishes records originating from California to Kafka This article is the first of a two part series. We will setup the demo environment including flows, controller services and reporting tasks. The second article will walk through the main flow step by step.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.3.0 Apache Kafka 0.10.2.1Environment ConfigurationKafka Setup In the bin directory of your Kafka install: Start ZooKeeper: ./zookeeper-server-start.sh ../config/zookeeper.properties Start Kafka: ./kafka-server-start.sh ../config/server.properties Create Kafka Topic: ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic Cali

In previous releases of HDP, client-side caching of keys could result in unexpected behavior with WebHDFS.Consider the following steps:1. Create two keys in ranger KMS: user1_key and user2_key2. Add two resource based policy one per above user.User1_encr_policy: Allow the Decrypt_EEK permissions to user1 onlyUser2_encr_policy: Allow the Decrypt_EEK permissions to user2 only.3. Add two encryption zones. user1_zone (using user1_key) and user2_zone (using user2_key)4. Run the following command, you may be able to access the content of test.csv file from user1_zone using user2curl -i -L "http://sandbox.hortonworks.com:50070/webhdfs/v1/customer/user1_zone/test.csv?user.name=user2&op=OPEN"HDP-2.6.1.2 includes HADOOP-13749, which fixes the caching issue. The FS cache and KMS provider cache can be disabled by changing the configuration as follows: "fs.hdfs.impl.disable.cache", "true" dfs.client.key.provider.cache.expiry, 0

The use case is to get the updates using timestamp column from multiple Databases and do some simple transformations in nifi and stream the data into hive transactional tables using puthivestreaming processor in nifi.If not careful, this can easily lead to platform wide instability for hive. ACID is for slowly changing tables and not for 100s of concurrent queries trying to update the same partition. ACID tables are bucketed tables. Have correct number of buckets and have uniform data distribution among buckets. Can easily lead to data skew and only one CPU core writes to a single bucket.Transactional manager and lock manager are stored in hive metastore. Transactional manager keeps the transactional state(open, commit and abort) and lock manager maintains the necessary locks for transactional tables. The recommendation is to separate Hive, oozie and ambari database and configure high availability for databases. Nifi can overwhelm hive ACID tables. Nifi will stream data using hive str

Thanks to @Matt Clarke for resolving this Major issue.In a typical customer environment there is a challenge while deploying HDF Cluster & enabling LDAPS Authentication because of Username case.In Active directory userid exists as (Ex for Empid:- X1122) but When I have imported users in Ranger by setting lowercase=true all imported users are displayed like this in lower case (x1122) .I have created all required policies for kafka and nifi .verified smoke tests for Kafka and they are PASSED. But smoke tests for NiFi are FAILED because because NiFi respects only AD value(X1122) and there is no inbuilt intelligence todo a case conversion. All the NiFI ranger policies has userid as (x112233).So Ranger Nifi policies are not applicable in this scenario and ranger nifi plugin authorization is not working correctly. So,NiFi Ranger Authorization has Failed to access View NiFI UI under /flow ranger policy.NiFi does not have a option to change case sensitive of returned results, but wit

Bridging the Process Time – Event Time gap with Hive (Part 1)SynopsisReconciling the difference between event time and collection/processing time is critical to understand for any system that analyses event data. This is important whether events are processed in batch or near real-time streams. This post focuses on batch processing with Hive, and demonstrates easily replicable mechanisms for bridging this gap. We will look at the issues surrounding this and prevent two repeatable solution patterns using Hive and Hive ACID. This first post will look at the issue and present the solution using Hive only, and the follow-up article will introduce Hive ACID and a solution using that technology.OverviewOne of the most common big data ingestion cases is event data, and as IoT becomes more important, so does this use case. This is one of the most common Hadoop use cases, but I have not found many detailed step by step patterns for implementing it. In addition, I think it is important to under

Often times, it requires us to use the plain Map-Reduce for some special custom use cases. There are many examples about how to use pure text format using Text File Input formats, but when coming to process ORC files, there aren't any. The Apache ORC website is a very good place to start with. I had a custom requirement to read data from a Hive ORC table and do some special processing which is quite difficult to do in Hive UDF. Below is the Map-Reduce code for that using all latest Hadoop 2.7.3 APIDriver Code:import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.conf.Configured;import org.apache.hadoop.fs.Path;import org.apache.hadoop.hive.ql.io.orc.OrcNewInputFormat;import org.apache.hadoop.io.NullWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.input.MultipleInputs;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.a

PROBLEM:A Sqoop export fails with the following error: 17/08/16 17:17:43 WARN conf.HiveConf: HiveConf of name hive.semantic.analyzer.factory.impl does not exist17/08/16 17:17:43 INFO hive.metastore: Trying to connect to metastore with URI thrift://XYZ:908317/08/16 17:17:47 WARN hive.metastore: set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection resetat org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:3692)at org.apac

PROBLEM:During rolling HDP upgrade, the package installation fails with the following error: 2017-08-10 22:29:01,914 - Package Manager failed to install packages. Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install hadoop_2_6_0_3_8-client' returned 1. error: rpmdb: BDB0113 Thread/process 9731/140223883491136 failed: BDB2052 locker has write lockserror: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recoveryerror: cannot open Packages index using db5 - (-30973)error: cannot open Packages database in /var/lib/rpmCRITICAL:yum.main:Error: rpmdb open failedTraceback (most recent call last):File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 376, in install_packagesretry_count=agent_stack_retry_countFile "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__self.env.run()File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in runself.run_ac

PROBLEM:When installing RPM during HDP Upgrade on few nodes, the installation fails with the following error: rpmdb: /var/lib/rpm/Packages: unexpected file format or type error: cannot open Packages index using db3 - Invalid argument (22)ROOT CAUSE This issue occurs when the RPM database is corrupt.RESOLUTION To resolve this issue, do the following: Take a backup of existing RPM database:mkdir /root/rpmdb.bakcp -rp /var/lib/rpm/__db* /root/rpmdb.bak/ Remove the corrupted RPM database files: cd /var/lib/rpmrm -rf __db* Rebuild the RPM database: rpm --rebuilddb

PROBLEM:With a requirement to disable auto-start services before an upgrade, the auto-start services option seems to be invisible in the Ambari server UI.ROOT CAUSE This issue occurs when the cascaded style sheets and javascripts from previous version still exists under webapp location on Ambari Server.RESOLUTION To resolve this issue, remove *.gz files from /usr/lib/ambari-server/web/javascripts on the Ambari server.

PROBLEM: When running a Distcp command with HTTPFS, it fails with the following error:Error: java.io.IOException: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.FileNotFoundException: File does not exist: /<some-location>/<file-name>/<same-file-name> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:223) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)Caused by: org.apache.hadoop.tools.mapred.

PROBLEM: ATS crashes silently. No errors or exceptions in the logs.ROOT CAUSE: Operating system's OOM Killer kills ATS because it uses highest memory(more memory than assigned heap) RESOLUTION : Please add below properties in yarn-site.xml to fix this:yarn.timeline-service.ttl-ms=604800000 yarn.timeline-service.rolling-period=daily yarn.timeline-service.leveldb-timeline-store.read-cache-size=4194304 yarn.timeline-service.leveldb-timeline-store.write-buffer-size=4194304 yarn.timeline-service.leveldb-timeline-store.max-open-files=500

ROOT CAUSE:Oozie service check runs mapreduce action, sometimes because of lack of available resources on Yarn cluster, script times out.RESOLUTION: Increase the service check script timeout by following below steps:1. Please SSH to Ambari server host 2. Navigate to /var/lib/ambari-server/resources/common-services/OOZIE/4.0.0.2.0/package/scripts 3. Backup service_check.py to some other location ( e.g. /root/ ) 4. Edit service_check.py and increase sleep/retires value

SYMPTOM FATAL Services - SERVER[deadpool.lab.local] E0113: class not found [org.apache.oozie.extensions.OozieELExtensions]org.apache.oozie.service.ServiceException: E0113: class not found [org.apache.oozie.extensions.OozieELExtensions]at org.apache.oozie.service.ELService.findMethod(ELService.java:226)at org.apache.oozie.service.ELService.extractFunctions(ELService.java:104)at org.apache.oozie.service.ELService.init(ELService.java:135)at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)at org.apache.oozie.service.Services.setService(Services.java:372)at org.apache.oozie.service.Services.loadServices(Services.java:305)at org.apache.oozie.service.Services.init(Services.java:213)at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210)at org.apache.catalina.core.StandardContext.start(StandardContext.java:4709)at org.apache.catalina.core.ContainerBase.addChi