Community Articles

IntroductionThis is a continuation of an article I wrote about 1 year ago: https://community.hortonworks.com/articles/60580/jmeter-setup-for-hive-load-testing-draft.htmlhttps://www.blazemeter.com/blog/windows-authentication-apache-jmeterSteps1) Enable Kerberos on your clusterPerform all steps specified here: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_security/content/configuring_amb_hdp_for_kerberos.html and connect successfully to hive service via command line using your user keytab. That implies a valid ticket.2) Install JMeterSee previous article mentioned in Introduction.3) Set Hive User keytab in jaas.confJMETER_HOME/bin/jaas.confYour jaas.conf should look something like this:JMeter {com.sun.security.auth.module.Krb5LoginModule requireduseTicketCache=falsedoNotPrompt=trueuseKeyTab=truekeyTab="/etc/security/keytabs/hive.service.keytab"principal="hive/server.example.com@EXAMPLE.COM"debug=true;};4) JMeter SetupThere are 2 files under /bin folder of the JMeter install

In preparing for my talk at DataWorksSummit in Australia, I wanted to try yet another way to integrate Apache NiFi with TensorFlow. The common ways being: calling a Python TensorFlow script from Execute Command, calling a TensorFlow Serving server via gRPC, calling a C++ TensorFlow executable via Execute Command, running TensorFlow on the edge and having MiniFi send it to NiFi or calling a TensorFlowOnSpark job via Kafka, Site-to-Site.TensorFlow has released a Java API (https://www.tensorflow.org/install/install_java), so I decided to write a quick custom processor to run TensorFlow Inception v3.It's a simple set of dependencies for Maven: <dependencies> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-api</artifactId> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-processor-utils</artifactId>

IntroductionThe term “Load Testing” has evolved over the years, however the core meaning still comes down to making sure that your system can handle a predefined amount of users at the same time. The load test is usually run over an extended period of time in a staggered manner, slowly ramping up the amount of users until you hit a predefined maximum that’s usually based on projected usage levels extrapolated from access logs with a buffer for surges. The goal of load testing is to determine the number of users that the system can typically cope with, this is called the systems “concurrency level” and give you a hard number to work with when dealing with capacity planning, performance and optimization, stability, SLAs etc.JMeter was originally designed for testing Web Applications but has since expanded to other test functions, including database via JDBC. As such, it has the ability to execute SQL queries against a given JDBC Driver. JMeter allows to define queries to execute against

This is part 1 of a miniseries. Part 2 can be found here.How do you count the number of lines in a FlowFile in NiFi? The naïve answer would be to run wc -l inside an ExecuteStreamCommand processor. This is unsatisfying because it requires firing up a shell command in a separate process. This is a scalability issue and also a security risk. But, there is a better way!The Record processors of NiFi 1.2+ offer the possibility to run real time, in-stream SQL against FlowFiles. Why not use this capability? Here's the approach:I am going to define a minimalistic schema that allows me to read each line of a CSV file as a record with one single field. Here is my example flow:Let's go through the steps. First, I use a GetFile processor to pick up files in my home directory. The actual configuration does not matter - use whatever your data source is. These data are passed to a QueryRecord processor, which is where the magic happens.My Query Record processor is configured like this:For reading the

Part 1: https://community.hortonworks.com/articles/82964/getting-started-with-apache-ambari-workflow-design.htmlPart 2: https://community.hortonworks.com/articles/82967/apache-ambari-workflow-designer-view-for-apache-oo.htmlPart 3: https://community.hortonworks.com/articles/82988/apache-ambari-workflow-designer-view-for-apache-oo-1.htmlPart 4: https://community.hortonworks.com/articles/83051/apache-ambari-workflow-designer-view-for-apache-oo-2.htmlPart 5: https://community.hortonworks.com/articles/83361/apache-ambari-workflow-manager-view-for-apache-ooz.htmlPart 6: https://community.hortonworks.com/articles/83787/apache-ambari-workflow-manager-view-for-apache-ooz-1.htmlPart 8: https://community.hortonworks.com/articles/84394/apache-ambari-workflow-manager-view-for-apache-ooz-3.htmlPart 9: https://community.hortonworks.com/articles/85091/apache-ambari-workflow-manager-view-for-apache-ooz-4.htmlPart 10: https://community.hortonworks.com/articles/85354/apache-ambari-workflow-manager-view-

Cloudbreak contains mini KNOX which is not managed by Ambari.Below are the steps to replace Self Signed Certificate with CA Signed CertificatesStep 1: Remove below two entries from /usr/hdp/current/knox-server/conf/gateway-site.xml and save it.<property> <name>gateway.signing.keystore.name</name> <value>signing.jks</value> </property> <property> <name>gateway.signing.key.alias</name> <value>signing-identity</value> </property> Step 2: Take a backup of original configuration:[~]$ cd /usr/hdp/current/knox-server/data/security/keystores/[~]$ mkdir backup[~]$ mv __gateway-credentials.jceks gateway.jks backup/Step 3: Create a keystore in PKCS12 format from your private key file, certificate, Intermediate certificate and root certificate[~]$ openssl pkcs12 -export -out corp_cert_chain.pfx -inkey <private-key>.key -in <cert.cer> -certfile <root_intermediate>.cer -certfile <root_ca>.cerStep 4: Regen

Objective This tutorial is the second article of a two part series. We will walk-through in detail a flow which: Uses the LookupRecord processor to parse NiFi provenance events in JSON format and add geolocation data Uses the PartitionRecord processor to group like records by State Publishes records originating from California to Kafka The first article can be found here. Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.3.0 Apache Kafka 0.10.2.1 NiFi LookupRecord Flow Walk Through Prerequisites The first article in this tutorial series has been completed. The file GeoLite2-City.mmdb has been downloaded locally into a directory named "enrichment-db" within your NiFi installation Provenance data has been generated and received by the "Provenance In" input port. All necessary controller services and reporting tasks are enabled/running. Kafka is running and a topic "California" has been created.

Objective This tutorial walks you through a NiFi flow that: Uses the LookupRecord processor to parse NiFi provenance events in JSON format and add geolocation data Uses the PartitionRecord processor to group like records by State Publishes records originating from California to Kafka This article is the first of a two part series. We will setup the demo environment including flows, controller services and reporting tasks. The second article will walk through the main flow step by step.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.3.0 Apache Kafka 0.10.2.1Environment ConfigurationKafka Setup In the bin directory of your Kafka install: Start ZooKeeper: ./zookeeper-server-start.sh ../config/zookeeper.properties Start Kafka: ./kafka-server-start.sh ../config/server.properties Create Kafka Topic: ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic Cali

In previous releases of HDP, client-side caching of keys could result in unexpected behavior with WebHDFS.Consider the following steps:1. Create two keys in ranger KMS: user1_key and user2_key2. Add two resource based policy one per above user.User1_encr_policy: Allow the Decrypt_EEK permissions to user1 onlyUser2_encr_policy: Allow the Decrypt_EEK permissions to user2 only.3. Add two encryption zones. user1_zone (using user1_key) and user2_zone (using user2_key)4. Run the following command, you may be able to access the content of test.csv file from user1_zone using user2curl -i -L "http://sandbox.hortonworks.com:50070/webhdfs/v1/customer/user1_zone/test.csv?user.name=user2&op=OPEN"HDP-2.6.1.2 includes HADOOP-13749, which fixes the caching issue. The FS cache and KMS provider cache can be disabled by changing the configuration as follows: "fs.hdfs.impl.disable.cache", "true" dfs.client.key.provider.cache.expiry, 0

The use case is to get the updates using timestamp column from multiple Databases and do some simple transformations in nifi and stream the data into hive transactional tables using puthivestreaming processor in nifi.If not careful, this can easily lead to platform wide instability for hive. ACID is for slowly changing tables and not for 100s of concurrent queries trying to update the same partition. ACID tables are bucketed tables. Have correct number of buckets and have uniform data distribution among buckets. Can easily lead to data skew and only one CPU core writes to a single bucket.Transactional manager and lock manager are stored in hive metastore. Transactional manager keeps the transactional state(open, commit and abort) and lock manager maintains the necessary locks for transactional tables. The recommendation is to separate Hive, oozie and ambari database and configure high availability for databases. Nifi can overwhelm hive ACID tables. Nifi will stream data using hive str

Thanks to @Matt Clarke for resolving this Major issue.In a typical customer environment there is a challenge while deploying HDF Cluster & enabling LDAPS Authentication because of Username case.In Active directory userid exists as (Ex for Empid:- X1122) but When I have imported users in Ranger by setting lowercase=true all imported users are displayed like this in lower case (x1122) .I have created all required policies for kafka and nifi .verified smoke tests for Kafka and they are PASSED. But smoke tests for NiFi are FAILED because because NiFi respects only AD value(X1122) and there is no inbuilt intelligence todo a case conversion. All the NiFI ranger policies has userid as (x112233).So Ranger Nifi policies are not applicable in this scenario and ranger nifi plugin authorization is not working correctly. So,NiFi Ranger Authorization has Failed to access View NiFI UI under /flow ranger policy.NiFi does not have a option to change case sensitive of returned results, but wit

Bridging the Process Time – Event Time gap with Hive (Part 1)SynopsisReconciling the difference between event time and collection/processing time is critical to understand for any system that analyses event data. This is important whether events are processed in batch or near real-time streams. This post focuses on batch processing with Hive, and demonstrates easily replicable mechanisms for bridging this gap. We will look at the issues surrounding this and prevent two repeatable solution patterns using Hive and Hive ACID. This first post will look at the issue and present the solution using Hive only, and the follow-up article will introduce Hive ACID and a solution using that technology.OverviewOne of the most common big data ingestion cases is event data, and as IoT becomes more important, so does this use case. This is one of the most common Hadoop use cases, but I have not found many detailed step by step patterns for implementing it. In addition, I think it is important to under

Often times, it requires us to use the plain Map-Reduce for some special custom use cases. There are many examples about how to use pure text format using Text File Input formats, but when coming to process ORC files, there aren't any. The Apache ORC website is a very good place to start with. I had a custom requirement to read data from a Hive ORC table and do some special processing which is quite difficult to do in Hive UDF. Below is the Map-Reduce code for that using all latest Hadoop 2.7.3 APIDriver Code:import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.conf.Configured;import org.apache.hadoop.fs.Path;import org.apache.hadoop.hive.ql.io.orc.OrcNewInputFormat;import org.apache.hadoop.io.NullWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.input.MultipleInputs;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.a

PROBLEM:A Sqoop export fails with the following error: 17/08/16 17:17:43 WARN conf.HiveConf: HiveConf of name hive.semantic.analyzer.factory.impl does not exist17/08/16 17:17:43 INFO hive.metastore: Trying to connect to metastore with URI thrift://XYZ:908317/08/16 17:17:47 WARN hive.metastore: set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection resetat org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:3692)at org.apac

PROBLEM:During rolling HDP upgrade, the package installation fails with the following error: 2017-08-10 22:29:01,914 - Package Manager failed to install packages. Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install hadoop_2_6_0_3_8-client' returned 1. error: rpmdb: BDB0113 Thread/process 9731/140223883491136 failed: BDB2052 locker has write lockserror: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recoveryerror: cannot open Packages index using db5 - (-30973)error: cannot open Packages database in /var/lib/rpmCRITICAL:yum.main:Error: rpmdb open failedTraceback (most recent call last):File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 376, in install_packagesretry_count=agent_stack_retry_countFile "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__self.env.run()File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in runself.run_ac

PROBLEM:When installing RPM during HDP Upgrade on few nodes, the installation fails with the following error: rpmdb: /var/lib/rpm/Packages: unexpected file format or type error: cannot open Packages index using db3 - Invalid argument (22)ROOT CAUSE This issue occurs when the RPM database is corrupt.RESOLUTION To resolve this issue, do the following: Take a backup of existing RPM database:mkdir /root/rpmdb.bakcp -rp /var/lib/rpm/__db* /root/rpmdb.bak/ Remove the corrupted RPM database files: cd /var/lib/rpmrm -rf __db* Rebuild the RPM database: rpm --rebuilddb

PROBLEM:With a requirement to disable auto-start services before an upgrade, the auto-start services option seems to be invisible in the Ambari server UI.ROOT CAUSE This issue occurs when the cascaded style sheets and javascripts from previous version still exists under webapp location on Ambari Server.RESOLUTION To resolve this issue, remove *.gz files from /usr/lib/ambari-server/web/javascripts on the Ambari server.

PROBLEM: When running a Distcp command with HTTPFS, it fails with the following error:Error: java.io.IOException: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.FileNotFoundException: File does not exist: /<some-location>/<file-name>/<same-file-name> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:223) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)Caused by: org.apache.hadoop.tools.mapred.

PROBLEM: ATS crashes silently. No errors or exceptions in the logs.ROOT CAUSE: Operating system's OOM Killer kills ATS because it uses highest memory(more memory than assigned heap) RESOLUTION : Please add below properties in yarn-site.xml to fix this:yarn.timeline-service.ttl-ms=604800000 yarn.timeline-service.rolling-period=daily yarn.timeline-service.leveldb-timeline-store.read-cache-size=4194304 yarn.timeline-service.leveldb-timeline-store.write-buffer-size=4194304 yarn.timeline-service.leveldb-timeline-store.max-open-files=500

ROOT CAUSE:Oozie service check runs mapreduce action, sometimes because of lack of available resources on Yarn cluster, script times out.RESOLUTION: Increase the service check script timeout by following below steps:1. Please SSH to Ambari server host 2. Navigate to /var/lib/ambari-server/resources/common-services/OOZIE/4.0.0.2.0/package/scripts 3. Backup service_check.py to some other location ( e.g. /root/ ) 4. Edit service_check.py and increase sleep/retires value

SYMPTOM FATAL Services - SERVER[deadpool.lab.local] E0113: class not found [org.apache.oozie.extensions.OozieELExtensions]org.apache.oozie.service.ServiceException: E0113: class not found [org.apache.oozie.extensions.OozieELExtensions]at org.apache.oozie.service.ELService.findMethod(ELService.java:226)at org.apache.oozie.service.ELService.extractFunctions(ELService.java:104)at org.apache.oozie.service.ELService.init(ELService.java:135)at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)at org.apache.oozie.service.Services.setService(Services.java:372)at org.apache.oozie.service.Services.loadServices(Services.java:305)at org.apache.oozie.service.Services.init(Services.java:213)at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210)at org.apache.catalina.core.StandardContext.start(StandardContext.java:4709)at org.apache.catalina.core.ContainerBase.addChi

PROBLEM: Hive metastore isn’t updating for tables after insert into the table. This issue is observed with improper ‘numRows’ value with the describle formatted <tb_name>RESOLUTION: Hive stats are autogathered properly till an 'analyze table [tablename] compute statistics for columns' is run. Then it does not auto-update the stats till the command is run again. This is due to a known issuehttps://issues.apache.org/jira/browse/HIVE-12661 and has been fixed in HDP-2.5.0

PROBLEM: While trying to execute a simple pyspark script that is trying to select data from Hive transactional table stored in ORC format, customer is facing following exception. java.lang.RuntimeException: serious problem at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1021) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1048) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:250) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.sc

This is an unsupported technology and a concept which hasn't been explored yet.There's no real modification time concept in object stores. It has just creation time, which is that of the observed time at the far end. If you upload a file to a remote timezone, you may get that as your time.The underlying issue here is not a bug. It is just a feature that distcp -update relies on using file checksums for comparing HDFS files, and (a) not all stores export their checksum through the Hadoop API (WASB does, s3a doesn't yet).In addition, because the checksums are different between blobstores and HDFS, you can't use checksum difference as a cue for files being changed.Note that this also occurs when trying to copy between HDFS encryption zones, as the checksums of the encrypted files will differ.

Background:When installing HDF only, there used to be a default Grafana dashboard for NiFi with the general metrics of NiFi available. When installing HDF on top of an existing HDP cluster (with latest versions of stacks), there is no default dashboard for NiFi even though there are some for Storm and Kafka. This should be fixed for the next version.Problem Summary:Nifi dashboard not visible in Grafana after installing HDF over HDP. Root cause: json file is missing the grafana directory Resolution: Copy json file as below cp /var/lib/ambari-server/resources/common-services/AMBARI_METRICS/0.1.0/package/files/grafana-dashboards/HDF/grafana-nifi-ho* /var/lib/ambari-server/resources/common-services/AMBARI_METRICS/0.1.0/package/files/grafana-dashboards/HDP/