Community Articles

Apache NiFi Processor for Apache MXNet SSD: Single Shot MultiBox Object Detector (Deep Learning)The news is out, Apache MXNet has added a Java API. So as soon as I could I got my hands on the maven repo and an example program and got to work writing a new Apache NiFi processor for it.I have run this on standalone Apache NiFi 1.8.0 and on HDF 3.3 - Apache NiFi 1.8.0 and both work. So anyone who wants to be an alpha tester, please download it and give it a try.Apache MXNet SSD is a good example of a pretrained deep learning model that works pretty well for general images in a use cases especially around people and cars. You can fine-tune this with some more images and runs: https://mxnet.incubator.apache.org/faq/finetune.htmlThe nice thing is now we can start including Apache MXNet as part of Java applications such as Kafka Streams, Apache Storm, Apache Spark, Spring Boot and other use cases using Java. I could potentially inject this into a Hive UDF (https://community.hortonwor

IntroductionPerformance of a cluster running on Hadoop can be impacted by the OS partitioning. This document is intended to understand the best practices to setup the “/var” folder/partition with optimum size.Lets try to approach this problem by asking some important questions. What is “/var” used for? How can the “/var” folder run out of disk space? Common issue to expect on a Hadoop cluster if “/var” is out of disk space. How is the current setup of “/var” in my cluster ?Question 1 - What is /var used for?From OS perspective, “/var” is commonly used for constantly changing files i.e. variable. The short form of which is “var”.Example of such files could be the log file, mail, transient file, the printer spool, temporary files, cached data, etc.For example - “/var/tmp” holds the temporary files between system reboots.On any node (Hadoop or non-Hadoop), /var directory holds content for a number of applications. It also is used to store downloaded update packages

The latest Docker version available form official Centos and RedHat repository has an issue preventing to exec the container.Issue when starting Cloudbreak is the following:Starting Cloudbreakgenerating docker-compose.ymlgenerating uaa.ymlInitialize and migrate databasesCreating cbreak_commondb_1 ...Creating cbreak_commondb_1 ... done[ERROR] The database hasn't started within 30 seconds.Docker version: docker-1.13.1-84.git07f3374.el7.centos.x86_64There are the following solutions, you can choose either:Downgrade to an older version:yum downgrade docker-1.13.1-75.git8633870.el7.centos.x86_64 docker-client-1.13.1-75.git8633870.el7.centos.x86_64 docker-common-1.13.1-75.git8633870.el7.centos.x86_64Upgrade to a newer version, using Docker repository:https://docs.docker.com/install/linux/docker-ce/centos/#set-up-the-repositoryAfter choosing either option docker restart should work.

Issue:Trying to run "msck repair table <tablename>" gives the below error Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. nullResolution:The above error occurs when hive.mv.files.thread=0, increasing the value of the parameter to 15 fixes the issueThis is a known bug

Decrypting OpenSSL Data in Apache NiFi Who Am I? My name is Andy LoPresto and I am a software engineer on the HDF team at Hortonworks. I previously worked in information security (encryption in financial systems & device drivers, PKI, etc.) at Apple and was a security architect at TigerText. The Issue A question I've been asked many times over my career thus far is, "When should we encrypt something?" Now, the most correct first answer to any information security question is, "It depends," but people get tired of hearing that very quickly. So my default response to this question is, "Whenever you don't have a good reason not to." Using encryption in high-level languages is usually straightforward in theory at this point, thanks to many helpful libraries such as BouncyCastle, but can be more difficult in practice. Implementation details matter. Unless you have a Ph.D. in cryptography or discrete mathematics and an extensive background in software engineering (in which case y

This article has been set on a HDP 2.5.3 version, you may consider adjusting some parameters to reflect your actual version.We'll here set Kafka loglevel through the Logging MBean with jConsole.For that, the first step is to enable JMX access:add in Kafka configs/kafka-env templateexport KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false"export JMX_PORT="9999" For avoiding JMX port conflicts like mentioned in https://community.hortonworks.com/articles/73750/kafka-jmx-tool-is-failing-with-port-already-in-use.html, let’s modify /usr/hdp/current/kafka/bin/kafka-run-class.sh on all broker nodes: replace# JMX port to useif [ $JMX_PORT ]; then KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"fiwith# JMX port to useif [ $ISKAFKASERVER = "true" ]; then JMX_REMOTE_PORT=$JMX_PORTelse JMX_REMOTE_PORT=$CLIENT_JMX_PORTfiif [ $JMX_REMO

Linkedin PostPresto is a tool designed to efficiently query vast amounts of data using distributed queries.We will be installing Presto in single server mode, Access Hive and then add worker node.Cross query - RBDMS, Hive, NoSqlTutorial**Java 8 must **Install - link (for the latest versions)wget https://repo1.maven.org/maven2/com/facebook/presto/presto-server/0.122/presto-server-0.122.tar.gztar xvfz presto-server-0.122.tar.gzLet's start with Single node setup (master and worker on the same node)cd presto-server-0.122mkdir etc[root@ns2 presto-server-0.122]# cd etc/mkdir catalog and we will create 3 files as shown below[root@ns2 etc]# lscatalog config.properties jvm.config log.properties node.properties[root@ns etc]# cat config.propertiescoordinator=truenode-scheduler.include-coordinator=truehttp-server.http.port=9080query.max-memory=10GBquery.max-memory-per-node=1GBdiscovery-server.enabled=truediscovery.uri=http://ns2:9080[root@ns2 etc]# cat log.propertiescom.facebook.presto=INFO[roo

Steps: When configuring a Kafka broker to use only SSL, you can have authentication and encryption by enabling 2-ways SSL by using parameter ssl.client.auth=required.Go to Ambari > Kafka > configs > custom Kafka-broker, add ssl.client.auth=required parameter.In this case certificate owner name (subject) works as username & authentication happens with this username. You have to add server certificate owner name from each broker as superuser (usernames separated by semicolon) so that each broker can access resources from other brokers. Example: In the case of two brokers you need to add two server certificate owner names to superuser.Go to Ambari > kafka > configs > custom kafka-broker, configure super.users parametersuper.users = User:CN=user1,OU=kafka,O=kafka,L=kafka,ST=kafka,C=xx;User:CN=user2,OU=kafka,O=kafka,L=kafka,ST=kafka,C=xxYou can create different client certificates for different users, if client certificate owner name is "CN=kafka,OU=kafka,O=kafka,L=k

MiniFi Java Agent 0.5Copy over necessary NARs from Apache NiFi 1.7 lib: nifi-ssl-context-service-nar-1.7.0.narnifi-standard-services-api-nar-1.7.0.narnifi-kafka-1-0-nar-1.7.0.narThis will support PublishKafka_1_0 and ConsumeKafka_1_0.Then create a consume and/or publish flow. You can combine the two based on your needs. In my simple example I consume the Kafka messages in MiniFi and write to a file. I also write the metadata to a JSON file.Consume Kafka Publish Electric Monitoring Data To KafkaLet's monitor the messages going through our topic, smartPlug.Publish Messages to KafkaConsume Any Messages From the smartPlug topicLogsProvenance Event file containing 377 records. In the past 5 minutes, 1512 events have been written to the Provenance Repository, totaling 839.32 KB2018-11-26 19:42:32,473 INFO [main] o.a.n.c.s.StandardProcessScheduler Starting PutFile[id=25a86505-031a-37d9-0000-000000000000]2018-11-26 19:42:32,474 INFO [main] o.a.n.c.s.StandardProcessScheduler Starting Update

HBase along with Phoenix is one of the most powerful NoSQL combinations. HBase/Phoenix capabilities allow users to host OLTPish workloads natively on Hadoop using HBase/Phoenix with all the goodness of HA and analytic benefits on a single platform (Ie Spark-hbase connector or Phoenix Hive storage handler). Often a requirement for HA implementations is a need for DR environment. Here I will describe a few common patterns and in no way is this the exhaustive HBase DR patterns. In my opinion, pattern 5 is the simplest to implement and provides operational ease & efficiency. Here are some of the high level replication and availability strategies with HBase/Phoenix HBASE provides High Availability within a cluster by managing region server failures transparently. HBASE provides various cross DC asynchronous replication schemes Master/Master replication topology Two clusters replicating all edits, bi-directionally to each other Master/Slave topology replic

IntroductionNaïve Bayes is a machine learning model that is simple and computationally light yet also accurate in classifying text as compared to more complex ML models. In this article I will use the Python scikit-learn libraries to develop the model. It is developed on a Zeppelin notebook on top of the Hortonworks Data Platform (HDP) and uses its %spark2.pyspark interpreter to run python on top of Spark. We will use a news feed to train the model to classify text. We will first build the basic model, then explore its data and attempt to improve the model. Finally, we will compare performance accuracy of all the models we develop.A note on code structuring: Python import statements are introduced when required by the code and not all at once upfront. This is to relate the packages closely to the code itself.The Zeppelin template for the full notebook can be obtained from here.A. Basic ModelGet the dataWe will use news feed data that has been classified as either 1 (Business), 2 (

Short Description:A NiFi connection is where FlowFiles are temporarily held between two connected NiFi Processor components. Each connection that contains queued NiFi FlowFiles will have a footprint in the JVM heap. This article will breakdown a connection to show how NiFi manages the FlowFiles that are queued in that connection and that affects heap and performance.Article:First let me share the 10,000 foot view and then I will discuss each aspect of the following image:*** NiFi FlowFiles consist of FlowFile Content and FlowFile Attributes/metadata. FlowFile content is never held in a connections heap space. Only the FlowFile Attributes/metadata is placed in heap by a connection.The "Connection Queue":The connection queue is where all FlowFiles queued in the connection are held. To understand how these queued FlowFiles affect performance and heap usage, lets start by focusing on the "Connection Queue" dissection at the bottom of the above image.The overall size of a connection is co

Using MQTT From MiniFi Java AgentFrom a standard Apache NiFi 1.5 download. Copy nifi-standard-services-api-nar-1.5.0.3.1.1.0-35.nar nifi-mqtt-nar-1.5.0.3.1.1.0-35.nar To minifi 0.40. lib directory. PublishMQTTPublish Sensor Data Via MQTTThis is our example MiniFi flow. We can drop out the GetFile (to grab images) and push to NiFi Flow if we just want to do a simple MQTT use case.I am using CloudMQTT, you can use any MQTT broker like Mosquitto or HiveMQ.Ingest in Apache NiFi 1.5We can process data from MQTT and/or standard Apache NiFi S2S HTTPS.Processing continues as regular. I add another route for mqtt source.References:If you are doing SSL, see https://community.hortonworks.com/articles/47854/accessing-facebook-page-data-from-apache-nifi.html

This article just gives an example of how 'grant'/'revoke' works when the Hive plugin is enabled with Ranger in CDP. A user who is 'admin' in Ranger, can manage access to Hive tables via 'grant'/'revoke' operation. In Ranger UI > Settings > Users and Groups > Users Note: User 'hive' is in role 'Admin' On the beeline, login as user 'hive'. Run the grant command to give select privileges on a table:   0: jdbc:hive2://a.b.c.co> grant select on table mix to user mugdha;INFO : Compiling command(queryId=hive_20211021024819_c3de84a7-a312-4a1f-9a8d-8b328cced054): grant select on table mix to user mugdhaINFO : Semantic Analysis Completed (retrial = false)INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)INFO : Completed compiling command(queryId=hive_20211021024819_c3de84a7-a312-4a1f-9a8d-8b328cced054); Time taken: 0.022 secondsINFO : Executing command(queryId=hive_20211021024819_c3de84a7-a312-4a1f-9a8d-8b328cced054): grant sel

Step-by-step instructions on how-to install a CRAN package from a local repo - without an internet connection. I'll be installing the package called "tidyr". In order to fully install the package, I need to first download tidyr and all dependencies. To do this, I use the CRAN PACKAGE: https://cran.r-project.org/src/contrib/PACKAGES 1) Finding "Package: tidyr", I can see the dependencies (imports): Imports: dplyr (>= 0.7.0), glue, magrittr, purrr, Rcpp, rlang, stringi, tibble, tidyselect and from them, I can build a list of all the imported packages I need: assertthat_0.2.0.tar.gz BH_1.66.0-1.tar.gz bindr_0.1.1.tar.gz bindrcpp_0.2.tar.gz cli_1.0.0.tar.gz crayon_1.3.4.tar.gz dplyr_0.7.4.tar.gz glue_1.2.0.tar.gz magrittr_1.5.tar.gz pillar_1.2.1.tar.gz plogr_0.1-1.tar.gz purrr_0.2.4.tar.gz Rcpp_0.12.16.tar.gz rlang_0.2.0.tar.gz stringi_1.1.7.tar.gz tibble_1.4.2.tar.gz tidyr_0.8.0.tar.gz tidyselect_0.2.4.tar.gz utf8_1.1.3.tar.gzR6_2.2.2.tar.gzpkgconfig_2.0.1.tar.gz 2) Do

By Default, HiveServer2 and HiveMetastore does not have configuration for HeapDump on OOM. Production clusters have OOM and since the HeapDump on OOM is not configured, root cause analysis of the issue is obstructed.Navigate as below in Ambari:Ambari UI > Hive > Configs > Advanced hive-env > hive-env templateAdd following :if [ "$SERVICE" = "metastore" ]; then export HADOOP_HEAPSIZE={{hive_metastore_heapsize}} # Setting for HiveMetastoreelse export HADOOP_HEAPSIZE={{hive_heapsize}} # Setting for HiveServer2 and Clientfiexport HADOOP_CLIENT_OPTS="-Xmx${HADOOP_HEAPSIZE}m -Xloggc:/var/log/hive/gc.log-$SERVICE-`date +'%Y%m%d%H%M'`-XX:ErrorFile=/var/log/hive/hive-metastore-error.log-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/hive/$HADOOP_CLIENT_OPTS"if [ "$SERVICE" = "hiveserver2" ]; thenexport HADOOP_CLIENT_OPTS="-Xmx${HADOOP_HEAPSIZE}m -Xloggc:/var/log/hive/gc.l

HBase, Phoenix And Ranger In Part 1, Part 2 and Part3 of this article series , we discussed internals of Phoenix Index maintenance and major issues hit around this feature. In this article we will discuss about Phoenix - Ranger relationship , how it works and what had broken until recently which caused reporting of several issues.How native HBase authorization work:ACLs in HBase are implemented as a coprocessor called AccessController. (hbase.security.authorization=true). Users are granted specific permissions such as Read, Write, Execute, Create, Admin against resources such as global, namespaces, tables, cells, or endpoints. ( all self explanatory) There is an additional user called “Superuser”. Superusers can perform any operation available in HBase, to any resource. The user who runs HBase on your cluster is a superuser, as are any principals assigned to the configuration property hbase.superuser in hbase-site.xml.Much more details on this subject are hereHow things Changed with R

Issues with Global IndexesIn Part 1 of this article series, we discussed the internals of Index maintenance, in this part we will cover some of the major issues we face during the life cycle of Index maintenance.Before we get into issues, we need to understand various “states” of Index table which reflect its health in general.BUILDING("b") : This will partially rebuild the index from the last disabled timestamp UNUSABLE (“d”) / INACTIVE ("i") : This will cause the index to no longer be considered for use in queries, however index maintenance will continue to be performed.ACTIVE("a") : Index ready to use and updated. DISABLE("x") : This will cause the no further index maintenance to be performed on the index and it will no longer be considered for use in queries.REBUILD("r") : This will completely rebuild the index and upon completion will enable the index to be used in queries again.---What happens when an Index update fails for any reason :The answer is not straight as there are choi

HDF or CFM best practices guide to configuring your system and NiFi for high performance dataflows.Note: The recommendation outlined in this article are for the NiFi service and apply whether the NiFi service is being deployed/managed via Ambari, Cloudera Manager, or neither.  NiFi is pre-configured to run with very minimal configuration needed out of the box. Simply edit the nifi.properties file located in the conf directory by adding a sensitive props key (used to encode all sensitive properties in flow) and you are ready to go. This may get you up and running, but that basic configuration is far from ideal for high volume/high performance dataflows. While the NiFi core itself does not have much impact on the system in terms of memory, disk/network I/O, or CPU, the dataflows that you build have the potential of impacting those performance metrics quite a bit. Some NiFi processors can be CPU, I/O or memory intensive. In fact some can be intensive in all three areas. We

While upgrading the cluster from HDP-2.6 to HDP-3.0, not able to generate ambari-infra-solr migration script due to below error .[root@hostname ambari-infra-solr-client]# /usr/bin/python /usr/lib/ambari-infra-solr-client/migrationConfigGenerator.py --ini-file $CONFIG_INI_LOCATION --host <ambari server> --port 8443 -s --cluster DEV --username rchaman --password rchaman --backup-base-path=/tmp/back/ --java-home /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre --ranger-hdfs-base-path=/ranger/auditStart generating config file: ambari_solr_migration.ini ...Get Ambari cluster details ...Set JAVA_HOME: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jreService detected: ZOOKEEPERZookeeper connection string: zk1.openstack:2181,zk2.openstack:2181,zk3.openstack:2181Service detected: AMBARI_INFRA_SOLRInfra Solr znode: /infra-solr_test2018-11-14 14:38:59,104 - Get clusterstate.json: waiting for 5 seconds before retyring again (retry count: 1)2018-11-14 14:39:04,1

There are certain circumstances where the Ambari CA needs to be recreated. Maybe it was corrupted, maybe Ambari's CA certificate expired, etc.... This how-to, assumes that the certificates signed by the Ambari CA are replaceable - which is generally the case for certificates used by Ambari agents for 2-way SSL connections. In the end, the Ambari server and all the agents will be restarted, causing a new CA certificate to be created along with new SSL certificates for each of the Ambari agents. On the Ambari server: Stop the Ambari serverBackup /var/lib/ambari-server/keys and it child directoriesDelete the following files from /var/lib/ambari-server/keysca.keyca.csrca.crtpass.txtkeystore.p12*.csr*.crtDelete the following files from /var/lib/ambari-server/keys/dbindex.txt.oldindex.txt.attr.oldserial.oldTruncate the following files from /var/lib/ambari-server/keys/dbindex.txtindex.txt.attrEdit the following files from /var/lib/ambari-server/keys/dbserialset the contents to be exactly00De

In Ambari when try to upgrade to HDP/HDF versions - after adding a new repo version, some times it might show disabled "install packages" button. There can be multiple reasons for this.1. Check to stack_id which stack_id clusters table is pointing to. it should be pointed to correct current stack_id if not correct with update query2. check clusterstate table - it should be pointing to correct current stack_id if not correct with update query3. How to find my correct current stack_idrun "select stack_id, version from repo_version" query - check the stack_id for your current hdp/hdf version.4. so step1 and step2 should be pointing to stack_id returned in step3.There can be many other reasons like not having "upgrades" folder in stack folder or some other.

When we access an HDFS directory using Ambari FileView then we see max 5000 sub directories/contents per page via File View. As following:We see a message "Showing 5000 files or folders of xxxx".The default "views.files.max.files.per.page" limit is set to 5000 as following: /ambari/view/commons/hdfs/FileOperationService.java#L53-L56This property was added from Ambari 2.6 as part of JIRA: https://issues.apache.org/jira/browse/AMBARI-21890 to avoid browser hung state while opening a HDFS folder which has huge number of files.So if user wants to see more items per page then it can be achieved as following:1. In order to set/add a new property with the value with a bit large page size please adjust the following property inside "ambari.properties". For example i am setting it to 7000# grep 'views.files.max.files.per.page' /etc/ambari-server/conf/ambari.propertiesviews.files.max.files.per.page=70002. Then restart the Ambari Server.# ambari-server restart3. Once ambari server is restarted th

In part 1 of, we looked at how we could use deep learning to classify Melanoma, using transfer learning with a pre-trained VGG16 convolutional neural network. Here we take a look at how to implement this using TensorFlow and Keras.First we import the packages we need:import numpy as npfrom keras.preprocessing.image import ImageDataGeneratorfrom keras.models import Sequentialfrom keras.layers import Dropout, Flatten, Densefrom keras import applicationsThese packages include, the network layers we will be using, the applications (which includes the pretrained VGG16 model), and some image pre-processing utilities.Next, we “strip” our model of it’s top layers. This is as simple as one line of code in Keras.base_model = applications.VGG16(include_top=False,weights='imagenet')Next we take our images of skin, and we transform them using Keras image pre-processing utility:datagen = ImageDataGenerator( rotation_range=40, width_shift_range=0.2, height_shift_range=0.2,

I have provided below the implementation steps for integrating KNOX with Loadbalancer assuming once your Loadbalancer is ready.SSL connection should terminate on Knox servers at Loadbalancer side Sticky session should be enabled. Currently the SSL connection terminates at load balancer side and internally it encrypts and loops through one of the KNOX channels. JKS file creation OPEN in IE browser the load balancer URL Example:-https://hadoop-knox.dev.XXXXXX.com/ Click on lock symbol click on view certificates and Certificate pathchoose Root click on view certificate--details--copytofile--base 509 format --save as .pem filechoose intermediateIssuer CA click on view certificate--details--copytofile--base 509 format --save as .pem filechoose loadbalncercert and click on details--copytofile--base 509 format --save as .pem filecopy these 3 files into knox edge node. I have copied to certfiles folder /tmp/knoxhacerts/new/certfiles/lb-rootca.pem /tmp/knoxhacerts/new/certfiles/lb-intermed