Resources

IntroductionNaïve Bayes is a machine learning model that is simple and computationally light yet also accurate in classifying text as compared to more complex ML models. In this article I will use the Python scikit-learn libraries to develop the model. It is developed on a Zeppelin notebook on top of the Hortonworks Data Platform (HDP) and uses its %spark2.pyspark interpreter to run python on top of Spark. We will use a news feed to train the model to classify text. We will first build the basic model, then explore its data and attempt to improve the model. Finally, we will compare performance accuracy of all the models we develop.A note on code structuring: Python import statements are introduced when required by the code and not all at once upfront. This is to relate the packages closely to the code itself.The Zeppelin template for the full notebook can be obtained from here.A. Basic ModelGet the dataWe will use news feed data that has been classified as either 1 (Business), 2 (

Short Description:A NiFi connection is where FlowFiles are temporarily held between two connected NiFi Processor components. Each connection that contains queued NiFi FlowFiles will have a footprint in the JVM heap. This article will breakdown a connection to show how NiFi manages the FlowFiles that are queued in that connection and that affects heap and performance.Article:First let me share the 10,000 foot view and then I will discuss each aspect of the following image:*** NiFi FlowFiles consist of FlowFile Content and FlowFile Attributes/metadata. FlowFile content is never held in a connections heap space. Only the FlowFile Attributes/metadata is placed in heap by a connection.The "Connection Queue":The connection queue is where all FlowFiles queued in the connection are held. To understand how these queued FlowFiles affect performance and heap usage, lets start by focusing on the "Connection Queue" dissection at the bottom of the above image.The overall size of a connection is co

Using MQTT From MiniFi Java AgentFrom a standard Apache NiFi 1.5 download. Copy nifi-standard-services-api-nar-1.5.0.3.1.1.0-35.nar nifi-mqtt-nar-1.5.0.3.1.1.0-35.nar To minifi 0.40. lib directory. PublishMQTTPublish Sensor Data Via MQTTThis is our example MiniFi flow. We can drop out the GetFile (to grab images) and push to NiFi Flow if we just want to do a simple MQTT use case.I am using CloudMQTT, you can use any MQTT broker like Mosquitto or HiveMQ.Ingest in Apache NiFi 1.5We can process data from MQTT and/or standard Apache NiFi S2S HTTPS.Processing continues as regular. I add another route for mqtt source.References:If you are doing SSL, see https://community.hortonworks.com/articles/47854/accessing-facebook-page-data-from-apache-nifi.html

This article just gives an example of how 'grant'/'revoke' works when the Hive plugin is enabled with Ranger in CDP. A user who is 'admin' in Ranger, can manage access to Hive tables via 'grant'/'revoke' operation. In Ranger UI > Settings > Users and Groups > Users Note: User 'hive' is in role 'Admin' On the beeline, login as user 'hive'. Run the grant command to give select privileges on a table:   0: jdbc:hive2://a.b.c.co> grant select on table mix to user mugdha;INFO : Compiling command(queryId=hive_20211021024819_c3de84a7-a312-4a1f-9a8d-8b328cced054): grant select on table mix to user mugdhaINFO : Semantic Analysis Completed (retrial = false)INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)INFO : Completed compiling command(queryId=hive_20211021024819_c3de84a7-a312-4a1f-9a8d-8b328cced054); Time taken: 0.022 secondsINFO : Executing command(queryId=hive_20211021024819_c3de84a7-a312-4a1f-9a8d-8b328cced054): grant sel

Step-by-step instructions on how-to install a CRAN package from a local repo - without an internet connection. I'll be installing the package called "tidyr". In order to fully install the package, I need to first download tidyr and all dependencies. To do this, I use the CRAN PACKAGE: https://cran.r-project.org/src/contrib/PACKAGES 1) Finding "Package: tidyr", I can see the dependencies (imports): Imports: dplyr (>= 0.7.0), glue, magrittr, purrr, Rcpp, rlang, stringi, tibble, tidyselect and from them, I can build a list of all the imported packages I need: assertthat_0.2.0.tar.gz BH_1.66.0-1.tar.gz bindr_0.1.1.tar.gz bindrcpp_0.2.tar.gz cli_1.0.0.tar.gz crayon_1.3.4.tar.gz dplyr_0.7.4.tar.gz glue_1.2.0.tar.gz magrittr_1.5.tar.gz pillar_1.2.1.tar.gz plogr_0.1-1.tar.gz purrr_0.2.4.tar.gz Rcpp_0.12.16.tar.gz rlang_0.2.0.tar.gz stringi_1.1.7.tar.gz tibble_1.4.2.tar.gz tidyr_0.8.0.tar.gz tidyselect_0.2.4.tar.gz utf8_1.1.3.tar.gzR6_2.2.2.tar.gzpkgconfig_2.0.1.tar.gz 2) Do

By Default, HiveServer2 and HiveMetastore does not have configuration for HeapDump on OOM. Production clusters have OOM and since the HeapDump on OOM is not configured, root cause analysis of the issue is obstructed.Navigate as below in Ambari:Ambari UI > Hive > Configs > Advanced hive-env > hive-env templateAdd following :if [ "$SERVICE" = "metastore" ]; then export HADOOP_HEAPSIZE={{hive_metastore_heapsize}} # Setting for HiveMetastoreelse export HADOOP_HEAPSIZE={{hive_heapsize}} # Setting for HiveServer2 and Clientfiexport HADOOP_CLIENT_OPTS="-Xmx${HADOOP_HEAPSIZE}m -Xloggc:/var/log/hive/gc.log-$SERVICE-`date +'%Y%m%d%H%M'`-XX:ErrorFile=/var/log/hive/hive-metastore-error.log-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/hive/$HADOOP_CLIENT_OPTS"if [ "$SERVICE" = "hiveserver2" ]; thenexport HADOOP_CLIENT_OPTS="-Xmx${HADOOP_HEAPSIZE}m -Xloggc:/var/log/hive/gc.l

HBase, Phoenix And Ranger In Part 1, Part 2 and Part3 of this article series , we discussed internals of Phoenix Index maintenance and major issues hit around this feature. In this article we will discuss about Phoenix - Ranger relationship , how it works and what had broken until recently which caused reporting of several issues.How native HBase authorization work:ACLs in HBase are implemented as a coprocessor called AccessController. (hbase.security.authorization=true). Users are granted specific permissions such as Read, Write, Execute, Create, Admin against resources such as global, namespaces, tables, cells, or endpoints. ( all self explanatory) There is an additional user called “Superuser”. Superusers can perform any operation available in HBase, to any resource. The user who runs HBase on your cluster is a superuser, as are any principals assigned to the configuration property hbase.superuser in hbase-site.xml.Much more details on this subject are hereHow things Changed with R

Issues with Global IndexesIn Part 1 of this article series, we discussed the internals of Index maintenance, in this part we will cover some of the major issues we face during the life cycle of Index maintenance.Before we get into issues, we need to understand various “states” of Index table which reflect its health in general.BUILDING("b") : This will partially rebuild the index from the last disabled timestamp UNUSABLE (“d”) / INACTIVE ("i") : This will cause the index to no longer be considered for use in queries, however index maintenance will continue to be performed.ACTIVE("a") : Index ready to use and updated. DISABLE("x") : This will cause the no further index maintenance to be performed on the index and it will no longer be considered for use in queries.REBUILD("r") : This will completely rebuild the index and upon completion will enable the index to be used in queries again.---What happens when an Index update fails for any reason :The answer is not straight as there are choi

HDF or CFM best practices guide to configuring your system and NiFi for high performance dataflows.Note: The recommendation outlined in this article are for the NiFi service and apply whether the NiFi service is being deployed/managed via Ambari, Cloudera Manager, or neither.  NiFi is pre-configured to run with very minimal configuration needed out of the box. Simply edit the nifi.properties file located in the conf directory by adding a sensitive props key (used to encode all sensitive properties in flow) and you are ready to go. This may get you up and running, but that basic configuration is far from ideal for high volume/high performance dataflows. While the NiFi core itself does not have much impact on the system in terms of memory, disk/network I/O, or CPU, the dataflows that you build have the potential of impacting those performance metrics quite a bit. Some NiFi processors can be CPU, I/O or memory intensive. In fact some can be intensive in all three areas. We

While upgrading the cluster from HDP-2.6 to HDP-3.0, not able to generate ambari-infra-solr migration script due to below error .[root@hostname ambari-infra-solr-client]# /usr/bin/python /usr/lib/ambari-infra-solr-client/migrationConfigGenerator.py --ini-file $CONFIG_INI_LOCATION --host <ambari server> --port 8443 -s --cluster DEV --username rchaman --password rchaman --backup-base-path=/tmp/back/ --java-home /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre --ranger-hdfs-base-path=/ranger/auditStart generating config file: ambari_solr_migration.ini ...Get Ambari cluster details ...Set JAVA_HOME: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jreService detected: ZOOKEEPERZookeeper connection string: zk1.openstack:2181,zk2.openstack:2181,zk3.openstack:2181Service detected: AMBARI_INFRA_SOLRInfra Solr znode: /infra-solr_test2018-11-14 14:38:59,104 - Get clusterstate.json: waiting for 5 seconds before retyring again (retry count: 1)2018-11-14 14:39:04,1

There are certain circumstances where the Ambari CA needs to be recreated. Maybe it was corrupted, maybe Ambari's CA certificate expired, etc.... This how-to, assumes that the certificates signed by the Ambari CA are replaceable - which is generally the case for certificates used by Ambari agents for 2-way SSL connections. In the end, the Ambari server and all the agents will be restarted, causing a new CA certificate to be created along with new SSL certificates for each of the Ambari agents. On the Ambari server: Stop the Ambari serverBackup /var/lib/ambari-server/keys and it child directoriesDelete the following files from /var/lib/ambari-server/keysca.keyca.csrca.crtpass.txtkeystore.p12*.csr*.crtDelete the following files from /var/lib/ambari-server/keys/dbindex.txt.oldindex.txt.attr.oldserial.oldTruncate the following files from /var/lib/ambari-server/keys/dbindex.txtindex.txt.attrEdit the following files from /var/lib/ambari-server/keys/dbserialset the contents to be exactly00De

In Ambari when try to upgrade to HDP/HDF versions - after adding a new repo version, some times it might show disabled "install packages" button. There can be multiple reasons for this.1. Check to stack_id which stack_id clusters table is pointing to. it should be pointed to correct current stack_id if not correct with update query2. check clusterstate table - it should be pointing to correct current stack_id if not correct with update query3. How to find my correct current stack_idrun "select stack_id, version from repo_version" query - check the stack_id for your current hdp/hdf version.4. so step1 and step2 should be pointing to stack_id returned in step3.There can be many other reasons like not having "upgrades" folder in stack folder or some other.

When we access an HDFS directory using Ambari FileView then we see max 5000 sub directories/contents per page via File View. As following:We see a message "Showing 5000 files or folders of xxxx".The default "views.files.max.files.per.page" limit is set to 5000 as following: /ambari/view/commons/hdfs/FileOperationService.java#L53-L56This property was added from Ambari 2.6 as part of JIRA: https://issues.apache.org/jira/browse/AMBARI-21890 to avoid browser hung state while opening a HDFS folder which has huge number of files.So if user wants to see more items per page then it can be achieved as following:1. In order to set/add a new property with the value with a bit large page size please adjust the following property inside "ambari.properties". For example i am setting it to 7000# grep 'views.files.max.files.per.page' /etc/ambari-server/conf/ambari.propertiesviews.files.max.files.per.page=70002. Then restart the Ambari Server.# ambari-server restart3. Once ambari server is restarted th

In part 1 of, we looked at how we could use deep learning to classify Melanoma, using transfer learning with a pre-trained VGG16 convolutional neural network. Here we take a look at how to implement this using TensorFlow and Keras.First we import the packages we need:import numpy as npfrom keras.preprocessing.image import ImageDataGeneratorfrom keras.models import Sequentialfrom keras.layers import Dropout, Flatten, Densefrom keras import applicationsThese packages include, the network layers we will be using, the applications (which includes the pretrained VGG16 model), and some image pre-processing utilities.Next, we “strip” our model of it’s top layers. This is as simple as one line of code in Keras.base_model = applications.VGG16(include_top=False,weights='imagenet')Next we take our images of skin, and we transform them using Keras image pre-processing utility:datagen = ImageDataGenerator( rotation_range=40, width_shift_range=0.2, height_shift_range=0.2,

I have provided below the implementation steps for integrating KNOX with Loadbalancer assuming once your Loadbalancer is ready.SSL connection should terminate on Knox servers at Loadbalancer side Sticky session should be enabled. Currently the SSL connection terminates at load balancer side and internally it encrypts and loops through one of the KNOX channels. JKS file creation OPEN in IE browser the load balancer URL Example:-https://hadoop-knox.dev.XXXXXX.com/ Click on lock symbol click on view certificates and Certificate pathchoose Root click on view certificate--details--copytofile--base 509 format --save as .pem filechoose intermediateIssuer CA click on view certificate--details--copytofile--base 509 format --save as .pem filechoose loadbalncercert and click on details--copytofile--base 509 format --save as .pem filecopy these 3 files into knox edge node. I have copied to certfiles folder /tmp/knoxhacerts/new/certfiles/lb-rootca.pem /tmp/knoxhacerts/new/certfiles/lb-intermed

NiFi Restricted components are those processors, controller services, or reporting tasks that have the ability to run user-defined code or access/alter localhost filesystem data.-The NiFi User guide explains this as follows:-----------------------------------------Restricted components will be marked with a  icon next to their name. These are components that can be used to execute arbitrary unsanitized code provided by the operator through the NiFi REST API/UI or can be used to obtain or alter data on the NiFi host system using the NiFi OS credentials. These components could be used by an otherwise authorized NiFi user to go beyond the intended use of the application, escalate privilege, or could expose data about the internals of the NiFi process or the host system. All of these capabilities should be considered privileged, and admins should be aware of these capabilities and explicitly enable them for a subset of trusted users. Before a user is allowed to create and modify

Update:See here for a Docker on YARN sandbox solution:https://community.hortonworks.com/articles/232540/docker-on-yarn-sandbox.htmlOverviewThis guide has been tested with and without Kerberos on HDP 3.0.1.YARN offers a DNS service backed by Zookeeper for service discovery, but that can be challenging to setup. For a quickstart scenario, I will use docker swarm and an overlay network instead. If your environment is a single host, the networking is even simpler. This configuration is not recommended for production.I will use pssh to run commands in parallel across the cluster based on a hostlist file and a workerlist file. The hostlist file should contain every host in the cluster, and the workerlist file should include every node except for the one chosen to be the docker swarm master node.PrerequisitesInstall HDP 3.0.1 with or without KerberosInstall Docker on every host in the cluster#pssh -i -h hostlist -l cloudbreak -x "-i ~/cloudbreak.pem -o 'StrictHostKeyChecking no'" "echo hostna

Hadoop archives is one of the methodology which is followed to reduce the load on the Namenode by archiving the files and referring all the archives as a single file via har reader.Testing: To understand the behavior of the HAR, we try following example. 1. Create test folders harSourceFolder2 : Where the initial set of small files are stored. Ex. (In HDFS ) /tmp/harSourceFolder2 harDestinationFolder2 : Where the final archived files are stored. Ex. (In HDFS) /tmp/harDestinationFolder2 2. Ingest small files in the source folder.sudo -u hdfs hadoop fs -copyFromLocal /tmp/SampleTest1.txt /tmp/harSourceFolder2NOTE: thid command shows one file "SampleTest1", however in our example we used five files with index extending till 5 (SampleTest5.txt)3. Capture fsck report across the "/" and NN report after small files are ingested.sudo -u hdfs hdfs fsck / -files > ./fsckWhenFilesCreated.txt143 files and directories, 48 blocks = 191 total filesystem object(s).4. Execute hadoop archive c

Introduction This article continues where part 2 left off. It describes how to enable two new features that make the HDFS NameNode more responsive to high RPC request loads. Audience This article is for Apache Hadoop administrators who are familiar with HDFS and its components. If you are using Ambari you should know how to manage services and configurations with Ambari. It is assumed that you have read part 1 and part 2 of the article. RPC Congestion Control Warning: You must first enable the service RPC port (described in part 1) and restart services so the service RPC setting is effective. Failure to do so will break DataNode-NameNode communication. RPC Congestion Control is a relatively new feature added by the Apache Hadoop Community to help Hadoop Services respond more predictably under high load (see Apache Hadoop Jira HADOOP-10597). In part 2, we discussed how RPC queue overflow can cause request timeouts, and eventually job failures. This problem can be mitigated if the NameNo

Introduction Before being able to predict one's Beast Mode Quotient, we must find an easy way to ingest data from multiple different sources, in this case flat files and APIs. Luckily for everyone in the world, HDF, specifically Nifi is designed to handle multiple data inputs and outputs extremely efficiently, as I will detail in this tutorial. A few notes about the choices of technologies selected for this tutorial and series of article: Target - MySQL: I selected MySQL because this is what my BMQ app is going to use, but it could be any target depending on your use-case. Sources - Fitbit, Strava, MyFitnessPal: There are a bunch of options out there to track health. Fitbit and Strava are the only ones offering a comprehensive and public API (looking at you Garmin, and Apple Health; both have no public APIs, and make it not easy to extract data via file, either having to parse .fit files or only exporting the full set of data); as for MyFitnessPal, I could have used Fitbit API (afte

Usually hive treats \n as newline character and everytime we ingest the data it will split single line into multiple lines depending on no of \n in the text.For eg. Step1: Create a table and try ingesting the data using insert into table command.Step2: List the contents of table. We will observe that single line with '\n' in between is split in two lines.WorkaroundThere are two ways to ingest data containing newline character1. Using escape character 2. Using load data inpath commandSolution1: When we use escape character "\" with "\n", hive will ignore \n as newline but will treat it as string. (abcd\ndefg)querying the tableSolution 2: Using "load data inpath" command.Step1: Upload the file on HDFSStep2: Load data from file to table

Recently, I have observed there are lot of questions in HCC on - how to reset Atlas and Is there a way to delete the registered types in Atlas. So, I thought of sharing this article with the community to clarify these queries.To give some context, Atlas uses HBase as its default datastore when its managed by Ambari. So, basically it uses two Hbase tables to store all its metadata.'atlas_titan' : stores all the metadata from various sources.'ATLAS_ENTITY_AUDIT_EVENTS': stores the audit information of the entities in AtlasThe above two table names can be changed using two properties "atlas.graph.storage.hbase.table" and "atlas.audit.hbase.tablename" respectively in the atlas-application.properties.Now, coming back to the actual question - how to wipe-out the metadata from the Atlas? Follow below steps to achieve the same.Stop Atlas via Ambari.In hbase terminal, to disable hbase table, run this command.disable 'atlas_titan'In hbase terminal, to drop hbase table, run this command.drop 'atl

IntroductionThe purpose of this article is to use Apache NiFi to ingest a huge csv file (few million rows) into a Kafka topic.The tricky point when dealing with big file (millions of rows) is that processing the file by NiFi on row-level will generate flow files for every row, this can produce java memory error. As a work around, we need to limit the generated number of rows by splitting the rows on multiple stages. In this article, the file was around 7 million rows, and 6 dividing stages (100K, 10K, 1K, 100, 10 then 1) were used to limit number of generated flow files.The other trick is to increase the “Splits” queue’s “Back Pressure Data Size Threshold” to an adequate size for handling file size, by default it’s 1GB. In this article, 2GB is used. Assumptions & DesignA standalone NiFi and Kafka instance is to be used for this exercise.The following NiFi flow will be used to split the workload of the multi-million row csv file to be ingested by dividing the ingestion into multi-s

In preparation for my talk at the Philadelphia Open Source Conference(https://phillyopensource.splashthat.com/), Apache Deep Learning 201, I wanted to have some good images for running various Apache MXNet GluonCV Deep Learning Algorithms for Computer Vision. See: https://gluon-cv.mxnet.io/Using Apache open source tools - Apache NiFi 1.8 and Apache MXNet 1.3 with GluonCV I can easily ingest live traffic camera images and run Object Detection, Semantic Segmentation and Instance Segmentation.Code: https://github.com/tspannhw/ApacheDeepLearning201It's so easy, I am running multiple on the data to see which gives me the results I like. I am like YOLO which is a one of my old favorites. yolonifitraffic.pydemo_mask_rcnn_nyc.pydemo_deeplab_nyc.pySo we can find the cars and people in these webcams. Use cases can be around traffic optimization, public safety and advertisement optimization. Due to licensing, I thought it better not to show Traffic Camera data here.To industrialize a

With the arrival of the Hive3Streaming processors, performance has never been better between NiFi and Hive3. Below, we'll be taking a look at the PutHive3Streaming (separate from the PutHiveStreaming processor) processor and how it can fit into a basic Change Data Capture workflow. We will be performing only inserts on the source Hive table and then carrying over those inserts into a destination Hive table. Updates are also supported through this process with the addition of a 'last_updated_datetime' timestamp, but that is out of scope for this article. This is meant to simulate copying data from one HDP cluster to another - we're using the same cluster as the source and destination here however.Here is the completed flow. Take note that most of this flow is just keeping track of the latest ID that we've seen, so that we can pull that back out of HDFS periodically and query Hive for records that were added beyond that latest ID. These last two processors, SelectHive3QL and PutHive3Stre