Resources

The below steps describe how to change the Namenode log level while logged on as hdfs with the below steps, without the need to restart the namenodeGet the current log level$ hadoop daemonlog -getlevel {namenode_host}:50070BlockStateChangeDesired OutputConnecting to http://{namenode_host}:50070/logLevel?log=BlockStateChange SubmittedLogName:BlockStateChangeLogClass: org.apache.commons.logging.impl.Log4JLoggerEffectiveLevel: INFOChange to DEBUG$ hadoop daemonlog -setlevel {namenode_host}:50070BlockStateChange DEBUG Desired OutputConnecting to http://{namenode_host}:50070/logLevel?log=BlockStateChange&level=DEBUG SubmittedLogName:BlockStateChangeLogClass: org.apache.commons.logging.impl.Log4JLoggerSubmittedLevel: DEBUG SettingLevel to DEBUG ...EffectiveLevel: DEBUGValidate DEBUG mode$ hadoop daemonlog -getlevel {namenode_host}:50070BlockStateChangeDesired OutputConnecting to http://{namenode_host}:50070/logLevel?log=BlockStateChange SubmittedLogName:BlockStateChangeLogClass: org.ap

From Ambari 2.7 (after an ambari-server upgrade) it is required to backup / migrate and restore Infra Solr collection data.That is required because from Ambari 2.7 Infra Solr uses Solr 7 instead of Solr 5 and optimize command won't work on a Lucene 5 index, so it is required to do this offline.In order to not get stuck on any steps it is useful if we understands correctly what happens in the background as the documentation covers the happy path with the usage of /usr/lib/ambari-infra-solr-client/ambariSolrMigration.sh script. (https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.0.0/bk_ambari-upgrade/content/backup_and_upgrade_ambari_infra_data.html). But fortunately every step can be done manually as well.Backup/Migration/Restore is only needed for Atlas and Ranger. Log Search will have a totally new schema, so there the Prerequisites: 1. First of all, upgrading the Ambari Infra Solr service is the next step right after Ambari server upgrade. (backup old Solr data then upgrade Infra S

In this tutorial, I create a NiFi cluster from the default blueprint provided with Cloudbreak 2.5.0 TP, but this post has been updated to reflect what is available in Cloudbreak 2.9.0.Update for Cloudbreak 2.9.0This post was originally written for Cloudbreak 2.5.0 TP, which introduced support for creating HDF Flow Management (NiFi) clusters. It has been updated to reflect the latest features available in Cloudbreak 2.9.0 general availability release.Cloudbreak 2.5.0 TP introduced support for creating HDF Flow Management (NiFi) clusters.The subsequent release, Cloudbreak 2.6.0 TP, introduced support for creating HDF Messaging Management (Kafka) clusters. Cloudbreak 2.7.0 GA introduced these as general availability features. In Cloudbreak 2.9.0 GA, two HDF 3.3 blueprints are included by default, one for Flow Management (NiFi) and one for Messaging Management (Kafka).What is Cloudbreak?Cloudbreak simplifies the provisioning, management, and monitoring of on-demand HDP and HDF clusters in

In this tutorial, I go through the steps of creating a data lake instance and creating a cluster attached to the data lake. I cover you how to use Cloudbreak web UI and CLI to perform these steps.Assumptions:These steps assume that you already have a running instance of Cloudbreak and that you have created a Cloudbreak credential that allows you to connect to your cloud provider.All the resources that I am creating in the tutorial are running on AWS. Whatever cloud provider you are using, I assume that you have an SSH key set up (if needed) and some network and subnet that you would like to use. Note that the data lake functionality cannot be used with OpenStack.The steps require that you create some resources (databases, LDAP, cloud storage) outside of Cloudbreak. See the "Meet the Prerequisites" section.This functionality is technical preview in Cloudbreak 2.9.x. It is not suitable for production.Update for Cloudbreak 2.9.0This post has been updated to reflect the latest functionalit

Cloudbreak 2.9.0 is now available! It is a general availability (GA) release, so - with an exception of some features that are marked as TP - it is suitable for production. Try it nowUpgrade to 2.9.0Quickly deploy by using quickstart on AWS, Azure, or Google CloudInstall manually on AWS, Azure, Google Cloud, or OpenStackNew featuresCloudbreak 2.9.0 introduces the following new features. While some of these features were introduced in Cloudbreak 2.8.0 TP, others are brand new:FeatureDescriptionDocumentationSpecifying resource group name on AzureWhen creating a cluster on Azure, you can specify the name for the new resource group where the cluster will be deployed. Resource group nameMultiple existing security groups on AWSWhen creating a cluster on AWS, you can select multiple existing security groups. This option is available only when an existing VPC is selected. Create a cluster on AWSEBS volume encryption on AWSYou can optionally configure encryption for EBS volumes attached to clus

How Does Tez determine the number of reducers? How can I control this for performance?In this article, I will attempt to answer this while executing and tuning an actual query to illustrate the concepts. Then I will provide a summary with a full explanation. if you wish, you can advance ahead to the summary.-------------0. Prep Work and Checklist We followed the Tez Memory Tuning steps as outlined in https://community.hortonworks.com/content/kbentry/14309/demystify-tez-tuning-step-by-step.htmlWe setup our environment, turning CBO and Vectorization On. set hive.support.sql11.reserved.keywords=false;set hive.execution.engine=tez;set hive.cbo.enable=true;set hive.compute.query.using.stats=true;set hive.stats.fetch.column.stats=true;set hive.stats.fetch.partition.stats=true;set hive.vectorized.execution.enabled=true;set hive.vectorized.execution.reduce.enabled = true;set hive.vectorized.execution.reduce.groupby.enabled = true;set hive.exec.parallel=true;set hive.exec.parallel.thread.num

Recently for a client I explored integrating nifi with Graylog. There is an AWS EC2 image for a full graylog install with the documentation for this to be found here http://docs.graylog.org/en/1.3/pages/installation/aws.html Configure a GELF HTTP connector to these settings http://docs.graylog.org/en/1.3/pages/sending_data.htmlMake a Python Script generating Dummy JSONS '{"short_message":"CHRIS HERE", "host":"example.org", "facility":"test", "_foo":"bar"}'Have a nifi HTTP post processor pointed at the Graylog inputConfirm messages have been received by graylog

Using Deployed Models as a Function as a Service Using Cloudera Data Science Workbench with Apache NiFi we can easily call functions within our deployed models from Apache NiFi as part of flows. I am working against CDSW on HDP (https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_hdp.html), but it will work for all CDSW regardless of install type.In my simple example, I built a Python model that uses TextBlob to run sentiment against a passed in sentence. It returns Sentiment Polarity and Subjectivity which we can immediately act upon in our flow.CDSW is extremely easy to work with and I was up and running in a few minutes. For my model, I created a python 3 script and a shell script for install details. Both of these artifacts are available here: https://github.com/tspannhw/nifi-cdswMy Apache NiFi 1.8 flow is here (I use no custom processors): cdsw-twitter-sentiment.xmlDeploying a Machine Learning Model as a REST ServiceOnce you login to CDSW

1. Now let's configure Knox to use our AD for authentication. Replace below content in Ambari > Knox > Config > Advanced topology<topology> <gateway> <provider> <role>authentication</role> <name>ShiroProvider</name> <enabled>true</enabled> <param> <name>sessionTimeout</name> <value>30</value> </param> <param> <name>main.ldapRealm</name> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> </param> <!-- changes for AD/user sync --> <param> <name>main.ldapContextFactory</name> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value&g

Introduction into Process Group Variables The ability configure variables at runtime in the UI was introduced in Apache NiFi 1.4 and will subsequently land in HDF 3.1. Variables can be referenced in property values of Processors and Controller Services that support expression language. This provides a centralized place to configure common values that may be referenced by multiple components and values that may be environmentally dependent. This guide provides a brief introduction to their usage. Configuration Variables are configured in a dialog that is accessed through the context menu. The menu item is available when the selection is a single Process Group. Below is an example of what the variable dialog would look like. The table will list all variables that are available in the scope of the current Process Group, the variable name, and variable value. Only variables defined in the current Process Group are editable. The table also contains a navigation link to go to the Process Gro

STOP Command::curl -u admin:admin -H "X-Requested-By:ambari" -i -X PUT http://172.26.78.29:8080/api/v1/clusters/Mycluster/hosts/prakash-ambariagent-node3/host_components/FLUME_HANDLER -d '{"RequestInfo":{"context":"Stop Flume","operation_level":{"level":"HOST_COMPONENT","cluster_name":"Mycluster","host_name":"prakash-ambariagent-node3","service_name":"FLUME"}},"Body":{"HostRoles":{"state":"INSTALLED"}}}'START Command:: curl -u admin:admin -H "X-Requested-By:ambari" -i -X PUT http://172.26.78.29:8080/api/v1/clusters/Mycluster/hosts/prakash-ambariagent-node3/host_components/FLUME_HANDLER -d '{"RequestInfo":{"context":"Start Flume","operation_level":{"level":"HOST_COMPONENT","cluster_name":"Mycluster","host_name":"prakash-ambariagent-node3","service_name":"FLUME"}},"Body":{"HostRoles":{"state":"STARTED"}}}'RESTART Command::curl --insecure -v -u admin:admin -H "X-Requested-By:ambari" -i -X POST http://prakash-ambariserver-node1:8080/api/v1/clusters/Mycluster/requests -d '{"RequestInfo":{"

Log Log LogSudo logs have a lot of useful information on hosts, users and auditable actions that may be useful for cybersecurity, capacity planning, user tracking, data lake population, user management and general security.Symbol Model 1Step 1 - Get a FileStep 2 - Split Into LinesStep 3 - Set the Mime Type to Plain TextStep 4 - Extract Grok Step 5 - Action! Checking for More Options (All Named elements in the GrokPatterns)Example Sudo Log (could be /auth.log, /var/log/sudo.log, secure, ...)Jan 31 19:17:20 princeton0 su: pam_unix(su-l:session): session opened for user ambari-qa by (uid=0)Jan 31 19:17:20 princeton0 su: pam_unix(su-l:session): session closed for user ambari-qaJan 31 19:18:19 princeton0 su: pam_unix(su-l:session): session opened for user zeppelin by (uid=0)Jan 31 19:18:19 princeton0 su: pam_unix(su-l:session): session closed for user zeppelinJan 31 19:18:20 princeton0 su: pam_unix(su-l:session): session opened for user ambari-qa by (uid=0)Jan 31 19:18:20 princeton0 su: pam

Hive Pre Upgrade tool command fails after not able to access HDFS The command is not able to find the right configuration of hadoop core components, primarily due to unavailability of hadoop core component configuration files in the command. Resolution steps There are different ways to export the hadoop core configuration, one of them is below. 1. Follow the below HWX document to make sure you have properly followed the steps. (Check appropriate link for your version) https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-upgrade-major/content/prepare_hive_for_upgrade.html 2. Follow till "Procedure for compacting Hive tables (no Kerberos)" step 3 and before running the pre-upgrade tool command, export hadoop and hive env filessource /etc/hadoop/conf/hadoop-env.sh source /etc/hive/conf/hive-env.sh 3. Now while running the pre-upgrade tool command, include two more paths in the command (/etc/hadoop/conf:/etc/hive/conf). So the command below similar to below. java

From Database MetaData to Data Ingest in a Few Quick Steps A company wants to know when new tables are added to a JDBC source (say a RDBMS). Using the ListDatabaseTables processor we can get a list of TABLEs (also views, system tables and other database objects), but for our purposes we want tables with data. I have used the ngdbc.jar from SAP HANA to connect and query tables with ease. For today's example I am connecting to MySQL as I have a MySQL database available for use and modification. Pre-Step mysql -u root -p test < person.sqlCREATE USER 'nifi'@'%' IDENTIFIED BY 'reallylongDifficultPassDF&^D&F^Dwird';GRANT ALL PRIVILEGES ON *.* TO 'nifi'@'%' WITH GRANT OPTION;COMMIT;mysql> show tables;+----------------+| Tables_in_test |+----------------+| MOCK_DATA || mock2 || personReg || visitor |+----------------+4 rows in set (0.00 sec) I created a user to use for my JDBC Connection Pool in NiFi to read the metadata and data. These ta

Introduction Finally, we get to put all this together! This is less of a tutorial, and more of a show case of the capabilities of this solution. Please refere to previous articles to build this on your own. The meat of this article is showcasing the React/NodeJS application you can build to communicate with Nifi, Cloudbreak and your ephemeral cluster running Zeppelin and Spark. Architecture Below is a high-level architecture of the solution: You can find all the code for this here: https://github.com/paulvid/bmq-app The code includes: Latest Zeppelin Prediction Notebook All Nifi templates All Blueprints/Recipes for Cloudbreak The source code of the app Agenda This article is divided in the following sections: Section 1: Monitoring your current data with Nifi Section 2: Launching an ephemeral cluster with Cloudbreak Section 3: Running a prediction model with Spark Zeppelin Section 1: Monitoring your current data with Nifi   Section 2: Launching an ephemeral cluster with Cloud

Introduction It's been a while, but I'm finally finishing this series on Best Mode Quotient. This article is fairly straight forward as we look how to build the infrastructure of model training using Zeppelin and Spark.Pre-Requisites In order to run this install, you will have to have deployed a BMQ ephemeral cluster as detailed in this article and this repository. Moreover you will have to have gathered data on your fitness as explained in part 1 of these series. Based on this data, I created 3 indexes: Intensity Index: How intense was the workout I did that day (based on distance, pace and elevation) Fatigue Index: How much rest I had on that day (based on hours of sleep and rest heart rate) BMQ Index: How my BMQ fairs compare to the max BMQ (5) You can then agglomerate these 3 indexes, having BMQ and Fatigue on the same day correlating to the Intensity Index of the previous day. All data will be shared in the last article of the series. You will also notice that the model uses

Customizing Atlas: Summary of Work to DateArticleKey PointsCustomized Type(s) DevelopedPart 1: Model governance, traceability and registry Quick primer on Atlas types, entities, attributes, lineage, searchQuick primer on customizing AtlasUse the Atlas Rest API to customize any type, entity, attributes you wishCustomizations integrate seamlessly with out of the box Atlas lineage and search capabilitiesNotes on operationalizing model to represent your deployed data science and complex Spark ETL models (what was deployed, which version, when, what are its concrete artifacts, etc)Part 2: Deep source metadata & embedded entities Use the Atlas Rest API to customize any type/entity/attributes you wishYou can use a hyplerlinked entity (vs text) as a value to an attribute (embedded entity pattern)HDFS entities can hold deep metadata from source device to represent a hardware device (in this case gene sequencing device)gene_sequence to represent gene sequencing data landed in HDFS, as well

In Apache NiFi, Controller Services are shared services that can be used by Processors, Reporting Tasks, and other Controller Services. In order to modify a Controller Service, all referencing components must be stopped. This means that the user attempting to modify the Controller Service, must also have permissions to modify all referencing components. With the introduction of multi-tenant authorization, Controller Service scoping was updated to ensure service usage does not become so broad that it becomes impossible to update the Controller Service because no users are allowed to modified all components that reference it. Within the dataflow, to scope a Controller Service and limit it's availability, a Controller Service can be created within any Process Group. Controller Services created within a Process Group will be available to all descendent components. This means that the service will be available to all Processors and Controller Services defined in that Process Group and below

CVE-2018-8012 vulnerability is reported here, https://nvd.nist.gov/vuln/detail/CVE-2018-8012   o Affected Versions: Apache ZooKeeper before 3.4.10, and 3.5.0-alpha through 3.5.3-beta. o Mitigations: - The Apache JIRA (ZOOKEEPER-1045) is fixed @ Zookeeper 3.4.10. - Currently, the Fix for ZOOKEEPER-1045 (Zookeeper 3.4.10) isn't planned for any HDP release as the vulnerability is less of a vulnerability and more of a lack of a feature - Zookeeper version 3.4.6 is present from HDP 2.3.0 to HDP 2.6.x   o Workaround: - The workaround for this is to implement firewall rules to disallow connections from ports other than 2181. - Customer may try to move the ZooKeeper Election and Follower ports into the privileged port range, but this would require additional logic to start ZK as root and then drop privileges which does not exist as of now. - As such, the feasible workaround for this CVE is to configure firewall to allow communication [for Leader Election] within cluster. A possibl

Introduction SoChain provides a fast set of public freely available APIs (don't abuse them) to access information on various networks. If you need this for critical work, please donate: https://chain.so/address/devfund. One of the things you will see in this simple flow is that NiFi excels in ingesting REST and working with JSON. As you can see with split it up, shred it, filter it, manipulate and extract from it. With the resulting usable objects we build a schema that will also us to do record processing. Once we have a set of records with a schema I can store it to I just hosted a Future of Data Princeton Meetup in Woodbridge New Jersey with some amazing speakers sponsored by ChainNinja. While this was all about Blockchain for Enterprise and no cryptocurrency was involved it made we want to investigate some cryptocurrency data. As you can see, manipulating complex JSON data, filtering, modifying, routing and scripting with it's values is trivial in Apache NiFi.

The terms of service for this community forum are posted here.

Less than 5% of the support cases have recommendations to engage professional services. I have seen this message create mixed emotions in the customer's mind. I would like to wear both customers and support engineer's hat to see how this can be handled efficiently.As a customer, I will be delighted if solutions come instantly, a setting like “hive.run.query.faster=true or spark.fix.error=true” would be very ideal. Requesting customers to connect with a different team could often be seen as a deflection strategy from support. Also, professional service engagement has its own sales cycle and costs are attached to it. This results in frustration.Let’s take an example of one such redirection, Customer requests support’s help refine their use case and suggest the right technology stack. From a support point of view, this request is considered out of scope and the engineer would direct the customer to engage professional services. As a professional services consultant in my past life, use ca

Working with a Proximity Beacon Network Part 1Introduction:In a retail environment, we want to be able to interact with people within the store. Our beacons can provide hyperlocalized information and also help us determine what's going on in the store for traffic patterns. Our beacons are also giving us temperature and other reading as any sensor we may have in our IoT retail environment.I have set up three Estimote Proximity beacons in an indoor environment broadcasting both Estimote and IBeacon messages.iBeacon is a Bluetooth advertising protocol by Apple that is built into iPhones. In this article we ingest, filter and route the data based on the beacon IDs. In a following article we will stream our data to a data store(s) and run machine learning and analytics on our streaming time series data.I tested the BLE library from the command line with a Python script:Cloud SetupSince we are using our own IoT gateways and networks, I am only using the Estimote Cloud to check the beac

The Data Load ProcessWe looked at the performance that these engines have in the last article, now it’s time to look at how the data got loaded in. There are trade-offs here to be aware of when loading data into each of these engines, as they use different mechanisms to accomplish the task.Hive LoadThere is no immediate need for a Schema-on-Write data load when you are using Hive with your native file format. Your only “load” operation is copying the files from your local file system to HDFS. With schema-on-read functionality, Hive can instantly access data as soon as its’ underlying file is loaded into HDFS. In our case the real data load step was in converting this Schema-on-Read external Hive table data into optimized ORC format, therefore loading it from an external table to a Hive-managed table. This was a relatively short process, coming in much under an hour.HBase LoadContrast that with HBase, where a bulk data load for our sample data set of 200M rows (around 30GB of disk size

As we know that the logs are important however many times we see that the logs consumes a lots of disk space. So if we want to use the log4j provided approach to control the logging behavior in a much efficient way, then in that case we can take advantage of the "apache-log4j-extra" package. More information about the extra packages can be found in :https://logging.apache.org/log4j/extras/In this article we will see how to use the log compression (zip) feature of log4j to compress the NameNode logs on a daily basis automatically.In order to achieve the same we will need the following Steps:Step-1). As the extras features are not shipped with the default log4j implementation hence the users will need to download the "Apache Extras™ for Apache log4j" (Like: apache-log4j-extras) : https://logging.apache.org/log4j/extras/download.htmlExample: For example download the jar "apache-log4j-extras-1.2.17.jar" and place it inside the Hadoop library location./usr/hdp/2.4.2.0-258/hadoop/lib/apache-