Resources

Apache Metron ExplainedApache Metron is a cyber security application framework that provides organizations the ability to ingest, process and store diverse security data feeds at scale in order to detect cyber anomalies and enable organizations to rapidly respond to them.As the diagram above indicates, the Metron framework provides 4 key capabilities:Security Data Lake / Vault - Platform provides cost effective way to store enriched telemetry data for long periods of time. This data lake provides the corpus of data required to do feature engineering that powers discovery analytics and provides a mechanism to search and query for operational analytics.Pluggable Framework - Platform provides not only a rich set of parsers for common security data sources (pcap, netflow, bro, snort, fireye, sourcefire) but also provides a pluggable framework to add new custom parsers for new data sources, add new enrichment services to provide more contextual info to the raw streaming data, pluggable exte

This is an article which describes how to access AMS embedded Hbase in a secured kerberized environment. This is extension of this document For a unsecured cluster, please read the above document however in case of secured environment you may need run one extra export command. The final command will be similar to below syntax<sqline.py path> <ams_collector_hostname>:<embedded_zk_port>:<znode>/usr/hdp/2.5.0.0-1245/phoenix/bin/sqlline.py rchaman.ambari.apache.org:61181:/ams-hbase-unsecurePlease find the below steps to get the parameters as per your environment. 1. We need to make sure that we have the phoenix client packages installed on the host from where you are running command. however, I suggest you to run the command from AMS collector host. If it is not installed then we can install it as following:yum install phoenix -y2. Search for sqlline.py script absolute path[root@c4140-node4 ~]# rpm -qa |grep phoenixphoenix_2_6_4_0_91-4.7.0.2.6.4.0-91.noarch[root

Simple Apache NiFi Operations Dashboard - Part 2Part 1:     https://community.cloudera.com/t5/Community-Articles/Building-a-Custom-Apache-NiFi-Operations-Dashboard-Part-1/ta-p/249060To access data to display in our dashboard we will use some Spring Boot 2.06 Java 8 microservices to call Apache Hive 3.1.0 tables in HDP 3.0 on Hadoop 3.1.We will have our web site hosted and make REST Calls to Apache NiFi, our microservices, YARN and other APIs.As you can see we can easily incorporate data from HDP 3 - Apache Hive 3.1.0 in Spring Boot java applications with not much trouble. You can see the Maven build script (all code is in github.)Our motivation is put all this data somewhere and show it in a dashboard that can use REST APIs for data access and updates. We may choose to use Apache NiFi for all REST APIs or we can do some in Apache NiFi. We are still exploring. We can also decide to change the backend to HBase 2.0, Phoenix or Druid or a combination. We will see. Sprin

Simple Apache NiFi Operations Dashboard This is an evolving work in progress, please get involved everything is open source. @milind pandit and I are working on a project to build something useful for teams to analyze their flows, current cluster state, start and stop flows and have a rich one look dashboard. There's a lot of data provided by Apache NiFi and related tools to aggregate, sort, categorize, search and eventually do machine learning analytics on. There are a lot of tools that come out of the box that solve parts of these problems. Ambari Metrics, Grafana and Log Search provide a ton of data and analysis abilities. You can find all your errors easily in Log Search and see nice graphs of what is going on in Ambari Metrics and Grafana. What is cool with Apache NiFi is that is has SitetoSite tasks for sending all the provenance, analytics, metrics and operational data you need to wherever you want it. That includes to Apache NiFi! This is Monitoring Driven Development (MDD). M

The following example provides a guide to connecting to HBase from Spark then perform a few operations using Spark on the data extracted from HBasePlease add the following repo to your package managerhttp://repo.hortonworks.com/content/repositories/releases/and import the following classcom.hortonworks:shc-core:1.1.0-2.1-s_2.11Keep in mind that new releases may be availableImport the following classesimport org.apache.spark.sql.{SQLContext, _}import org.apache.spark.sql.execution.datasources.hbase._import org.apache.spark.{SparkConf, SparkContext}import spark.sqlContext.implicits._import org.apache.spark.sql.types._The first step in connecting to HBase is defining how Spark should interpret each row of data found in HBase. This is done by defining a catalog to use with the connectordef catalog = s"""{ |"table":{"namespace":"default", "name":"timeseries"}, |"rowkey":"key", |"columns":{ |"col0":{"cf":"rowkey", "col":"key", "type":"string"}, |"col1":{"cf":"cf", "col":"t", "type":"strin

Restarting NiFi nodes fails with an error message such as the one below: <br>File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 278, in Master().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute method(env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 993, in restart self.start(env, upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 152, in start nifi_cli.create_or_update_reg_client(params.nifi_registry_host, params.nifi_registry_url) File "/u/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_cli.py", line 175, in create_or_update_reg_client existing_clients = list_reg_clients() File "/u/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_cli.py", line 144, in list_reg_clients outputType="json" File

This article goes on how Nifi, with the of the OPC UA api along with Druid for dashboarding can help with industrial iot use cases. We have seen a trend where several clients have shown interest in HDP as a platform for storing all of the sensor data and third party data. This allows these companies store their sensor data over a longer period of time, easily corelate them with other data sets and enrich them in a very cost effective manner. Additionally, they can run ML algorithms on this data set to better predict outages, malfunctions or maintenance. Using NiFi,Kafka and Druid, can provide means for a very easy to maintain and superfast ingestion mechanism from the several edge devices be it windmills, substations, meters etc. Druid can then provide a robust platform for real-time querying dashboarding on this sensor data so operations can keep an eagle eye on whats going on at a plant or wind farm.For the pupose of this article, i will be consuming data from a demo OPC server which

Prerequisites 1)Download HDP Sandbox 2)MySQL database (Should already be present in the sandbox)3)Nifi 0.6 or later ( Download and install a new version of NIFI or use Ambari to install NIFI in the sandbox)MySQL setup (Source Database)In this setup we will create a table in MySQL tables and create a few triggers on the tables to emulate transactions. These triggers will find out if the change introduced was an insert or an update also will update the time stamp on the updated/inserted row. ( This is very important as Nifi Will be polling on this column to extract changes based on the time stamp)unix> mysql –u root –punix>Enterpassword:<enter>mysql>mysql> create databasetest_cdc;mysql> create user'test_cdc'@'localhost' identified by 'test_cdc';mysql> GRANT ALLPRIVILEGES ON *.* TO 'test_CDC'@'%' IDENTIFIED BY 'test_CDC' WITH GRANT OPTION;mysql>Flush Privilegesmysql> exit;unix> mysql –u test_cdc –ptest_cdcmysql>create table CDC_TEST(Column_A int, Colu

Adding HBase HA to a blueprint with high availability is a very straightforward process. The blueprint you start from has to include a zookeeper quorum, like the blueprint that provides HA namenodes and resourcemanagers that is discussed in "Cloudbreak blueprint for high-availability Hadoop and Hive". We will add HBase Master services to two master host groups, HBase Regionservers to the datanode host group, and an hbase-site configuration section.Required Zookeeper Quorum To properly specify the zookeepers to be used, you will need at least three separate host groups with cardinality of 1 to provide the zookeepers. In the blueprint provided in the previous article, those hostgroups are: master_mgmt, master_nn1 and master_nn2. In that blueprint, master_mgmt contains the Ambari, metrics and other management services, while master_nn1 and master_nn2 contain the redundant namenodes and resourcemanagers. In this example, we will add redundant HBase masters to those host groups as well

Dr. ElephantDr. Elephant is a performance monitoring and tuning tool for Hadoop and Spark. It automatically gathers a job's metrics, analyzes them, and presents them in a simple way for easy consumption. Its goal is to improve developer productivity and increase cluster efficiency by making it easier to tune the jobs. It analyzes the Hadoop and Spark jobs using a set of pluggable, configurable, rule-based heuristics that provide insights on how a job performed and then uses the results to make suggestions about how to tune the job to make it perform more efficiently. It also computes a number of metrics for a job which provides valuable information about the job performance on the cluster.Build Stepscd ~;sudo yum update -y;sudo yum upgrade -y;sudo curl --silent --location https://rpm.nodesource.com/setup_8.x | sudo bash -sudo yum install -y nodejs wget unzip git zip gcc-c++ make java-1.8.0-openjdk-devel;wget https://github.com/linkedin/dr-elephant/archive/v2.1.7.tar.gz;wget https://dow

While accessing Ambari 2.5 or 2.4 Hive Views (1.5.0/2.0) we see the following error in the log:ERROR [ambari-client-thread-62] ContainerResponse:537 - Mapped exception to response: 500 (Internal Server Error) org.apache.ambari.view.hive2.utils.ServiceFormattedException at org.apache.ambari.view.hive2.client.NonPersistentCursor.getNextRows(NonPersistentCursor.java:132) at org.apache.ambari.view.hive2.client.NonPersistentCursor.fetchIfNeeded(NonPersistentCursor.java:119) at org.apache.ambari.view.hive2.client.NonPersistentCursor.getDescriptions(NonPersistentCursor.java:84) at org.apache.ambari.view.hive2.resources.jobs.ResultsPaginationController.request(ResultsPaginationController.java:145) at org.apache.ambari.view.hive2.resources.jobs.JobService.getResults(JobService.java:361) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.refl

Example topology for kerberos auth and hive:[root@groot1 hive]# cat /etc/knox/2.6.0.3-8/0/topologies/kerberos.xml <topology> <gateway> <provider> <role>authentication</role> <name>HadoopAuth</name> <enabled>true</enabled> <param> <name>config.prefix</name> <value>hadoop.auth.config</value> </param> <param> <name>hadoop.auth.config.signature.secret</name> <value>hadoop12345!</value> </param> <param> <name>hadoop.auth.config.type</name> <value>kerberos</value> </param> <param> <name>hadoop.auth.config.simple.anonymous.allowed</name> <value>false</value>

Overview Last article of the series, and a fun one. As usual, this article is the continuation of part 1, part 2 and part 3, so make sure you read those before starting. The completion of this tutorial will finalize your end to end architecture, as depicted here: [ADD IMAGE] This tutorial will be divided in 2 sections: Section 1: Create a Web Service with Nifi Section 2: Create a consumer application with code Igniter Note: Section 2 will not be detailed. First, because web application development is not the main goal of Hortonworks Community Connection. Second, because I am pretty sure I did a terrible job at it considering it is not my domain of expertise, and I shouldn't spread mediocrity to the world!Section 1: Create a Web Service with Nifi For your entire project to work, you will have to create more than one web service although this section only goes through the creation of one (see the flow overview below). To get all the flows required for your app to work, check githu

Overview Last article of the series, and a fun one. As usual, this article is the continuation of part 1, part 2 and part 3, so make sure you read those before starting. The completion of this tutorial will finalize your end to end architecture, as depicted here: This tutorial will be divided in 2 sections: Section 1: Create a Web Service with Nifi Section 2: Create a consumer application with code Igniter Note: Section 2 will not be detailed. First, because web application development is not the main goal of Hortonworks Community Connection. Second, because I am pretty sure I did a terrible job at it considering it is not my domain of expertise, and I shouldn't spread mediocrity to the world! Section 1: Create a Web Service with Nifi For your entire project to work, you will have to create more than one web service although this section only goes through the creation of one (see the flow overview below). To get all the flows required for your app to work, check gi

Running TensorFlow on YARN 3.1 with or without GPU You have the option to run with or without Docker containers. If you are not using Docker containers you will need CUDA, TensorFlow and all your Data Science libraries. See: https://community.hortonworks.com/articles/222242/running-apache-mxnet-deep-learning-on-yarn-31-hdp.html Tips from Wangda Basically GPU on YARN give you isolation of GPU device. Let's say a Node with 4 GPUS. First task comes ask 1 GPU. (Yarn.io/gpu=1). And YARN NM gives the task GPU0. Then the second task comes, ask 2 GPUs. And YARN NM gives the task GPU1/GPU2. So from TF perspective, you don't need to specify which GPUs to use. TF will automatically detect and consume whatever available to the job. For this case, task2 cannot see other GPUs apart from GPU1/GPU2. If you wish to run Apache MXNet deep learning programs, see this article: https://community.hortonworks.com/articles/222242/running-apache-mxnet-deep-learning-on-yarn-31-hdp.html Installation Install CU

Troubleshooting Oozie job is a pain! It kills your time and patience 🙂 .Here are few steps which can save your valuable time:.1. Always check Oozie launcher's stderr section to see if there is any error.Please find an useful article here to see how to check Oozie launcher logs..2. Check stdout logs to see if Oozie has launched any child job which has some error and because of which launcher got failed.Expand the stdout section and search for string "Submitted application" to see what all child jobs got triggered by launcher..3. Few situations are complex to troubleshoot. Child job gets completed successfully. There is no error in the stderr section and still your launcher gets failed with "Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]" error..Sample stdout logs:2016-12-06 09:03:39,986 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 100% reduce 0%2016-12-06 09:03:39,991 INFO [main] mapreduce.Job (Job.java:monitorAndP

Cloudbreak 2.7.2 maintenance release is now available. If you are using an earlier version of Cloudbreak, you can upgrade now to pick up the latest bug fixes.If you are new to Cloudbreak, you can get started by launching Cloudbreak on AWS, Azure, GCP, or OpenStack from a template.Useful links:Release notesUpgrade stepsGet started

This has been tested on Ambari 2.6.2.0 and DLM 1.1.2.0If there is broken symlink or unwanted directory under /var/lib/ambari-server/resources on Ambari Server, you get below error while installing mpack(management pack) for Beacon service.[root@XXXXXX ~]# ambari-server install-mpack --mpack /root/beacon-ambari-mpack-1.1.2.0-37.tar.gz --verboseUsing python /usr/bin/python Installing management pack INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Installing management pack /root/beacon-ambari-mpack-1.1.2.0-37.tar.gz INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Download management pack to temp location /var/lib/ambari-server/data/tmp/beacon-ambari-mpack-1.1.2.0-37.tar.gz INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO: Expand management pack at temp location /var/lib/ambari-server/data/tmp/beacon-ambari-mpack-1.1.2.0-37/ INFO: Loading properties from /etc/ambari-server/conf/ambari.properties INFO:

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'loginIdentityProvider': FactoryBean threw exception on object creation; nested exception is java.lang.Exception: The specified login identity provider 'kerberos-provider' could not be found.CloudBreak 2.7 built HDF 3.1 has error: 'kerberos-provider' could not be found.The reason is HDF 3.1's blueprint has problem:nifi.security.user.login.identity.provider= is set to kerberos-provider but that provider doesn't exist in login-identity-providers.xml, or is commented out

Usually service can be removed using API calls, but if the service is inconsistent state then API's does not work.so only way to delete is by running SQL queries. here is the list of steps to delete KNOX service.1. delete from serviceconfigmapping where service_config_id in (select service_config_id from serviceconfig where service_name like '%KNOX%') 2. delete from confgroupclusterconfigmapping where config_type like '%knox%'3. delete from clusterconfig where type_name like '%knox%'4. delete from clusterconfigmapping where type_name like '%knox%' 5. delete from serviceconfig where service_name = 'KNOX' 6. delete from servicedesiredstate where service_name = 'KNOX' 7. delete from hostcomponentdesiredstate where service_name = 'KNOX' 8. delete from hostcomponentstate where service_name = 'KNOX' 9.delete from servicecomponentdesiredstate where service_name = 'KNOX' 10.delete from clusterservices where service_name = 'KNOX' 11. DELETE from alert_history where alert_definition_id i

This article will demonstrate how to rapidly launch a Spark cluster on AWS via CloudBreak.The prerequisites are documented here. Once you have a AWS account and credentials, launching a Spark cluster is simple.CloudBreak is your command and control center UI for rapidly launching clusters on AWS, Az\ure, GCP, and on prem. Once the UI is up, add your AWS credentials Select AWS as your cloud provider Select the method for authentication. Key or Role. I prefer role but both work well. Click on the help button and follow the directions on how to setup auth for either method. Now that credentials have been setup, cluster creation may begin. Click on "Clusters" on top left and then click on "Create Cluster" on top right Select Advanced on top leftSelect Credential: Your AWS CredentialsCluster Name: Name your clusterRegion: AWS RegionPlatform Version: HDP 3.0Cluster Type: To run data science and ETL workloads, select HDP 3.0 Data Science blueprintClick Next Choose Image Type: Select Base Ima

Using cloudbreak to launch HDF (specifically NiFi), an error may prevent launching an instance. In the logs (/usr/hdf/current/nifi/logs/nifi-app.log) the following error is produced:pache.nifi.processors.hive.PutHiveStreaming could not be instantiatedjava.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.hive.PutHiveStreaming could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:232) at java.util.ServiceLoader.access$100(ServiceLoader.java:185) .... Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:239)....This is generally caused by by snappy library extracting to /tmp (generally defined in java.io.tmpdir).To fix this, do one of the following Grant nifi user access to /tmp Create a tmp directory where nifi user has access. For example, create tmp directory /home/tmp Then go into ambari and find

Zookeeper is one of the most critical components in an HDP cluster, but it is also one that is given least importance usually when tuning cluster for performance and while troubleshooting slowness in a cluster. Here is a basic checklist for zookeeper health check that one must go through to ensure that Zookeeper is running fine.Let's keep the zookeeper happy to be able to better manage the occupants of the zoo 🙂 1. Are all the Zookeeper servers given dedicated disks for transaction log directory ('dataDir' / 'dataLogDir') ?It is very important to have fast disks to complete 'fsync' of new transactions to the log, where zookeeper writes before any update takes place and before sending a response back to the client. Slower 'fsync' for transaction log is one of the most common reasons seen in the past for slower zookeeper response. Yes, the disk space requirement is usually not very high by the zookeeper and one might wonder if its worth to dedicate a complete disk to zookeeper log dire

I have been struggling to configure Spark through Mongodb with SSL. Unfortunately, the steps are not well documented there. Here is some quick guidance: Command line ( spark-shell / spark-submit / pyspark ) 1) According with the JIRA below: https://jira.mongodb.org/browse/SPARK-115?jql=project%20%3D%20SPARK%20AND%20component%20%3D%20Documentation Copy both Spark Connector and Mongo-Java-Driver among your datanodes, ( Nodes where your spark executors / driver are supposed to be running ). The Spark Connector uses the Mongo-Java-Driver and the driver will need to be configured to work with SSL. See the ssl tutorial in the java documentation. Ex. - spark_mongo-spark-connector_2.11-2.1.0.jar - mongodb_mongo-java-driver-3.4.2.jar OBS: Find yours at the mongodb website. 2) Go to ambari > Spark > Custom spark-defaults, now pass these two parameters in order to make spark (executors/driver) aware about the certificates. Example from my lab: spark.driver.extraJavaOptions=-Djavax.net.ssl.t

Starting with HDP 3.0 release, Hortonworks has stopped CentOS 6 as a supported platform. This would mean existing cluster nodes needs to be migrated to CentOS 7 before install/upgrade to HDP 3.0. However, CentOS has no official support for upgrading OS from CentOS 6 to CentOS 7. This article provides an unofficial guide for migrating cluster nodes to CentOS 7 without losing data. This is downtime migration. We would strongly recommend trying these steps on test clusters. These steps are applicable only for a smaller cluster ( <= 10 nodes ) and larger nodes we would need different strategy + automation for the same.Below are some of the assumptions or pre-requisite to be performed before upgrading the OS.Pre-RequisiteVisit all service configuration for e.x. HDFS, YARN etc make sure none of the configurations are pointing to non-root disk. If the services are pointing to root disk then upgrading the OS will format the root disk and we will lose data. Please make sure to migrate all co