Community Articles

Objective In a separate HCC article, I detailed how to offload nodes using the NiFi UI. This article covers how to perform the same operation using the NiFi Toolkit CLI. Note: This tutorial assumes you have already setup a running NiFi cluster. The examples included are for a 2 node NiFi cluster.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.8.0 Apace NiFi Toolkit 1.8.0Offload Nodes via CLI TutorialQueue FlowFiles First, generate some queued flowfiles: The queued flowfiles are distributed between both nodes in the cluster: Disconnect Node From a terminal window, navigate to the directory where the NiFi Toolkit was installed. From the /nifi-toolkit-1.8.0/bin directory, run: ./cli.sh Enter the command nifi get-nodes to retrieve the node IDs: #> nifi get-nodes# Node ID Node Address API Port Node Status- ------------------------------------ ------------ --------

Deep learning offers the opportunity to improve clinical outcomes in medicine, enabling healthcare providers to serve growing patient populations, and providing care givers with the tools they need to focus on patients with critical conditions. The applications for the use of deep learning in medicine extend across all areas where visual diagnosis is used in patient care. This includes radiology (arteriography, mammography, radiomics), dermatology, and oncology, as well as clinical and biotech R&D. In this series, we will explore these applications, including the staging of data, training of deep learning models, evaluating models, the use of distributed compute, and the operationalization of our models. In this article, we will take a look using deep learning to classify Melanoma. Specifically, we will use images of skin, to classify whether the patient associated with the image, has a malignant tumor. The diagram below shows a classification model based on Stanford M

If MySQL Driver does not exist by default, or not installed when MySQL is installed, then the provisioned cluster has the following errors. In this case, create a Recipe in Cloudbreak to install MySQL jdbc driver at pre-ambari-start time:During Create Cluster, use Advance option. Pick the recipe pk-install-mysql-driver-pre from the drop down and click Attach. This will ensure that the jdbc driver is installed before Ambari is started. This should then result in a clean install.

If you see errors related to YARN Registry DNS Bind Port. Where the YARN Registry DNS is stopped, most probably due to a port conflict. Then go into the Advanced Configurations. Look for the parameter RegistryDNS Bind Port and change is to a port that does not have a conflict. I changed it from 53 to 553.Save configuration changes. Restart all components that are impacted. This should take care of the issue and YARN Registry DNS will start successfully.

OBJECTIVE: Delete the cluster added in the Dataplane using utility script provided in the product. OVERVIEW:DPS-1.1 doesn't support the deletion of added clusters from the UI. If the users want to delete the clusters added in the Dataplane , an utility script is provided which can be used for deleting the HDP Clusters added. PREREQUISITES: Install jq : jq is a lightweight and flexible command-line JSON processor. STEPS: By default, "rm_dp_cluster.sh" script will be located at /usr/dp/current/core/bin. Execute the rm_dp_cluster.sh script by providing the following parameters DP_JWT : Value of dp_jwt cookie from a valid users browser sessionnHADOOP_JWT : Value of hadoop-jwt cookie from a valid users browser sessionDP_HOST_NAME : Hostname or IP address of the DataPlane serverCLUSTER_NAME: Ambari Cluster Name of the cluster to deleteDATA_CENTER_NAME : Name of the cluster datacenter to delete For e.g:Format:./rm_dp_cluster.sh <DP_JWT> <HADOOP_JWT> <DP_HOST_NAME> <CLUST

Q. What is Delegated Admin Privilege? Instead of having a single central authority creating security policies for the entire Hadoop resources, Apache Ranger provides a feature, “Delegated-Admin” to delegate the security policy management to other resource administrators. For example, the permissions associated with hdfs:/app/finance folder and all its contents can be entirely managed finance-admin group by granting “delegated-admin” permission to finance-admin group on “hdfs:/app/finance with recursive=true” In simple-terms, Delegated-Admin permission allows other resource administrators to manage permissions for their managed-resources. Q. What are the roles assigned to a user within ranger-ui? In Apache Ranger, an user can be set with one of the following role: ADMIN, USER, KEYADMIN. When a user is assigned with KEYADMIN role, he/she will have the ability to manage all ranger-kms policies and kms-audit information within the ranger-admin UI. Also, he/she can provide

Problem: How to enable LDAP for users to view Activity Explorer Dashboard. We may have shiro.ini enabled for LDAP for Zepplin but it is different for Activity Explorer Dashboard.Solution: On SmartSense Activity Explorer node, under [main] sectioncd /etc/smartsense-activity/conf/vi shiro.ini[main]sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManagersecurityManager.sessionManager = $sessionManagersecurityManager.sessionManager.globalSessionTimeout = 86400000shiro.loginUrl = /api/loginldapRealm = org.apache.zeppelin.realm.LdapRealmldapRealm.contextFactory.systemUsername = <fill your details>ldapRealm.contextFactory.systemPassword = <fill your details>ldapRealm.contextFactory.authenticationMechanism = SIMPLEldapRealm.contextFactory.url = <fill your details>ldapRealm.pagingSize = 200ldapRealm.authorizationEnabled = trueldapRealm.searchBase = <fill your details>ldapRealm.memberAttributeValueTemplate = <fill your details>ldapRealm.userSearch

General Recommendations - Only fail tuples if there's a recoverable exception - Use an Exception/Error Logging Bolt/Stream and send the output to a Kafka topic so all logs are consolidated in place - Be careful about supervisor saturation / overallocation i.e. running more threads than available CPU resources - Configure parallelism based on throughput requirements (use Throughput Equation to calculate) - Be aware of data spikes and plan for them in your topology / use cases / operations - If using Kafka, setup Kafka Producer-Consumer Lag analysis and dashboard - Benchmark topologies to assess "health" operations numbers for individual bolt performance ## Troubleshooting topology performance issues: - Check the topology stats from the Nimbus UI - Look for Failures (in spout or specific bolt) if any that module is having issues. - If a topology has Kafka error bolt then open a console consumer to looks at the error logs from Kafka - Now find out what supervisor the worker is running on

When Oozie launcher(map only mapreduce job) gets scheduled to run on RHEL7 node in a mixed OS environment, it may get failed with below ERROR(stderr section of Oozie launcher logs):Container: container_e1XX_XXXXXXX_0X_00000X on XXXXXX_XXX_XXXXXXLogAggregationType: AGGREGATED===============================================================================================================LogType:stderrLog Upload Time:Tue XXXXXXXXXXXXLogLength:XXLog Contents:SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".SLF4J: Defaulting to no-operation (NOP) logger implementationSLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details./usr/bin/env: bash: No such file or directoryFailing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.impl.MetricsSystemImpl).log4j:WARN Please initialize the log4j system properly.log4j:WARN See http://logging.apache.org/l

Apache Metron ExplainedApache Metron is a cyber security application framework that provides organizations the ability to ingest, process and store diverse security data feeds at scale in order to detect cyber anomalies and enable organizations to rapidly respond to them.As the diagram above indicates, the Metron framework provides 4 key capabilities:Security Data Lake / Vault - Platform provides cost effective way to store enriched telemetry data for long periods of time. This data lake provides the corpus of data required to do feature engineering that powers discovery analytics and provides a mechanism to search and query for operational analytics.Pluggable Framework - Platform provides not only a rich set of parsers for common security data sources (pcap, netflow, bro, snort, fireye, sourcefire) but also provides a pluggable framework to add new custom parsers for new data sources, add new enrichment services to provide more contextual info to the raw streaming data, pluggable exte

This is an article which describes how to access AMS embedded Hbase in a secured kerberized environment. This is extension of this document For a unsecured cluster, please read the above document however in case of secured environment you may need run one extra export command. The final command will be similar to below syntax<sqline.py path> <ams_collector_hostname>:<embedded_zk_port>:<znode>/usr/hdp/2.5.0.0-1245/phoenix/bin/sqlline.py rchaman.ambari.apache.org:61181:/ams-hbase-unsecurePlease find the below steps to get the parameters as per your environment. 1. We need to make sure that we have the phoenix client packages installed on the host from where you are running command. however, I suggest you to run the command from AMS collector host. If it is not installed then we can install it as following:yum install phoenix -y2. Search for sqlline.py script absolute path[root@c4140-node4 ~]# rpm -qa |grep phoenixphoenix_2_6_4_0_91-4.7.0.2.6.4.0-91.noarch[root

Simple Apache NiFi Operations Dashboard - Part 2Part 1:     https://community.cloudera.com/t5/Community-Articles/Building-a-Custom-Apache-NiFi-Operations-Dashboard-Part-1/ta-p/249060To access data to display in our dashboard we will use some Spring Boot 2.06 Java 8 microservices to call Apache Hive 3.1.0 tables in HDP 3.0 on Hadoop 3.1.We will have our web site hosted and make REST Calls to Apache NiFi, our microservices, YARN and other APIs.As you can see we can easily incorporate data from HDP 3 - Apache Hive 3.1.0 in Spring Boot java applications with not much trouble. You can see the Maven build script (all code is in github.)Our motivation is put all this data somewhere and show it in a dashboard that can use REST APIs for data access and updates. We may choose to use Apache NiFi for all REST APIs or we can do some in Apache NiFi. We are still exploring. We can also decide to change the backend to HBase 2.0, Phoenix or Druid or a combination. We will see. Sprin

Simple Apache NiFi Operations Dashboard This is an evolving work in progress, please get involved everything is open source. @milind pandit and I are working on a project to build something useful for teams to analyze their flows, current cluster state, start and stop flows and have a rich one look dashboard. There's a lot of data provided by Apache NiFi and related tools to aggregate, sort, categorize, search and eventually do machine learning analytics on. There are a lot of tools that come out of the box that solve parts of these problems. Ambari Metrics, Grafana and Log Search provide a ton of data and analysis abilities. You can find all your errors easily in Log Search and see nice graphs of what is going on in Ambari Metrics and Grafana. What is cool with Apache NiFi is that is has SitetoSite tasks for sending all the provenance, analytics, metrics and operational data you need to wherever you want it. That includes to Apache NiFi! This is Monitoring Driven Development (MDD). M

The following example provides a guide to connecting to HBase from Spark then perform a few operations using Spark on the data extracted from HBasePlease add the following repo to your package managerhttp://repo.hortonworks.com/content/repositories/releases/and import the following classcom.hortonworks:shc-core:1.1.0-2.1-s_2.11Keep in mind that new releases may be availableImport the following classesimport org.apache.spark.sql.{SQLContext, _}import org.apache.spark.sql.execution.datasources.hbase._import org.apache.spark.{SparkConf, SparkContext}import spark.sqlContext.implicits._import org.apache.spark.sql.types._The first step in connecting to HBase is defining how Spark should interpret each row of data found in HBase. This is done by defining a catalog to use with the connectordef catalog = s"""{ |"table":{"namespace":"default", "name":"timeseries"}, |"rowkey":"key", |"columns":{ |"col0":{"cf":"rowkey", "col":"key", "type":"string"}, |"col1":{"cf":"cf", "col":"t", "type":"strin

Restarting NiFi nodes fails with an error message such as the one below: <br>File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 278, in Master().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute method(env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 993, in restart self.start(env, upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 152, in start nifi_cli.create_or_update_reg_client(params.nifi_registry_host, params.nifi_registry_url) File "/u/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_cli.py", line 175, in create_or_update_reg_client existing_clients = list_reg_clients() File "/u/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_cli.py", line 144, in list_reg_clients outputType="json" File

This article goes on how Nifi, with the of the OPC UA api along with Druid for dashboarding can help with industrial iot use cases. We have seen a trend where several clients have shown interest in HDP as a platform for storing all of the sensor data and third party data. This allows these companies store their sensor data over a longer period of time, easily corelate them with other data sets and enrich them in a very cost effective manner. Additionally, they can run ML algorithms on this data set to better predict outages, malfunctions or maintenance. Using NiFi,Kafka and Druid, can provide means for a very easy to maintain and superfast ingestion mechanism from the several edge devices be it windmills, substations, meters etc. Druid can then provide a robust platform for real-time querying dashboarding on this sensor data so operations can keep an eagle eye on whats going on at a plant or wind farm.For the pupose of this article, i will be consuming data from a demo OPC server which

Prerequisites 1)Download HDP Sandbox 2)MySQL database (Should already be present in the sandbox)3)Nifi 0.6 or later ( Download and install a new version of NIFI or use Ambari to install NIFI in the sandbox)MySQL setup (Source Database)In this setup we will create a table in MySQL tables and create a few triggers on the tables to emulate transactions. These triggers will find out if the change introduced was an insert or an update also will update the time stamp on the updated/inserted row. ( This is very important as Nifi Will be polling on this column to extract changes based on the time stamp)unix> mysql –u root –punix>Enterpassword:<enter>mysql>mysql> create databasetest_cdc;mysql> create user'test_cdc'@'localhost' identified by 'test_cdc';mysql> GRANT ALLPRIVILEGES ON *.* TO 'test_CDC'@'%' IDENTIFIED BY 'test_CDC' WITH GRANT OPTION;mysql>Flush Privilegesmysql> exit;unix> mysql –u test_cdc –ptest_cdcmysql>create table CDC_TEST(Column_A int, Colu

Adding HBase HA to a blueprint with high availability is a very straightforward process. The blueprint you start from has to include a zookeeper quorum, like the blueprint that provides HA namenodes and resourcemanagers that is discussed in "Cloudbreak blueprint for high-availability Hadoop and Hive". We will add HBase Master services to two master host groups, HBase Regionservers to the datanode host group, and an hbase-site configuration section.Required Zookeeper Quorum To properly specify the zookeepers to be used, you will need at least three separate host groups with cardinality of 1 to provide the zookeepers. In the blueprint provided in the previous article, those hostgroups are: master_mgmt, master_nn1 and master_nn2. In that blueprint, master_mgmt contains the Ambari, metrics and other management services, while master_nn1 and master_nn2 contain the redundant namenodes and resourcemanagers. In this example, we will add redundant HBase masters to those host groups as well

Dr. ElephantDr. Elephant is a performance monitoring and tuning tool for Hadoop and Spark. It automatically gathers a job's metrics, analyzes them, and presents them in a simple way for easy consumption. Its goal is to improve developer productivity and increase cluster efficiency by making it easier to tune the jobs. It analyzes the Hadoop and Spark jobs using a set of pluggable, configurable, rule-based heuristics that provide insights on how a job performed and then uses the results to make suggestions about how to tune the job to make it perform more efficiently. It also computes a number of metrics for a job which provides valuable information about the job performance on the cluster.Build Stepscd ~;sudo yum update -y;sudo yum upgrade -y;sudo curl --silent --location https://rpm.nodesource.com/setup_8.x | sudo bash -sudo yum install -y nodejs wget unzip git zip gcc-c++ make java-1.8.0-openjdk-devel;wget https://github.com/linkedin/dr-elephant/archive/v2.1.7.tar.gz;wget https://dow

While accessing Ambari 2.5 or 2.4 Hive Views (1.5.0/2.0) we see the following error in the log:ERROR [ambari-client-thread-62] ContainerResponse:537 - Mapped exception to response: 500 (Internal Server Error) org.apache.ambari.view.hive2.utils.ServiceFormattedException at org.apache.ambari.view.hive2.client.NonPersistentCursor.getNextRows(NonPersistentCursor.java:132) at org.apache.ambari.view.hive2.client.NonPersistentCursor.fetchIfNeeded(NonPersistentCursor.java:119) at org.apache.ambari.view.hive2.client.NonPersistentCursor.getDescriptions(NonPersistentCursor.java:84) at org.apache.ambari.view.hive2.resources.jobs.ResultsPaginationController.request(ResultsPaginationController.java:145) at org.apache.ambari.view.hive2.resources.jobs.JobService.getResults(JobService.java:361) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.refl

Example topology for kerberos auth and hive:[root@groot1 hive]# cat /etc/knox/2.6.0.3-8/0/topologies/kerberos.xml <topology> <gateway> <provider> <role>authentication</role> <name>HadoopAuth</name> <enabled>true</enabled> <param> <name>config.prefix</name> <value>hadoop.auth.config</value> </param> <param> <name>hadoop.auth.config.signature.secret</name> <value>hadoop12345!</value> </param> <param> <name>hadoop.auth.config.type</name> <value>kerberos</value> </param> <param> <name>hadoop.auth.config.simple.anonymous.allowed</name> <value>false</value>

Overview Last article of the series, and a fun one. As usual, this article is the continuation of part 1, part 2 and part 3, so make sure you read those before starting. The completion of this tutorial will finalize your end to end architecture, as depicted here: [ADD IMAGE] This tutorial will be divided in 2 sections: Section 1: Create a Web Service with Nifi Section 2: Create a consumer application with code Igniter Note: Section 2 will not be detailed. First, because web application development is not the main goal of Hortonworks Community Connection. Second, because I am pretty sure I did a terrible job at it considering it is not my domain of expertise, and I shouldn't spread mediocrity to the world!Section 1: Create a Web Service with Nifi For your entire project to work, you will have to create more than one web service although this section only goes through the creation of one (see the flow overview below). To get all the flows required for your app to work, check githu

Overview Last article of the series, and a fun one. As usual, this article is the continuation of part 1, part 2 and part 3, so make sure you read those before starting. The completion of this tutorial will finalize your end to end architecture, as depicted here: This tutorial will be divided in 2 sections: Section 1: Create a Web Service with Nifi Section 2: Create a consumer application with code Igniter Note: Section 2 will not be detailed. First, because web application development is not the main goal of Hortonworks Community Connection. Second, because I am pretty sure I did a terrible job at it considering it is not my domain of expertise, and I shouldn't spread mediocrity to the world! Section 1: Create a Web Service with Nifi For your entire project to work, you will have to create more than one web service although this section only goes through the creation of one (see the flow overview below). To get all the flows required for your app to work, check gi

Running TensorFlow on YARN 3.1 with or without GPU You have the option to run with or without Docker containers. If you are not using Docker containers you will need CUDA, TensorFlow and all your Data Science libraries. See: https://community.hortonworks.com/articles/222242/running-apache-mxnet-deep-learning-on-yarn-31-hdp.html Tips from Wangda Basically GPU on YARN give you isolation of GPU device. Let's say a Node with 4 GPUS. First task comes ask 1 GPU. (Yarn.io/gpu=1). And YARN NM gives the task GPU0. Then the second task comes, ask 2 GPUs. And YARN NM gives the task GPU1/GPU2. So from TF perspective, you don't need to specify which GPUs to use. TF will automatically detect and consume whatever available to the job. For this case, task2 cannot see other GPUs apart from GPU1/GPU2. If you wish to run Apache MXNet deep learning programs, see this article: https://community.hortonworks.com/articles/222242/running-apache-mxnet-deep-learning-on-yarn-31-hdp.html Installation Install CU

Troubleshooting Oozie job is a pain! It kills your time and patience 🙂 .Here are few steps which can save your valuable time:.1. Always check Oozie launcher's stderr section to see if there is any error.Please find an useful article here to see how to check Oozie launcher logs..2. Check stdout logs to see if Oozie has launched any child job which has some error and because of which launcher got failed.Expand the stdout section and search for string "Submitted application" to see what all child jobs got triggered by launcher..3. Few situations are complex to troubleshoot. Child job gets completed successfully. There is no error in the stderr section and still your launcher gets failed with "Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]" error..Sample stdout logs:2016-12-06 09:03:39,986 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 100% reduce 0%2016-12-06 09:03:39,991 INFO [main] mapreduce.Job (Job.java:monitorAndP