Community Articles

Some times though moving master wizard is completed still UI shows the message as "Move Master Wizard In Progress".To fix this problem you can follow below steps.Step-1)curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST -d '{"wizard-data":"{\"userName\":\"admin\",\"controllerName\":\"reassignMasterController\"}"}'http://ambari.example.com:8080/api/v1/persistStep-2). Login to the Ambari UI with "admin" user.Step-3). Create a new "local" user with some name like "admin1" (any password like "admin1")Step-4). Open another browser and login with the "admin1" user. User might see that the "Add Service Wizard in Progress" link is blinking at the top.Step-5). Now from the earlier browser where we logged in with "admin" user. We will need to click on the "Actions" => "Add Service"Step-6). Now if the "admin" user does nothing. Or Shuts down his/her laptop. Still the other admin users (like "admin1") will not be able to perform any action in the ambari UI and will keep seeing the "Add

During upgrade, if Namenode restarts timeout, it may not appear to be a problem as the request times out from the Ambari UI but the restart process continue to run in the background.However, this can lead to inconsistencies in Ambari database and cause further issues at Finalize upgrade step.Note: This article is only useful upto Ambari-2.5.x version and must be performed before starting the upgrade process.Ambari-2.6 onwards it is a one step change where you only need to update the ambari.properties file instead of all the xml changes listed below.For Ambari-2.6 onwards: refer this document https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-upgrade/content/preparing_to_upgrade_ambari_and_hdp.htmlCaution: The steps described below are a hack! It is not recommended that we go about making changes to upgrade XML files based on our needs. Please exercise caution and measure your risks before following the steps.You can increase the timeouts for namenode restart using the f

Using GluonCV 0.3 with Apache MXNet 1.3 source code: https://github.com/tspannhw/nifi-gluoncv-yolo3 *Captured and Processed Image Available for Viewing in Stream in Apache NiFi 1.7.xuse case:I need to easily monitor the contents of my security vault. It is a fixed number of known things. What we need in the real world is a nice camera(s) (maybe four to eight depending on angles of the room), a device like an NVidia Jetson TX2, MiniFi 0.5 Java Agent, JDK 8, Apache MXNet, GluonCV, Lots of Python Libraries, a network connection and a simple workflow. Outside of my vault, I will need a server(s) or clusters to do the more advanced processing, though I could run it all on the local box. If the number of items or certain items I am watching are no longer in the screen, then we should send an immediate alert. That could be to an SMS, Email, Slack, Alert System or other means. We had most of that implemented below. If anyone wants to do the complete use case I can assist.demo i

This article is for those who want a cheat sheet for a smooth installation of HDP in a Dev, or Test with one or more of the following requirements: Place all the log data into a different directory, not /var/log All your service user names must be prefixed with the cluster name. The requirement is that these users must be centrally managed by AD or an LDAP. You do not have any local users in the Hadoop cluster, including Hadoop service users. This becomes important if you wish to have Centrify deployed also, or if you would be deploying multiple clusters with a single LDAP/ AD integration. Once again, these service names should have a cluster-prefix. You want to set appropriate YARN, Tez and MapReduce, Amabri Metrics Memory Parameters during Install. Side Note: It is always prudent to get Professional Service assistance to either install or configure your production deployment, to make sure all the per-requisites, unique to your environment are covered and met. ----------------------

One quirk of Apache Phoenix when compared to traditional RDBMS is that Phoenix provides no notion of simple username/password based authentication. This largely stems from Apache HBase, which Phoenix is built on, also not providing this as a form of authentication. With the introduction of the Phoenix Query Server, we have a number of new means which can be used to interact with Phoenix. We also have the ability to hook together new systems to provide features, like username/password authentication, which are not traditionally supported. There are multiple products available which can perform this kind of authentication, but we can trivially show that this works via a common HTTP load balancer, HAProxy. Let's assume that we have the Phoenix Query Server running on our local machine listening on the standard 8765 port. We can enable some trivial authentication using HAProxy. First, we need to create our HAProxy configuration file. global maxconn 256defaults mode http option redisp

The quickest way to launch a cluster with Cloudbreak is through the CLI. This allows you to specify all of the required settings and submit the request to Cloudbreak. The cluster will show up in Cloudbreak just like it does when launched via the wizard. The quickest way to generate the JSON template is to set up a cluster through the wizard.Generating the JSON Template from Cloudbreak GUIThe first way you can create the template is from the Create Cluster wizard. In the Cloudbreak GUI, walk through all of the steps to configure a cluster. At the end, before you click "Create Cluster" you can click "Show CLI Command". The second way is to go the a running cluster in the Cloudbreak GUI. Click the "Actions" button and you can click "Show CLI Command".The CLI command dialog comes up with the JSON template in the top text box and the Cloudbreak CLI command in the lower text box. Click the "Copy the JSON" button and the paste it into a text file. Copy the CLI command to a note for la

This article explains how to delete a Registered HDP cluster from DPSSteps:1. Download the attached file dp-cluster-remove.txt and store it in postgres db server. 2. psql -f ./dp_cluster_remove.txt -d dataplane -v cluster_name=c149-d --> <dataplane database name>cluster_name --> Your HDP clustername that should be removed from DPS cluster.Example Output:-[root@rraman bin]# docker exec -it dp-database /bin/bash bash-4.3# su - postgres de0ff40ad912:~$ ls -lrt total 8 drwx------ 19 postgres postgres 4096 Sep 21 01:27 data -rwxrwxrwx 1 postgres postgres 892 Sep 21 17:59 dp_cluster_remove.txtde0ff40ad912:~$ psql -f ./dp_cluster_remove.txt -d dataplane -v cluster_name=c149 CREATE FUNCTION remove_cluster_from_dps ------------------------- (1 row) DROP FUNCTION

Running Apache MXNet Deep Learning on YARN 3.1 - HDP 3.0 With Hadoop 3.1 / HDP 3.0, we can easily run distributed classification, training and other deep learning jobs. I am using Apache MXNet with Python. You can also do TensorFlow or Pytorch. If you need GPU resources, you can specify them as such: yarn.io/gpu=2 My cluster does not have an NVidia GPU unfortunately. See: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/data-operating-system/content/dosg_recommendations_for_running_docker_containers_on_yarn.html Running App on YARN [root@princeton0 ApacheDeepLearning101]# ./yarn.sh18/09/21 15:31:22 INFO distributedshell.Client: Initializing Client18/09/21 15:31:22 INFO distributedshell.Client: Running Client18/09/21 15:31:22 INFO client.RMProxy: Connecting to ResourceManager at princeton0.field.hortonworks.com/172.26.208.140:805018/09/21 15:31:23 INFO client.AHSProxy: Connecting to Application History server at princeton0.field.hortonworks.com/172.26.208.140:1020018

There can be situation where you are adding a Host in Ambari and the Add Host Wizard is stuck in UI with message :Please wait while the hosts are being checked for potential problemsIt will be tougher for you to proceed as next button will be disabled and you migt need to wait indefinitely for the Host to respond.To analyze what happened.We might need to look on ambari-agent hosts and ambari-server logs.As usually this Host checking will take longer we can actually skip the same using the Ambari experimental Wizards Disabling HostCheck On AddHost WizardSteps1) navigate in another Tab to URL :

Cloudbreak 2.8.0 Technical Preview release is now available!New featuresCloudbreak 2.8.0 TP introduces the following new features:FeatureDescriptionDocumentationAWS GovCloudCloudbreak supports installing Cloudbreak and creating Cloudbreak-managed clusters on AWS GovCloud. Deploying on AWS vs AWS GovCloudEBS volume encryption on AWSYou can optionally configure encryption for EBS volumes attached to cluster instances running on EC2. Default or customer-managed encryption keys can be used.EBS encryption on AWSGCP volume encryptionBy default, Compute Engine encrypts data at rest stored on disks. You can optionally configure encryption for the encryption keys used for disk encryption. Customer-supplied (CSEK) or customer-managed (CMEK) encryption keys can be used.Disk encryption on GCPUser authorizationCloudbreak introduces a new authorization model, which allows resource sharing via organizations.User authorizationOperations audit loggingCloudbreak records an audit trail of the actions per

Hi All, In this article we will talk about how to setup a separate zookeeper quorum for Kafka which is managed by Ambari. Please note at this moment Amabri only can support one ZK Quorum, in order to have two Quorum in a cluster (1) for Kafka dedicatedly (2) for all other Hadoop services Only supported option by HWX is to setup another setoff Zookeeper server (3 or 5) and refer those ZK servers as quorum in Amabari managed Kafka Prerequisite: - All Servers should have java Installed Download the zk tar files:- http://www-eu.apache.org/dist/zookeeper/stable/ Password less ssh with all the host servers. Steps (to setup Zookeeper Server) :- Please run all the below command in all the nodes where zookeeper will be installed manually Create hadoop group and zookeeper ID groupadd hadoop useradd zookeeper -g hadoop 2. Setting up the environment variables (ease of deployment) export ZOOKEEPER_CONF_DIR=/etc/zookeeper/conf export ZOOKEEPER_LOG_DIR=/var/log/zookeeper ex

If cluster is installed with Ambari 2.5 , Upgrading to Ambari2.6.x cause service re-start ( client) failure and Add service also fails with installation.with above combination, you might be hitting https://issues.apache.org/jira/browse/AMBARI-24629 if you see error like below 2018-09-15 08:37:13,748 - Looking for matching packages in the following repositories: HDP-2.6, HDP-UTILS-1.1.0.21 2018-09-15 08:37:15,841 - No package found for hive_${stack_version}(hive_(\d|_)+$) 2018-09-15 08:37:15,844 - The repository with version 2.6.1.0-129 for this command has been marked as resolved. It will be used to report the version of the component which was installedroot cause is already explained in the BUG.How to fix the issue:1. First find out what is your current version repo version id. in browser go to http://apappu3.hdp.com:8080/api/v1/stacks/HDP/versions/2.6/repository_versions?fields=operating_systems/repositories/Repositories/base_url From the response find out the repository_version_id f

This article will (hopefully) be helpful if you are trying to use LLAP/Hive Interactive, but it fails or gets stuck during startup. This is often caused by misconfiguration, esp. w.r.t. sizing, but could also be a bug. Ambari performs 2 major activities when (re)starting Hive Interactive - first it starts LLAP itself on the YARN cluster, then it starts HiveServer2 for Hive Interactive. Usually the problems happen during LLAP startup.This article will help you to pinpoint the problem if there's some bug or non-sizing-related misconfiguration. It will also point at the sizing document where appropriate; the sizing and setup document can be found at https://community.hortonworks.com/articles/149486/llap-sizing-and-setup.html. For general information (e.g. where the logs are) see the parent article.Note: for all the logs, it is sometimes the case that the last error in the log is InterruptedException or some other similar error during component shutdown after something else has already fai

Many times Ambari throws installation errors like "No package found for hive_${stack_version}(hive_(\d|_)+$) ..." There can be many reasons for this failure.The root cause for this issue is, basically Ambari could not be able to determine the installable package.Reason1: Check if the same package is already installed using different repoid/sateliterepotag. that can be check using below commandyum list installed | grep packagenameEx: yum list installed | grep hiveIf it shows the entries then determine if those are installed with different repoid/sateliterepotag.In Ambari task logs you can find what tag/repo Ambari is using to installNow see "yum list installed" has the correct repoId - if it is not matching then manually uninstall that library and let Ambari install it.Reason2: If you are using Satelite repository - Ambari should show the repo tag name at "Applicable repositories: Satelitehdp265.."If it is not showing correctly, that means you repo version definition is incorrect. Make

Although as of Cloudbreak 2.8.0 there is no way to manage cluster templates in the Cloudbreak web UI, you can generate a reusable template from the UI and then reuse it for creating clusters via Cloudbreak CLI. In this blog post, I describe the steps for how to do this.Step 1: In the web UI, provide cluster parameters and generate a cluster template1.In the Cloudbreak web UI, navigate to Clusters and then click on Create cluster.2.To obtain a full list of cluster options, click on Advanced. 3.Provide all the parameters for your cluster.4.Once done, on the last page of the create cluster wizard, click on Show CLI Command. 5.Click on Copy the JSON to copy the generated CLI skeleton.6.Paste it into a text editor and save the file onto your computer in .JSON format.7.Open the file and review it to ensure that no parameters are missing. Save the file once you are done editing.Step 2: Create clusters via CLI from the template 1.Download and configure Cloudbreak CLI. Refer to Install CLI and

Thanks to Christoph Gutsfeld, Matthias von Görbitz and Rene Pajta for all their valuable pointers for writing this article. The article provides a indetailed and thorugh understanding of Hive LLAP. Understanding YARN YARN is essentially a system for managing distributed applications. Itconsists of a central Resource manager, which arbitrates all available cluster resources, and a per-node Node Manager, which takes direction from the Resource manager. Resource Manager and node Manager follow a master slave relationship. The Node manager is responsible for managing available resources on a single node. Yarn defines a unit of work in terms of container.. It is available in each node. Application Master negotiates container with the scheduler(one of the component of Resource Manager). Containers are launched by Node ManagerUnderstanding YARN Memory configuration Memory allocated for all YARN container on a node : Total amount of memory that can be used by Node manager on every node for

Overview The whole purpose of that list is to be able to search through all the entities contained within your data lake and identify specifically those entities which will lead your analysis. A Common complaint with the idea of the data lake is that you load too many files into the data lake and it becomes unwieldy. Atlas addresses this problem by providing powerful search tools to identify all of the Data entities located within the data lake. The purpose of this article is to explore the four Entity search techniques available within Atlas.Atlas search options compared The following chart summarizes the capabilities for each of the four atlas entity search techniques explored in this article. The sub sections below in this article we’ll go into more detail on how to set up the search techniques. Attribute Entity Search Qualified Name Search DSL Search Full Text Search Can identify multiple entities Yes No Yes Yes Supports filtering No Yes Yes

While Upgrading/Downgrading HDP in Ambari, if you see "Unable to determine the stack and stack version" it is possible the repo version has some inconsistency.Post Ambari2.6.x, should have repo_version table "resolved" column as 1 - If it shows "0" then run below update query.update repo_version set resolved = 1Re-start Ambari and then attempt the operation again - it should go through.Note: As of now it is not a BUG as we could not able to reproduce the issue.

In this episode we welcome Phil Radley, Chief Data Architect at BT to talk about the Big Data deployment at BT. https://roaringelephant.org/2018/09/11/episode-105-big-data-at-british-telecom-with-phillip-radley/ Play in new window | Download (Duration: 1:06:32 — 45.9MB)Phillip Radley (Linkedin)Chief Data Architect @ BThttps://home.bt.com/ Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

In this Big Data News episode, we discuss an article with guidelines on how you should arrange your data gathering projects with the customer in mind. Dave brings a matrix of visualization products.https://roaringelephant.org/2018/09/04/episode-104-roaring-news/ Play in new window | Download (Duration: 36:55 — 25.6MB)The five Cs: Five framing guidelines to help you think about building data products. https://www.oreilly.com/ideas/the-five-cs?utm_medium=social&utm_source=twitter.com&utm_campaign=awareness&utm_content=radar+contentThe Chartmaker Directory http://chartmaker.visualisingdata.com/ Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

This article assumes that, you have a HDP-2.5 cluster with Atlas & Hive enabled. Also make sure that atlas is up and running on that cluster. Please refer to this documentation link for deploying cluster with Atlas enabled.Atlas provides a script/tool to import metadata from hive for all hive entities like tables, database, views, columns etc... This tool/script requires hadoop and hive classpath jars, to make them available to the scriptMake sure that the environment variable HADOOP_CLASSPATH is set OR HADOOP_HOME to point to root directory of your Hadoop installation.set HIVE_HOME env variable to the root of Hive installation.Set env variable HIVE_CONF_DIR to Hive configuration directory.Copy <atlas-conf>/atlas-application.properties to the hive conf directory Once the above steps are successfully completed, now we are ready to run the script.Usage: <atlas package>/hook-bin/import-hive.shWhen you run the above command, you should see the below messages over console.[r

# Max Timer Driven Thread Count and Max Event Driven Thread Count:-Out of the box NiFi sets the Max Timer Thread Counts relatively low to support operating on the simplest of hardware. This default setting can limit the performance of very large high volume dataflow that must perform a lot of concurrent processing.General guidance for setting this value is 2 - 4 times the number of cores available to the hardware on which the NiFi service is running. With a NiFi cluster where each server has different hardware (not recommended), this would be set to the highest value possible based on the server with the fewest cores.NOTE: Remember that all configurations you apply within the NIFi UI are applied to every node in a NiFi cluster. None of the settings apply as a total to the cluster itself.NOTE: The cluster UI can be used to see how the total active threads are being used per node.Closely monitoring system CPU usage over time on each of your cluster nodes will help you identify regular

# Processor Run Duration:Some processors support configuring a run duration. This setting tells a processor to continue to use the same task to work on as many FlowFiles (or batches of flowfiles) from an incoming queue in a single task. This is ideal for processors where the individual tasks themselves are completed very fast and the volume of FlowFile are large as well.In the above example, the exact same feed of FlowFiles were passed to both these processors which are configured to perform the same Attribute updated.Both processed the same number of FlowFiles in the past 5 minutes; however, the processor configured with a run duration consumed less overall CPU time to do so.Not all processors support setting a run duration. The nature of the processor function, the methods being used, and/or client lib used may not support this capability. You will not be able to set a run duration on such processors.How this works:Processor has thread assigned to its task. Processor grabs highest pr

Introduction Cloudbreak uses SaltStack to manage nodes of the cluster, install packages, change configuration files and execute recipes. Motivation Cloudbreak gives the opportunity to the users to manage their virtual machines. When user SSH into a machine and updates/installs packages using system package manager, SaltStack can fail with version incompatibility issues. This scenario led us to create a solution which ensures SaltStack’s correct behavior independently of the user actions with system package manager. SaltStack in virtualenv Since version 2.8, Cloudbreak provides official images to every cloud provider which contains Saltstack in a separated virtual environment. This prevents the mentioned version incompatibility. The virtual machines created by the official images creates a separated environment for SaltStack and installs it using the Python package manager of that environment. Virtualenv Virtualenv is widely used tool which can used to create different Python e

Why do we need a Load Balancer for NiFi cluster? The easiest way to start using NiFi is deploying it as a standalone NiFi instance. However, when you need more throughput, NiFi can form a cluster to distribute load among multiple NiFi nodes. From 1.0, thanks to the Zero Master Clustering architecture, we can access NiFi Web UI via any node in a cluster. Although it’s more reliable, it can be unclear which endpoint to specify for things like Remote Process Group url, clients for processors like ListenTCP or ListenSyslog acting as servers. NiFi Site-to-Site protocol is cluster-topology aware and automatically distributes load among nodes, but users may wonder which hostname to specify. You can point to any node in the cluster, but what if the node goes down? If ListenXXX processors runs in a cluster, we could configure it with On Primary Node scheduler to make it simply runs only on a single node (we used to specify a primary node). But it doesn’t scale, and will be a SPOF. If ever