Resources

Running Apache MXNet Deep Learning on YARN 3.1 - HDP 3.0 With Hadoop 3.1 / HDP 3.0, we can easily run distributed classification, training and other deep learning jobs. I am using Apache MXNet with Python. You can also do TensorFlow or Pytorch. If you need GPU resources, you can specify them as such: yarn.io/gpu=2 My cluster does not have an NVidia GPU unfortunately. See: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/data-operating-system/content/dosg_recommendations_for_running_docker_containers_on_yarn.html Running App on YARN [root@princeton0 ApacheDeepLearning101]# ./yarn.sh18/09/21 15:31:22 INFO distributedshell.Client: Initializing Client18/09/21 15:31:22 INFO distributedshell.Client: Running Client18/09/21 15:31:22 INFO client.RMProxy: Connecting to ResourceManager at princeton0.field.hortonworks.com/172.26.208.140:805018/09/21 15:31:23 INFO client.AHSProxy: Connecting to Application History server at princeton0.field.hortonworks.com/172.26.208.140:1020018

There can be situation where you are adding a Host in Ambari and the Add Host Wizard is stuck in UI with message :Please wait while the hosts are being checked for potential problemsIt will be tougher for you to proceed as next button will be disabled and you migt need to wait indefinitely for the Host to respond.To analyze what happened.We might need to look on ambari-agent hosts and ambari-server logs.As usually this Host checking will take longer we can actually skip the same using the Ambari experimental Wizards Disabling HostCheck On AddHost WizardSteps1) navigate in another Tab to URL :

Cloudbreak 2.8.0 Technical Preview release is now available!New featuresCloudbreak 2.8.0 TP introduces the following new features:FeatureDescriptionDocumentationAWS GovCloudCloudbreak supports installing Cloudbreak and creating Cloudbreak-managed clusters on AWS GovCloud. Deploying on AWS vs AWS GovCloudEBS volume encryption on AWSYou can optionally configure encryption for EBS volumes attached to cluster instances running on EC2. Default or customer-managed encryption keys can be used.EBS encryption on AWSGCP volume encryptionBy default, Compute Engine encrypts data at rest stored on disks. You can optionally configure encryption for the encryption keys used for disk encryption. Customer-supplied (CSEK) or customer-managed (CMEK) encryption keys can be used.Disk encryption on GCPUser authorizationCloudbreak introduces a new authorization model, which allows resource sharing via organizations.User authorizationOperations audit loggingCloudbreak records an audit trail of the actions per

Hi All, In this article we will talk about how to setup a separate zookeeper quorum for Kafka which is managed by Ambari. Please note at this moment Amabri only can support one ZK Quorum, in order to have two Quorum in a cluster (1) for Kafka dedicatedly (2) for all other Hadoop services Only supported option by HWX is to setup another setoff Zookeeper server (3 or 5) and refer those ZK servers as quorum in Amabari managed Kafka Prerequisite: - All Servers should have java Installed Download the zk tar files:- http://www-eu.apache.org/dist/zookeeper/stable/ Password less ssh with all the host servers. Steps (to setup Zookeeper Server) :- Please run all the below command in all the nodes where zookeeper will be installed manually Create hadoop group and zookeeper ID groupadd hadoop useradd zookeeper -g hadoop 2. Setting up the environment variables (ease of deployment) export ZOOKEEPER_CONF_DIR=/etc/zookeeper/conf export ZOOKEEPER_LOG_DIR=/var/log/zookeeper ex

Hi,I am a new memberPleased to interact with youThank you,Quda

If cluster is installed with Ambari 2.5 , Upgrading to Ambari2.6.x cause service re-start ( client) failure and Add service also fails with installation.with above combination, you might be hitting https://issues.apache.org/jira/browse/AMBARI-24629 if you see error like below 2018-09-15 08:37:13,748 - Looking for matching packages in the following repositories: HDP-2.6, HDP-UTILS-1.1.0.21 2018-09-15 08:37:15,841 - No package found for hive_${stack_version}(hive_(\d|_)+$) 2018-09-15 08:37:15,844 - The repository with version 2.6.1.0-129 for this command has been marked as resolved. It will be used to report the version of the component which was installedroot cause is already explained in the BUG.How to fix the issue:1. First find out what is your current version repo version id. in browser go to http://apappu3.hdp.com:8080/api/v1/stacks/HDP/versions/2.6/repository_versions?fields=operating_systems/repositories/Repositories/base_url From the response find out the repository_version_id f

This article will (hopefully) be helpful if you are trying to use LLAP/Hive Interactive, but it fails or gets stuck during startup. This is often caused by misconfiguration, esp. w.r.t. sizing, but could also be a bug. Ambari performs 2 major activities when (re)starting Hive Interactive - first it starts LLAP itself on the YARN cluster, then it starts HiveServer2 for Hive Interactive. Usually the problems happen during LLAP startup.This article will help you to pinpoint the problem if there's some bug or non-sizing-related misconfiguration. It will also point at the sizing document where appropriate; the sizing and setup document can be found at https://community.hortonworks.com/articles/149486/llap-sizing-and-setup.html. For general information (e.g. where the logs are) see the parent article.Note: for all the logs, it is sometimes the case that the last error in the log is InterruptedException or some other similar error during component shutdown after something else has already fai

Many times Ambari throws installation errors like "No package found for hive_${stack_version}(hive_(\d|_)+$) ..." There can be many reasons for this failure.The root cause for this issue is, basically Ambari could not be able to determine the installable package.Reason1: Check if the same package is already installed using different repoid/sateliterepotag. that can be check using below commandyum list installed | grep packagenameEx: yum list installed | grep hiveIf it shows the entries then determine if those are installed with different repoid/sateliterepotag.In Ambari task logs you can find what tag/repo Ambari is using to installNow see "yum list installed" has the correct repoId - if it is not matching then manually uninstall that library and let Ambari install it.Reason2: If you are using Satelite repository - Ambari should show the repo tag name at "Applicable repositories: Satelitehdp265.."If it is not showing correctly, that means you repo version definition is incorrect. Make

Although as of Cloudbreak 2.8.0 there is no way to manage cluster templates in the Cloudbreak web UI, you can generate a reusable template from the UI and then reuse it for creating clusters via Cloudbreak CLI. In this blog post, I describe the steps for how to do this.Step 1: In the web UI, provide cluster parameters and generate a cluster template1.In the Cloudbreak web UI, navigate to Clusters and then click on Create cluster.2.To obtain a full list of cluster options, click on Advanced. 3.Provide all the parameters for your cluster.4.Once done, on the last page of the create cluster wizard, click on Show CLI Command. 5.Click on Copy the JSON to copy the generated CLI skeleton.6.Paste it into a text editor and save the file onto your computer in .JSON format.7.Open the file and review it to ensure that no parameters are missing. Save the file once you are done editing.Step 2: Create clusters via CLI from the template 1.Download and configure Cloudbreak CLI. Refer to Install CLI and

Thanks to Christoph Gutsfeld, Matthias von Görbitz and Rene Pajta for all their valuable pointers for writing this article. The article provides a indetailed and thorugh understanding of Hive LLAP. Understanding YARN YARN is essentially a system for managing distributed applications. Itconsists of a central Resource manager, which arbitrates all available cluster resources, and a per-node Node Manager, which takes direction from the Resource manager. Resource Manager and node Manager follow a master slave relationship. The Node manager is responsible for managing available resources on a single node. Yarn defines a unit of work in terms of container.. It is available in each node. Application Master negotiates container with the scheduler(one of the component of Resource Manager). Containers are launched by Node ManagerUnderstanding YARN Memory configuration Memory allocated for all YARN container on a node : Total amount of memory that can be used by Node manager on every node for

Overview The whole purpose of that list is to be able to search through all the entities contained within your data lake and identify specifically those entities which will lead your analysis. A Common complaint with the idea of the data lake is that you load too many files into the data lake and it becomes unwieldy. Atlas addresses this problem by providing powerful search tools to identify all of the Data entities located within the data lake. The purpose of this article is to explore the four Entity search techniques available within Atlas.Atlas search options compared The following chart summarizes the capabilities for each of the four atlas entity search techniques explored in this article. The sub sections below in this article we’ll go into more detail on how to set up the search techniques. Attribute Entity Search Qualified Name Search DSL Search Full Text Search Can identify multiple entities Yes No Yes Yes Supports filtering No Yes Yes

While Upgrading/Downgrading HDP in Ambari, if you see "Unable to determine the stack and stack version" it is possible the repo version has some inconsistency.Post Ambari2.6.x, should have repo_version table "resolved" column as 1 - If it shows "0" then run below update query.update repo_version set resolved = 1Re-start Ambari and then attempt the operation again - it should go through.Note: As of now it is not a BUG as we could not able to reproduce the issue.

In this episode we welcome Phil Radley, Chief Data Architect at BT to talk about the Big Data deployment at BT. https://roaringelephant.org/2018/09/11/episode-105-big-data-at-british-telecom-with-phillip-radley/ Play in new window | Download (Duration: 1:06:32 — 45.9MB)Phillip Radley (Linkedin)Chief Data Architect @ BThttps://home.bt.com/ Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

In this Big Data News episode, we discuss an article with guidelines on how you should arrange your data gathering projects with the customer in mind. Dave brings a matrix of visualization products.https://roaringelephant.org/2018/09/04/episode-104-roaring-news/ Play in new window | Download (Duration: 36:55 — 25.6MB)The five Cs: Five framing guidelines to help you think about building data products. https://www.oreilly.com/ideas/the-five-cs?utm_medium=social&utm_source=twitter.com&utm_campaign=awareness&utm_content=radar+contentThe Chartmaker Directory http://chartmaker.visualisingdata.com/ Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

This article assumes that, you have a HDP-2.5 cluster with Atlas & Hive enabled. Also make sure that atlas is up and running on that cluster. Please refer to this documentation link for deploying cluster with Atlas enabled.Atlas provides a script/tool to import metadata from hive for all hive entities like tables, database, views, columns etc... This tool/script requires hadoop and hive classpath jars, to make them available to the scriptMake sure that the environment variable HADOOP_CLASSPATH is set OR HADOOP_HOME to point to root directory of your Hadoop installation.set HIVE_HOME env variable to the root of Hive installation.Set env variable HIVE_CONF_DIR to Hive configuration directory.Copy <atlas-conf>/atlas-application.properties to the hive conf directory Once the above steps are successfully completed, now we are ready to run the script.Usage: <atlas package>/hook-bin/import-hive.shWhen you run the above command, you should see the below messages over console.[r

# Max Timer Driven Thread Count and Max Event Driven Thread Count:-Out of the box NiFi sets the Max Timer Thread Counts relatively low to support operating on the simplest of hardware. This default setting can limit the performance of very large high volume dataflow that must perform a lot of concurrent processing.General guidance for setting this value is 2 - 4 times the number of cores available to the hardware on which the NiFi service is running. With a NiFi cluster where each server has different hardware (not recommended), this would be set to the highest value possible based on the server with the fewest cores.NOTE: Remember that all configurations you apply within the NIFi UI are applied to every node in a NiFi cluster. None of the settings apply as a total to the cluster itself.NOTE: The cluster UI can be used to see how the total active threads are being used per node.Closely monitoring system CPU usage over time on each of your cluster nodes will help you identify regular

# Processor Run Duration:Some processors support configuring a run duration. This setting tells a processor to continue to use the same task to work on as many FlowFiles (or batches of flowfiles) from an incoming queue in a single task. This is ideal for processors where the individual tasks themselves are completed very fast and the volume of FlowFile are large as well.In the above example, the exact same feed of FlowFiles were passed to both these processors which are configured to perform the same Attribute updated.Both processed the same number of FlowFiles in the past 5 minutes; however, the processor configured with a run duration consumed less overall CPU time to do so.Not all processors support setting a run duration. The nature of the processor function, the methods being used, and/or client lib used may not support this capability. You will not be able to set a run duration on such processors.How this works:Processor has thread assigned to its task. Processor grabs highest pr

Introduction Cloudbreak uses SaltStack to manage nodes of the cluster, install packages, change configuration files and execute recipes. Motivation Cloudbreak gives the opportunity to the users to manage their virtual machines. When user SSH into a machine and updates/installs packages using system package manager, SaltStack can fail with version incompatibility issues. This scenario led us to create a solution which ensures SaltStack’s correct behavior independently of the user actions with system package manager. SaltStack in virtualenv Since version 2.8, Cloudbreak provides official images to every cloud provider which contains Saltstack in a separated virtual environment. This prevents the mentioned version incompatibility. The virtual machines created by the official images creates a separated environment for SaltStack and installs it using the Python package manager of that environment. Virtualenv Virtualenv is widely used tool which can used to create different Python e

Why do we need a Load Balancer for NiFi cluster? The easiest way to start using NiFi is deploying it as a standalone NiFi instance. However, when you need more throughput, NiFi can form a cluster to distribute load among multiple NiFi nodes. From 1.0, thanks to the Zero Master Clustering architecture, we can access NiFi Web UI via any node in a cluster. Although it’s more reliable, it can be unclear which endpoint to specify for things like Remote Process Group url, clients for processors like ListenTCP or ListenSyslog acting as servers. NiFi Site-to-Site protocol is cluster-topology aware and automatically distributes load among nodes, but users may wonder which hostname to specify. You can point to any node in the cluster, but what if the node goes down? If ListenXXX processors runs in a cluster, we could configure it with On Primary Node scheduler to make it simply runs only on a single node (we used to specify a primary node). But it doesn’t scale, and will be a SPOF. If ever

SYMPTOM: user is frequently seeing following exceptions: org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept. org.apache.kafka.common.errors.RecordTooLargeException: The message is 5745799 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration. ROOT CAUSE: This is happening due to max byte settings at producer level and broker level Broker side: message.max.bytes - this is the largest size of the message that can be received by the broker from a producer Producer side: max.request.size is a limit to send the larger message. WORKAROUND: NA RESOLUTION: message.max.bytes by default is 1M (1000012 bytes) for kafka 0.10.0. if you need to publish larger messages, you will need to adjust that on the brokers and then restart them.to avoid exception at producer side you need to increase max.request.size to send the larger message. P

Failed to list function from phoenix sqlline.pyCommand run : select * from SYSTEM.FUNCTION;Error: ERROR 604 (42P00): Syntax error. Mismatched input. Expecting "NAME", got "FUNCTION" at line 1, column 22. (state=42P00,code=604)org.apache.phoenix.exception.PhoenixParserException: ERROR 604 (42P00): Syntax error. Mismatched input. Expecting "NAME", got "FUNCTION" at line 1, column 22.at org.apache.phoenix.exception.PhoenixParserException.newException(PhoenixParserException.java:33)at org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:111)at org.apache.phoenix.jdbc.PhoenixStatement$PhoenixStatementParser.parseStatement(PhoenixStatement.java:1280)at org.apache.phoenix.jdbc.PhoenixStatement.parseStatement(PhoenixStatement.java:1363)at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1434)at sqlline.Commands.execute(Commands.java:822)at sqlline.Commands.sql(Commands.java:732)at sqlline.SqlLine.dispatch(SqlLine.java:808)at sqlline.SqlLine.begin(SqlLine.java:

This article gives an 8 step procedure to recover the accidentally deleted HDFS file which is not even available in the Trash. Use these procedure with caution on production system. I strongly suggest to take supports help if you are not familiar with the internal workings.IMPORTANT :Please make sure NameNode is stopped immediatly after file deletion otherwise it's hard to recover as namenode already send out block deletion request to datanode causing physical block might get deleted by datanodeAlways take backup of "fsimage" and "edits" file before you perform this step.1).Lets create a sample file in HDFSsu - hdfshadoop fs -put /etc/passwd /tmp[hdfs@hdpnode1 current]$ hadoop fs -ls /tmp/passwd -rw-r--r-- 3 hdfs hdfs 2370 2016-04-06 12:45 /tmp/passwd2).Delete the file and also make sure that the file doesnt go to trash.hadoop fs -rmr -skipTrash /tmp/passwd3).Stop the HDFS service via Ambari.4). Go to Namenode metadata directory(dfs.namenode.name.dir).In my lab environment it is set to

OBJECTIVE:Resolve issues with some lightweight LDAP services such as the HDP Demo LDAP ProviderOVERVIEW:Some LDAP services do not properly support paging for LDAP query results. In order to support these LDAP services, paging of results needs to be disabled in the LDAP provider properties for the NiFi Registry serviceSYMPTOM:The NiFi Registry does not respond or times out, and the following error is seen repeatedly in the NiFi Registry log (nifi-registry-app.log):2018-09-01 01:01:05,189 INFO [main] o.s.l.c.AbstractRequestControlDirContextProcessor No matching response control found - looking for 'class javax.naming.ldap.PagedResultsResponseControl2018-09-01 01:01:05,274 INFO [main] o.s.l.c.AbstractRequestControlDirContextProcessor No matching response control found - looking for 'class javax.naming.ldap.PagedResultsResponseControl2018-09-01 01:01:05,359 INFO [main] o.s.l.c.AbstractRequestControlDirContextProcessor No matching response control found - looking for 'class javax.naming.lda

IoT Edge Processing with Apache NiFi and MiniFi and Multiple Deep Learning Libraries SeriesFor: https://conferences.oreilly.com/strata/strata-ny/public/schedule/detail/68140See Part 1: https://community.hortonworks.com/articles/215079/iot-edge-processing-with-deep-learning-on-hdf-32-a.htmlSee Part 2: https://community.hortonworks.com/articles/215258/iot-edge-processing-with-deep-learning-on-hdf-32-a-1.htmlSee Part 3: https://community.hortonworks.com/articles/215271/iot-edge-processing-with-deep-learning-on-hdf-32-a-2.htmlYou will notice a bit of a travel theme this article, it's because some of the images and work was done while on various holidays in August and September.Deep LearningWe are running TensorFlow 1.10, Apache MXNet 1.3, NCSDK 2.05 and Neural Compute Application Zoo (NC App Zoo).Device Type 1: Plain Raspberry Pi(Found some old Kodak slides...)Main things to do is upgrade to Python 3.6, upgrade Raspberry PI to Stretch, upgrade libraries and a few reboots.Install OpenCV (

IntroductionWhen the NiFi bootstrap starts or stops NiFi, or detects that it has died unexpectedly, it is able to notify configured recipients. Currently, the only mechanism supplied is to send an e-mail notification.Prerequisite1) Assuming you already have HDF-2.x Installed, Ambari and NiFi is up and running. If not, I would recommend "Ease of Deployment" section of this article to install it [You can also follow this article for Automated installation of HDF cluster or refer hortonworks.com for detailed steps]Configuring NiFi property files in Ambari1) To setup email notifications we have to update only two configurations file bootstrap.conf and bootstrap-notification-services.xml 2) We have to update appropriate properties in Ambari to configure it, first we have to edit Template for bootstrap.conf to update below properties. Uncomment below lines in the properties file:nifi.start.notification.services=email-notification nifi.stop.notification.services=email-notificationnifi.dead.n