Resources

1. WASB --> https://blogs.msdn.microsoft.com/cindygross/2015/02/04/understanding-wasb-and-hadoop-storage-in-azure/WASB is a storage model which allows storage of data in Blobs within storage accounts/ containers in Azure cloud. 2. DASH --> http://sequenceiq.com/cloudbreak-deployer/latest/azure_pre_prov/ --> This link describes a few scale-out related limits of WASB, and proposes DASH as the solution. DASH is not supported as a storage option, and there are scalability limitations on the number of storage accounts. To quote, "When WASB is used as a Hadoop filesystem the files are full-value blobs in a storage account. It means better performance compared to the data disks and the WASB filesystem can be configured very easily but Azure storage accounts have their own limitations as well. There is a space limitation for TB per storage account (500 TB) as well but the real bottleneck is the total request rate that is only 20000 IOPS where Azure will start to throw errors when

Prior to Version 1.6.2-rc.8 version of Cloudbreak, provisioning of HDP Clusters on Azure was by Public IP only. This article describes provisioning of a cluster based on Private IP. While creating a Network within Cloudbreak, there are options to 1. Create a New Network and a new subnet2. Use an existing subnet in an existing virtual networkUnder the Option 2 "Use an existing subnet in an existing virtual network" There are 2 options added , a) Don't create public IPsb) Don't create new firewall rulesUpon selecting these options ( and using Private IP "export PUBLIC_IP=xx.xx.xx.xx" in the Profile file for Cloudbreak, ), Cloudbreak can install clusters using Private IP only in Azure cloud. A screen shot of this feature is attached here --> networkoptions.png

A usual query that comes up is on using snappy and other compression codes when loading data from hdfs using the nifi puthdfs component. A common error that users come across is "java.lang.UnsatisfiedLinkError". This error occurs because snappy and other compression codecs, which are part of the native linux binaries and which the java libraries for these codecs utilize to do the actual compression, are not in the path of the JVM. Since the JVM cannot find them the bindings fails and you get a java.lang.UnsatisfiedLinkError.Follow the following steps to resolve this issue.1. copy the native folder containing the compression libraries, from one of your hadoop nodes. cd /usr/hdp/x.x.x.x-xx/hadoop/lib/tar -cf ~/native.tar native/2. scp the native.tar from your hadoop node to your NiFi node, untar to a location of your choice. in my case i use /home/myuser/hadoop/ cd ~ mkdir hadoop cd hadoop tar -cf /path/to/native.tar3. go to your nifi folder , and open conf/bootstr

To upgrade JDK across all HDP services on all nodes using Ambari, the following instructions have proven to be helpful. This requires downtime on ALL HDP components.a. Use the 'Stop all' from the Actions button off the main Ambari dashboard to shut down all HDP components. b. Using Clush/Pdsh/Cluster scripting tool of choice, confirm that ALL Java processes are fully stopped across ALL cluster nodes (excluding Ambari), killing any orphaned processes that Ambari was unable to stop. c. Run the following from Ambari master node (as root) to stop the Ambari-agent and Ambari-server processes. service ambari-agent stop service ambari-server stop d. Run the following from Ambari-server master node (as root): ambari-server setup (and follow the prompts, example provided below) e.g.: [root@sandbox conf]# ambari-server setup Using python /usr/bin/python2.6 Setup ambari-server Checking SELinux... SELinux status is 'disabled' Customize user account for ambari-server daemon [y/n] (n)? n Adjusting a

Ambari REST APIs to add a component in kerberized cluster. I tried and tested these in Ambari 2.4.1 versionHere is an example to install SECONDARY_NAMENODE1. API to Add a Componentcurl -s -H "X-Requested-By:ambari" --user admin:admin -i -X POST -d '{"host_components":[{"HostRoles":{"component_name":"SECONDARY_NAMENODE"}}]}' http://AMBARIHOST:8080/api/v1/clusters/CLUSTERNAME/hosts?Hosts/host_name=COMPONENTHOST2. API to install the Above added component.curl -s -H "X-Requested-By:ambari" --user admin:admin -i -X PUT -d '{"RequestInfo":{"context":"Install SECONDARY_NAMENODE","operation_level":{"level":"HOST_COMPONENT","cluster_name":"test","host_name":"COMPONENTHOST","service_name":"HDFS"}},"Body":{"HostRoles":{"state":"INSTALLED"}}}' http://AMBARIHOST:8080/api/v1/clusters/CLUSTERNAME/hosts/COMPONENTHOST/host_components/SECONDARY_NAMENODE3. API to provide kerberos credentials.curl -s -H "X-Requested-By:ambari" --user admin:admin -i -X POST -d '{ "Credential" : { "principal" : "admin/admi

Status, Icons, and Color:The Apache Nifi 1.0 release brought its users a welcome refresh of the UI with emphasis on a flatter more modern design that included muting the color palette. While these updates have been well received the community around Apache Nifi requested that we "bring some color back". Well we listened and in Apache Nifi 1.1 we have introduced more color in the UI for easier recognition of status changes and necessary user actions!One area (identified by the community) where color makes sense are the icons. In many cases in the Nifi UI icons provide valuable feedback worthy of drawing the attention from the user. However not all icons meet this requirement. To differentiate it was decided that wherever icons are reporting status or require the user to take an action then the icon should receive color and if an icon invokes an action in the UI then it should not.Let's take a look at the canvas:At first glance you will notice the red bulletins and yellow warning icons "

If you wish to reference a file in S3 from a pig script you might do something like this:set fs.s3n.awsSecretAccessKey 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxx';set fs.s3n.awsAccessKeyId 'xxxxxxxxxxxxxxxxxxxxx';A = load 's3n://<bucket>/<path-to-file>' USING TextLoader;If you're on HDP 2.2.6, you'll likely see this error:Error: java.io.Exception, no filesystem for scheme: s3nThe following steps resolve this issue:In core-site.xml add:<property><name>fs.s3n.impl</name><value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value><description>The FileSystem for s3n: (Native S3) uris.</description></property>Then add to the MR2 and/or TEZ class path(s):/usr/hdp/${hdp.version}/hadoop-mapreduce/*These configs ensure 2 things:That the worker YARN containers spawned by pig have access to the hadoop-aws.jar fileThat the worker YARN containers know which class implements the file system type identified by "s3n://"References: Apache Mailing Lis

1. Install MySQL data Directories on a Non-root partition (Not /var/lib) 2. Create a dedicated least privileged account for Mysql deamon3. Disable MySQL Command History, Command History may contain passwords which is viewable by other users4. Disable interactive Login5. Disable login from nodes other than the those used by hive services6. Provide only hive user permission to the Hive metadata database within MySQL7. During installation do not specifify passwords in command line8. Ensure Data Directories for Mysql has appropriate permissions and ownerships9. Ensure only DBA administrators have full database access10. Ensure that database logging is enabled for error logs and log files are maintained on a non system partition11. Ensure that old passwords is not set to 1 12. Ensure secure_auth is set to ON13. Consider if your Component can work with the MYSQL "Connect using SSL" option

Objective The objective of this tutorial is to walk you through process of updating your Cloudbreak Deployer on Amazon AWS from 1.3 to 1.4. Once this is complete, you can now deploy an HDP 2.5 TP cluster.   Prerequisites You should have deployed Cloudbreak 1.3 on Amazon AWS using the instructions found here: Cloudbreak Documentation 1.3 - AWS. You should add TCP port 443 to the security group on Amazon AWS as Cloudbreak 1.4+ appears to proxy requests from port 443 now. Do not run cbd start.   Note: This process will update the Cloudbreak Deployer to the latest available version. At the time of initial writing, this was 1.4. As of September 1, 2016, the latest version is 1.6. Steps 1. Connect to your Cloudbreak Amazon AWS instance You should have access to your key file from amazon. Log into your Cloudbreak deployer instance using: ssh -i <amazon key file> cloudbreak@<amazon instance public ip> Note: If you have permission issues connecting via ssh, make sure you

- One approach can be to check the Ambari Blueprint to see what changes are made by comparing last two results. Following approach can be integrated inside a shell script (cronjob) to poll the config changes at some regular interval and to find the "diff" between current and last output files "/tmp/cluster-`date +'%Y%m%d%H%M'`.txt"Example: curl -o "/tmp/cluster-`date +'%Y%m%d%H%M'`.txt" -u admin:admin -H "X-Requested-By: ambari" -X GET http://c6401.ambari.apache.org:8080/api/v1/clusters/ClusterDemo?format=blueprint .- Users can also check the output of below REST call to browse "service_config_versions"curl -u admin:admin -i -H 'X-Requested-By: ambari' -X GET http://c6401.ambari.apache.org:8080/api/v1/clusters/ClusterDemo/configurations/service_config_versions.Above would list the service config versions of all services. At any time users make the above call, it would provide the complete history of service config changes. For more information on this please refer to: https://cwiki.

Maintenance Mode supports suppressing alerts and skipping bulk operations for specific services, components and hosts in an Ambari-managed cluster.There are many scenarios when we may want to turn on Maintenance Mode specially when we are performing hardware or software maintenance, changing configuration settings, troubleshooting, decommissioning, or removing cluster nodes. In such scenario we can place a service, component, or host object in Maintenance Mode before we start to perform necessary maintenance or troubleshooting tasks. In maintenance mode following things happen: 1. It causes suppression of alerts, warnings and status change indicators generated for the object. 2. It exempts an object from host-level or service-level bulk operations. Although it can be done from the ambari UI, However here in this article we will see how to manage the maintenance mode for some Services like HBase using the APIs.Turning ON Maintenance Mode for any specific component (like: region server o

Many times it is needed to monitor the ambari memory usage. JConsole and JVisualVM are the JDK provided JMX client utilities that can be used for the same. Here in this demo we will see how to monitor ambari server's JVM memory using $JAVA_HOME/bin/jvisualvm (same steps will be applicable for jconsole as well)Step-1). We will need to add the "com.sun.management.jmxremote" related JVM remoted jmx options to the "/var/lib/ambari-server/ambari-env.sh" as following.export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=4444 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false'For more information on JVMs standard jmxremote options please refer to : http://docs.oracle.com/javase/7/docs/technotes/guides/management

Steps to configure ambari-server to archive log files.1. Open /etc/ambari-server/conf/log4j.properties file,log4j.appender.file=org.apache.log4j.RollingFileAppenderlog4j.appender.file.File=${ambari.log.dir}/${ambari.log.file}log4j.appender.file.MaxFileSize=80MBlog4j.appender.file.MaxBackupIndex=60log4j.appender.file.layout=org.apache.log4j.PatternLayoutlog4j.appender.file.layout.ConversionPattern=%d{DATE} %5p [%t] %c{1}:%L - %m%ntolog4j.appender.file=org.apache.log4j.rolling.RollingFileAppenderlog4j.appender.file.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy log4j.appender.file.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy log4j.appender.file.File=${ambari.log.dir}/${ambari.log.file}log4j.appender.file.triggeringPolicy.MaxFileSize=10485760log4j.appender.file.layout=org.apache.log4j.PatternLayoutlog4j.appender.file.layout.ConversionPattern=%d{DATE} %5p [%t] %c{1}:%L - %m%nlog4j.appender.file.rollingPolicy.FileNamePattern=${ambari.log.dir}/${amba

This has been observed for Ambari 1.7. I know Ambari 1.7 is very old now however if some people are still using it and facing same issue then this post can save your lot of time! 🙂.We know that Ambari 1.7 has Ganglia for reporting metrics..Our issue was, we were unable to get service metrics for YARN service, below was the exact scenario:1. In Ganglia UI, I was able to get graphs for the YARN Metrics.2. In Ambari UI, I was able to see metrics for other services like HDFS etc.3. This issue was observed on one of our customer's cluster.4. I did setup same cluster on my local system however I was not able to reproduce this issue.5. Only difference in customer's and my cluster was, customer was having RM HA and I had installed single node instance.6. There was no error while getting metrics, please see below screenshot when metrics were not visible..How to troubleshoot:1. Click on any of the metrics, say 'Nodemanagers' --> This will open graph in big window.2. Open the developer tools

Apache Knox provides a gateway to HiveServer2 which can be used to proxy connections to Hive from BI tools like Tableau Desktop. With a secure cluster HiveServer2 is kerberized, which usually means the client must have a valid TGT in their ticket cache in order to authenticate to HiveServer2.The Knox gateway can simplify this in many environments, by supporting client authentication to kerberized Hive without a TGT. The client authenticates to Knox via, say, LDAP authentication, and HiveServer2 trusts Knox to proxy connections on behalf of this user. Another benefit of using Knox is that client doesn't need to know where HiveServer2 is hosted, so if an administrator needs to move it to another master node, this is transparent to the applications connecting via Knox.In the environment for this walkthrough, I have Tableau Desktop 10.0.1 installed, and I'll be using HDP 2.5. The first step is to ensure you have the latest Hortonworks ODBC Driver for Apache Hive, 2.1.5 at the time of this

Sqoop OverviewSqoop is a tool designed to transfer data between Hadoop and relational databases or mainframes. You can use Sqoop to import data from a relational database management system (RDBMS) such as MySQL or Oracle or a mainframe into the Hadoop Distributed File System (HDFS), transform the data in Hadoop MapReduce, and then export the data back into an RDBMS.Sqoop automates most of this process, relying on the database to describe the schema for the data to be imported. Sqoop uses MapReduce to import and export the data, which provides parallel operation as well as fault tolerance.Sqoop Performance Tuning Best PracticesTune the following Sqoop arguments in JDBC connection or Sqoop mapping to optimize performance batch•split-by and boundary-query•direct•fetch-size•num-mapper•2.Inserting Data in BatchesSpecifies that you can group the related SQL statements into a batch when you export data. The JDBC interface exposes an API for doing batches in a prepared statement with multiple

Using SQOOP with MySQL as metastoreTo set up MySQL for use with SQOOP: On the SQOOP Server host, install the connector. Install the connector RHEL/CentOS/Oracle Linux yum install mysql-connector-java SLES zypper install mysql-connector-java Confirm that .jar is in the Java share directory. ls /usr/share/java/mysql-connector-java.jar Make sure the .jar file has the appropriate permissions - 644. Create a user for SQOOP and grant it permissions. For example, using the MySQL database admin utility: # mysql -u root -p CREATE USER '<SQOOPUSER>'@'%' IDENTIFIED BY '<SQOOPPASSWORD>'; GRANT ALL PRIVILEGES ON *.* TO '<SQOOPUSER>'@'%'; CREATE USER '<SQOOPUSER>'@'localhost' IDENTIFIED BY '<SQOOPPASSWORD>'; GRANT ALL PRIVILEGES ON *.* TO '<SQOOPUSER>'@'localhost'; CREATE USER '<SQOOPUSER>'@'<SQOOPSERVERFQDN>' IDENTIFIED BY '<SQOOPPASSWORD>'; GRANT ALL PRIVILEGES ON *.* TO '<SQOOPUSER>'@'<SQOOPSERVERFQDN>'; FLUSH PR

Steps to connect from a client machine (MAC in this case) to Hadoop cluster using hive JDBC. Here are couple of links that i have used to build this out Link that talks about hive drivers and jar files https://streever.atlassian.net/wiki/pages/viewpage.action?pageId=4390924 Link that talks about how to setup java jdbc setup for hive jar files https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients1.Here are the jar files you need to connect to HS2 For HDP 2.3 You'll only need 2 jar files for the JDBC Client with HDP 2.3 # From /usr/hdp/current/hive-client hive-jdbc.jar (should be a symlink to the hive-jdbc-xxx-standalone.jar) # From /usr/hdp/current/hadoop-client hadoop-common.jar (hadoop-common-....jar) 2. Make sure that java home is set on your machine This is value that i have on my machine for JAVA_HOME echo $JAVA_HOME /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home 3. Move these jar files to java library directory on my machine /Library/Java/Ex

IntroductionBy default, Zookeeper runs without the option of becoming a superuser to administrate znodes in the ZK ensemble, for example, to fix ACLs, remove znodes that are not required anymore, or create new ones in specific locations. Zookeeper grants permissions through ACLs through different schemas or authentication methods, such as 'world', 'digest', or 'sasl' if we use Kerberos. We can potentially we locked out if we were to grant everyone just read permissions to a znode, as we would not be able to delete it or modify it anymore.Example of the problemFor example, we connect to Zookeeper through zkCli:/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server `hostname -f`:2181[zk: sandbox.hortonworks.com:2181(CONNECTED) 1] getAcl /config/topics'world,'anyone: r If we need to modify that znode so that, for example, user 'kafka' can have access to it to create new topics:[zk: sandbox.hortonworks.com:2181(CONNECTED) 2] setAcl /config/topics world:anyone:r sasl:kafka:cdrwaAuthenticati

ObjectiveIf you are managing multiple copies of the HDP sandbox for Docker (see my article here:How to manage multiple copies of the HDP Docker Sandbox.), you may find yourself running out of storage within your Docker VM image on your laptop. There is a way to increase the available storage space of the Docker VM image. Increasing the storage space will allow you to have more copies of sandbox containers in addition to other images and containers.This tutorial will guide you through the process of increasing the size of the base Docker for Mac VM image. We will increase the size from 64GB to 120GB. This tutorial is the second in a two part series. The first tutorial in the series is: How to move Docker for Mac vm image from internal to external hard drive.For a tutorial on increasing the base Docker VM image on CentOS 7, read my tutorial here: How to modify the default Docker configuration on CentOS 7 to import HDP sandboxPrerequisitesYou should have already completed the following tu

In previous post we have seen how to Automate HDP installation with Namenode HA using Ambari Blueprints. In this post, we will see how to Deploy single node HDP Cluster with Kerberos authentication via Ambari blueprint.Note - For Ambari 2.6.X onwards, we will have to register VDF to register internal repository, or else Ambari will pick up latest version of HDP and use the public repos. please see below document for more information. For Ambari version less than 2.6.X, this guide will work without any modifications.Document - https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-release-notes/content/ambari_relnotes-2.6.0.0-behavioral-changes.html.Below are simple steps to install HDP single node cluster with Kerberos Authentication(MIT KDC) using internal repository via Ambari Blueprints..Step 1: Install Ambari server using steps mentioned under below linkhttp://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_Installing_HDP_AMB/content/_download_the_ambari_repo_lnx6.h

SYMPTOM: While performing step for Namenode HA, on the step of 'hdfs namenode -initializeSharedEdits' it failed with below error - ERROR: [root@localhost conf]# sudo su hdfs -l -c 'hdfs namenode -initializeSharedEdits'16/11/22 09:43:56 INFO namenode.NameNode: STARTUP_MSG:/************************************************************STARTUP_MSG: Starting NameNodeSTARTUP_MSG: host = locahost/10.200.206.12STARTUP_MSG: args = [-initializeSharedEdits]STARTUP_MSG: version = 2.7.1.2.3.0.0-2557STARTUP_MSG: classpath = /usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/commons-cli-1.2.jar:/us r/hdp/2.3.0.0-2557/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.3.0.0-2557/hadoop/lib/jsch-0.1.42.jar:/usr/hdp/2.3.0.0-2557/ ...STARTUP_MSG: build = git@github.com:hortonworks/hadoop.git -r 9f17d40a0f2046d217b2bff90ad6e2fc7e41f5e1; compiled by 'jenkins' on 2015-07-14T13:08ZSTARTUP_MSG: java = 1.8.0_51************************************************************/16/

To post random data to Azure Event hubs using HTTP, use a generate flowfile processor to first generate the random data. Connect this to an UpdateAttribute processor to add your SAS (Share Access Signature) as a new attribute called "Authorization" to the flow file's attributes. Finally use PostHTTP to submit the data to Event HubsPlace the following processors on the pallet and set the run schedule of the generate flowfile processor to something reasonable like 10 seconds.Configure Update Attribute, adding the "Authorized" property with a SAS token. For more information on creating a SAS token see https://community.hortonworks.com/articles/69823/creating-shared-access-signature-sas-for-posting-d.htmlConfigure the PostHTTP to post the URL https://eventhub-nifi.servicebus.windows.net/hub-nifi/messagesNote: I set the content type to text/plain you should set this to suit your use case.Only the truststore needs to be configured for this example as we only need to trust the server certific

ISSUE : Ranger HA fails if admin password is changed to non default passwordError stack :Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 208, in <module> RangerAdmin().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 59, in install setup_ranger_admin_passwd_change() File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py", line 598, in setup_ranger_admin_passwd_change Execute(cmd, environment={'JAVA_HOME': params.java_home, 'RANGER_ADMIN_HOME': params.ranger_home}, user=params.unix_user) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_manag

ISSUE : Ranger HA fails if admin password is changed to non default passwordError stack :Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 208, in <module> RangerAdmin().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 59, in install setup_ranger_admin_passwd_change() File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py", line 598, in setup_ranger_admin_passwd_change Execute(cmd, environment={'JAVA_HOME': params.java_home, 'RANGER_ADMIN_HOME': params.ranger_home}, user=params.unix_user) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_manag