Resources

Periodically, and when there are no regions in transition, a load balancer will run and move regions around to balance the cluster’s load. See Balancer for configuring this property. The Load Balancer ensures that the region replicas are not co-hosted in the same region servers and also in the same rack (if possible).The HDFS balancer attempts to spread HDFS blocks evenly among DataNodes. HBase relies on compactions to restore locality after a region split or failure. These two types of balancing do not work well together.In the past, the generally accepted advice was to turn off the HDFS load balancer and rely on the HBase balancer, since the HDFS balancer would degrade locality. HDFS-6133 provides the ability to exclude favored-nodes (pinned) blocks from the HDFS load balancer, by setting thedfs.datanode.block-pinning.enabled property to true in the HDFS service configuration.HBase can be enabled to use the HDFS favored-nodes feature by switching the HBase balancer class (conf: hbase

Problem There are time we would want to remove a ZK node in a secure cluster which is ACL protected. Something as below ACLs[zk: xyz.com:2181(CONNECTED) 0] getAcl /infra-solr 'sasl,'infra-solr : cdrwa 'world,'anyone : r [zk: xyz.com:2181(CONNECTED) 0] rmr /testAuthentication is not valid : /testHere only read privilege is available to rest. Soln Goto zookeeper home. for e.x cd /usr/hdp/current/zookeeper-serverRun below commandjava -cp "./zookeeper.jar:lib/slf4j-api-1.6.1.jar" org.apache.zookeeper.server.auth.DigestAuthenticationProvider super:password SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. super:password->super:DyNYQEQvajljsxlhf5uS4PJ9R28= Copy the super:DyNYQEQvajljsxlhf5uS4PJ9R28= text and login to Ambari and goto zookeeper config.Add below to zookeeper-env template configexport SERVER_JVMFLAGS="$SERVER_JVMFLAGS

Steps to enable SSL for Grafana from the keystore file.1. First create keystore file using below command.$JAVA_HOME/bin/keytool -genkey -keyalg RSA -alias grafana -keystore keystore.jks -validity 360 -keysize 20482. Now export cert from JKS filekeytool -export -alias grafana -file server.cer -keystore keystore.jks3. Convert cert from DER to PEM formatopenssl x509 -in server.cer -inform der -outform pem -out cert.pem4. Export privatekey to PKCS12 filekeytool -v -importkeystore -srckeystore keystore.jks -srcalias ambari -destkeystore myp12file.p12 -deststoretype PKCS125. Now export private key from PKCS12 to PEM format.openssl pkcs12 -in myp12file.p12 -clcerts -nodes -nocerts | openssl rsa > private.keyNote: private.key should be unencrypted file. it should be like -----BEGIN RSA PRIVATE KEY-----xxxx-----END RSA PRIVATE KEY-----Should be not having like "-----BEGIN ENCRYPTED PRIVATE KEY-----"The reason for using un encrypted key is because Grafana does not have any option to take

Hive metastore DB Connection verification from Command line:-You can run the following on any node where ambari agent installed on the cluster,You need the following information to run this test.You can use this to validate if the password stored in ambari and the actual mysql db password are same.1. mysql db hostname2. mysql db port number3. mysql database name which hive metastore uses4. mysql username5. mysql passwordSyntax:-java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/share/java/mysql-connector-java.jar -Djava.library.path=/usr/lib/ambari-agent org.apache.ambari.server.DBConnectionVerification "jdbc:mysql://<mysql db hostname>:<mysql db port number>/<mysql database name>" "<mysql username>" "<mysql password>" com.mysql.jdbc.DriverExample:-/usr/jdk64/jdk1.8.0_112/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/share/java/mysql-connector-java.jar -Djava.library.path=/usr/lib/ambari-agent org.apache.ambari.server.

Below steps are for taking backup of ranger audits and then restore them back to new collection which will be using updated manage-schema fix given in RANGER-1938. Note: Steps are only for Solr version 5.x and Ranger service managed by Ambari. 1) Backup existing ranger_audits collection: curl -X GET "http://${SOLR_HOST}:${SOLR_PORT}/solr/ranger_audits/replication?command=backup&location=/tmp" 2) Download existing managed-schema from Zookeeper: /usr/lib/ambari-infra-solr/server/scripts/cloud-scripts/zkcli.sh --zkhost "${ZK_HOST}:${ZK_PORT}/${SOLR_ZNODE}" -cmd getfile /configs/ranger_audits/managed-schema /tmp/managed-schema 3) Upload updated managed-schema given here to Zookeeper: /usr/lib/ambari-infra-solr/server/scripts/cloud-scripts/zkcli.sh --zkhost "${ZK_HOST}:${ZK_PORT}/${SOLR_ZNODE}" -cmd putfile /configs/ranger_audits/managed-schema managed-schema 4) Delete existing ranger_audits collection: curl -i "http://${SOLR_HOST}:$

For certain large environments, it's very easy to for Spark History Server to get overwhelmed by the large number of applications being posted and number of users / developers viewing history data. Spark jobs create an artifact called the history file which is what is parsed by the Spark History Server (SHS) and served via the UI. The size of this file has a huge impact in driving the load on the SHS also note that the size of history file is determined by the number of events generated by the SHS (small executor heart beat interval)Workaround:If you are still interested in analyzing performance issues with these large history files, one option is to download these files and browse them from a locally hosted SHS instance. To run this:Download Spark 1.6 https://spark.apache.org/downloads.htmlUnpackCreate a directory to hold the logs called spark-logsCreate a properties file called test.propertiesInside test.properties add spark.history.fs.logDirectory=<path to the spark-logs director

Initial Login to ODroid XU4GPS Command Line CheckWiFi Signal StrengthInstall Ubuntu ExtrasAbove is our device with GPSThe ODroid XU4 is a powerful little inexpensive IoT device. It is similar to a Raspberry PI and compatible with many of the libraries. I chose to install ARMBIAN Ubuntu which is a good OS. It is very easy to add GPS libraries, Apache MiniFi, JDK 8 and Python for processing.Apache NiFi processing flowApache MiniFi Flow on DeviceApache NiFi ProcessingSee: https://community.hortonworks.com/content/kbentry/155604/iot-ingesting-camera-data-from-nanopi-duo-devices.htmlSource Codehttps://github.com/tspannhw/odroidxu4-gps-python-minifiSoftwaresudo apt-get update sudo apt-get upgrade sudo apt-get dist-upgrade armbian-configsudo add-apt-repository ppa:webupd8team/javasudo apt-get updatesudo apt-get install oracle-java8-installer -ysudo apt-get install oracle-java8-set-default -yblkidswapon /dev/sda1sudo apt-get install gpsd gpsd-clientssudo dpkg-reconfigure gpsdsudo gpsmo

This is pretty close to a Raspberry Pi Zero. This inexpensive box is another useful IoT device for ingesting some data very inexpensively.Machine Browsingroot@NanoPi-Duo:/home/pi# cpu_freqINFO: HARDWARE=sun8iCPU0 online=1 temp=48092 governor=ondemand cur_freq=312000CPU1 online=1 temp=48092 governor=ondemand cur_freq=312000CPU2 online=1 temp=48092 governor=ondemand cur_freq=312000CPU3 online=1 temp=48092 governor=ondemand cur_freq=312000Source Code for Python and Shell Scripthttps://github.com/tspannhw/nifi-nanopi-duoHardwareThe FA-CAM202 is a 200M USB camera.512MB RAM Ubuntu 16.04.3 LTS 4.11.2Similar SBC to Raspberry PIARMSoftware Setupsudo apt-get install fswebcam -ysudo apt-get install libv4l-dev -ysudo apt-get install python-opencv -ysudo npi-configsudo apt-get update sudo apt-get install libcv-dev libopencv-dev -ypip install psutil pip2 install psutilcurl http://192.168.1.193:8080/nifi-api/site-to-site/ -vinferred.avro.schema { "type" : "record", "name" : "NANO", "fields" : [ { "

package com.big.data.sparkserver2;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.security.UserGroupInformation;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import java.io.IOException;import java.sql.*;public class SparkServer2Client { private static final Logger LOGGER = LoggerFactory.getLogger(SparkServer2Client.class); private static String HIVESERVE2DRIVER = "org.apache.hive.jdbc.HiveDriver"; public static void main(String[] args) throws SQLException, IOException { // set the configurationConfiguration conf = new Configuration(); conf.set("hadoop.security.authentication", "Kerberos"); UserGroupInformation.setConfiguration(conf); // /etc/security/keytab/hive.service.keytab is from local machine, This is the user which is executing the commandUserGroupInformation.loginUserFromKeytab("hive/hostname.com@FIELD.HORTONWORKS.COM", "/etc/security/keytab/hive.service.keytab"); // load the drivertry { Class.

package com.big.data.hiveserver2;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.security.UserGroupInformation;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import java.io.IOException;import java.sql.*;public class HiveServer2Client { private static final Logger LOGGER = LoggerFactory.getLogger(HiveServer2Client.class); private static String HIVESERVE2DRIVER = "org.apache.hive.jdbc.HiveDriver"; public static void main(String[] args) throws SQLException, IOException { // set the configuration Configuration conf = new Configuration(); conf.set("hadoop.security.authentication", "Kerberos"); UserGroupInformation.setConfiguration(conf); // /etc/security/keytab/hive.service.keytab is from local machine, This is the user which is executing the command UserGroupInformation.loginUserFromKeytab("hive/cluster20182.field.fieldorg.com@REALM_NAME", "/etc/security/keytab/hive.service.keytab"); // load the driver

Hive is very powerful, but sometimes you need to add some procedural code for a special circumstance such as complex parsing of a field. Hive provides the ability to easily create User Defined Table Functions (UDFT’s). These allow you to transform your Hive results, pass them through the UDTF and return data as a set of rows that can then be used like any other Hive result set. These can be written in Java or Python, and we will use Python for this article. However, the techniques here are applicable to both with some syntax changes.There are a lot of great articles on building these such as https://community.hortonworks.com/articles/72414/how-to-create-a-custom-udf-for-hive-using-python.html.These pretty much work as advertised, but don’t get into how to debug or troubleshoot your code. However, when you get into any significant logic (and sometimes not so significant!), you are likely to create a few bugs. This can be an issue parsing, or in the case of Python, can even be syntax err

Oracle -> GoldenGate -> Apache Kafka -> Apache NiFi / Hortonworks Schema Registry -> JDBC Database Sometimes you need to process any number of table changes sent from tools via Apache Kafka. As long as they have proper header data and records in JSON, it's really easy in Apache NiFi. Requirements: Process Each Partition Separately Process Records in Order as each message is an Insert, Update or Delete to an existing table in our receiving JDBC store. Re-process if data lost   For The Main Processor for Routing, It must only run on the Primary Node. Enforcing Order We use the Kafka.Offset to order the records, which makes sense in Apache Kafka topics. After Insert, Update, Delete queries are built, let's confirm and enforce that strict ordering. To further confirm processing in order, we make each connection in the flow FirstInFirstOutPrioritizer. To Route, We Route Each Partition to A Different Processor Group (One Local, The Other Remote) Let's Store Some Da

Processing Voice DataUsing a Raspberry Pi 3 with Google AIY Voice Kit is an easy way to control data flows in an organization.The AIY Voice Kit is a good way to prototype voice ingest. It is not perfect, but for low end inexpensive hardware it is a good solution for those that speak clearly and are willing to repeat a phrase a few times for activation.I was able to easily add a word for it to react to. I also have it waiting for someone to press the button to activate.Setup and Run Steps1. Install box 2. Setup GCP Account 3. As Pi User ~/bin/AIY-voice-kit-shell.sh src/assistant_library_demo.py 4. Download and Unzip Apache MiniFi 5. Custom Action controlx 6. Run it pi@raspberrypi:~/AIY-voice-kit-python/src $ ./controlx.py 7. start minifi 8. Apache MiniFi will tail the command file and look for audio filesCode Added to Google Example elif event.type == EventType.ON_RECOGNIZING_SPEECH_FINISHED and event.args: print('You said:', event.args['text

Author: Gerdan Rezende dos Santos - https://community.hortonworks.com/users/12342/gerdan.html Co-author: Pedro Drummond - https://community.hortonworks.com/users/10987/pedrodrummond.html 1 - PREREQUISITES Before you begin, you need to have: 1. HDP 2.4 kerberized cluster; 2. Configured repos for HDPSearch; 3. Java + JCE installed with defined JAVA_HOME; 4. Knowledge of you kerberos’ REALM. 2 - KERBEROS CONFIGURATION 2.1 - “Principal” Definition It’s a Kerberos term which defines an entity by means of an address, user or service. This address is commonly divided into 3 parts: The first part contains the username or the service name; The second part contains the hostname if one is defining a service, or left in blank if one is defining a user; The third part contains the REALM. ◦Ex: user1/solr.community.hortonworks.com@LOCAL.COM 2.2 - Installing Kerberos Client

Adding a Movidius NCS to a Raspberry Pi is a great way to augment your deep learning edge programming. Hardware Setup Raspberry Pi 3 Movidius NCS PS 3 Eye Camera 128GB USB Stick Software Setup Standard NCS Setup Python 3 sudo apt-get update sudo apt-get install python-pip python-dev sudo apt-get install python3-pip python3-dev sudo apt-get install fswebcam sudo apt-get install curl wget unzip sudo apt-get install oracle-java8-jdk sudo apt-get install sense-hat sudo apt-get install octave -y sudo apt install python3-gst-1.0 pip3 install psutil Apache MiniFi https://www.apache.org/dyn/closer.lua?path=/nifi/minifi/0.3.0/minifi-0.3.0-bin.zip git clone https://github.com/movidius/ncsdk.git git clone https://github.com/movidius/ncappzoo.git     Run Example ~/workspace/ncappzoo/apps/image-classifier Shell Script /root/workspace/ncappzoo/apps/image-classifier/run.sh #!/bin/bashDATE=$(date +"%Y-%m-%d_%H%M")fswebcam -r 1280x720 --no-banner /opt/demo/images/$DATE.jpgpython3 -W ig

There is an open source model server for Apache MXNet that I recently tried. It's very easy to install and use. You must have Apache MXNet and Python installed. Installation and Setup To install the Model Server it's a simple. I am using Pip3 to make sure I install to Python3 as I also have Python 2.7 installed on my laptop. pip3 install mxnet --pre --userpip3 install mxnet-model-serverpip3 install imdbpypip3 install dataset http://127.0.0.1:9999/api-description { "description": { "host": "127.0.0.1:9999", "info": { "title": "Model Serving Apis", "version": "1.0.0" }, "paths": { "/api-description": { "get": { "operationId": "api-description", "produces": [ "application/json" ], "responses": { "200": { "description": "OK", "schema": { "properties": { "description": { "type": "string"

Affected versions:1.4.0, 1.4.1, 1.4.2, 1.4.3Symptoms: A Bundle is available in Datalake but the status is still Upload Failed. Following error is reported in the hst-server.log.2017-09-08 17:44:09,319 ERROR [pool-1-thread-1] UploadRunnable:236 -Error occurred during upload bundlecom.hortonworks.support.tools.server.SystemException:java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1column 10 atcom.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:182) atcom.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) atcom.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) atcom.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201) atcom.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:176) atcom.hortonworks.support.tools.gateway.UploadRunnable.retryUploadBundle(UploadRunnable.java:201

Good design goes mostly unnoticed. If a user interface works well, you simply complete a task and move on to the next. That time when something doesn't work as expected, or you think why do I have to do it this way, is when the design becomes much more noticeable. The following is a look back over recent months at some notable UI/UX improvements making their way to Apache NiFi. They vary a lot in size and complexity, but overall they contribute equally to make NiFi better by design. This second edition is light due to a lot of contributor focus on NiFi sub-projects MiNiFi and Registry. There will be many new additions to the NiFi UI related to the NiFi Registry integration and those will be covered in separate articles. The previous (and first) edition highlighting recent NiFi UI/UX improvements from September 2017 can be found here.Available Cores In the UI there were several places where it stated "available processors" to indicate how many cores were available to NiFi. In thes

I wanted to see offer alternatives to running Deep Learning and Machine Learning locally and take advantage of some free hours of cloud time. It seems that it is trivially easy to integrate calling IBM Watson APIs for all your microservice needs. Apache NiFi makes it super easy and fun. Using out of the box processors, we can call the InvokeHTTP POST and GET REST APIs of the IBM Watson Platform for Natural Language, Visual Recognition, Personality Analysis, Language Translation and others. These use SSL, so you have to setup a simple StandardSSLContextService in Apache NiFi. Once you know the JVM you are running Apache NiFi with, it's trivial to grab the requirements for that. By default the password is the very secure, changeit. A Get REST CALL to NLP A Post to a Watson REST URL Some URLs and Calls curl -X POST --form "images_file=@mypic.jpg" "https://gateway-a.watsonplatform.net/visual-recognition/api/v3/classify?api_key={api-key}&version=2016-05-20"curl -X

IntroductionEDW (Enterprise Data Warehouse) is a traditional methodology developed and grown over past three decades, highly matured with its own best practices. Hadoop has emerged a decade ago but one of the fastest growing eco system. Merging these two systems without giving importance to details of both the worlds would easily make a scenario of not utilizing strengths of both and eventually lead to frustrations and failures. Most of EDW migration fails due to ignoring fundamentals of EDW and Hadoop.EDWAn Enterprise Data warehouse is set of tools and techniques to enable business to answer compelling questions about what happened. It enables businesses to view the facts of various business processes from a perspective of different dimensions. Example: What is profit generated out of customers from Texas during Christmas season on each year for the past 5 years.In the above profit is a fact or “some form of transaction”, customers, state Texas and Christmas season are dimensions. T

Happy Holidays! Apache NiFi makes it easy to build your own integration tests. So I am generating tests to test Turning On and Off My Christmas Tree Hat. I also testing taking a picture. My use case is to send an HTTP message to trigger a Raspberry Pi to turn on a physical device like a camera or light. This is pretty cool and secure with Apache MiniFi and Apache NiFi. A little Python script is all the code and that's basic example code. This code is a modified TensorFlow classify.py that adds turning on the Christmas Tree. So we turn on the tree and then take a picture with the PiCamera and then run it through a Tensorflow classifier. root@vid5:/opt/demo# cat classifytree.py# Copyright 2015 The TensorFlow Authors. All Rights Reserved.## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0

In the Holidays, it's nice to know how much energy you are using. So one small step is I bought a low-end inexpensive TPLink Energy Monitoring plug for one device. I have been monitoring phone charging and my Apple monitor.Let's read the data and do some queries in Apache Hive and Apache Spark 2 SQL.Processing Live Energy Feeds in The CloudMonitor Energy From a Local OSXIf your local instance does not have access to Apache Hive, you will need to send the data via Site-to-Site to a Remote Apache NiFi / HDF server/cluster that can.For Apache Hive Usage, Please Convert to Apache ORC FilesTo Create Your New Table, Grab the hive.ddlInside of Apache Zeppelin, we can create our table based on the above DDL. We could have also let Apache NiFi create the table for us. I like to keep my DDL with my notebook. Just a personal choice.We can then query our table in Apache Zeppelin utilizing Apache Spark 2 SQL and Apache Hive QL.OverviewStep 1: Purchase an inexpensive energy monitoring pl

Once Kerberos is enabled, it is possible to get a listing of the expected Kerberos principals and keytab files. This data is typically used when manually managing the Kerberos Identities for the Ambari cluster and may be easily downloaded as a CSV file from the Enable Kerberos Wizard in the Ambari UI. After Kerberos is enabled in Ambari, details about the expected Kerberos Identities may be obtained using the following REST API call:GET /api/v1/clusters/:cluster_name/kerberos_identitiesNote: replace :cluster_name with the name of your cluster. The result of this query is a JSON formatted document containing a high level listing of the expected identities. { "href" : "http://ambari-server:8080/api/v1/clusters/c1/kerberos_identities", "items" : [ { "href" : "http://ambari-server:8080/api/v1/clusters/c1/kerberos_identities/HTTP%2Fhost-1.example.com%40EXAMPLE.COM", "KerberosIdentity" : { "cluster_name" : "c1", "host_name" : "host-1.example.com", "princi

1.Configure sample ambari alert using Ambari UI..2.Login to ambari webui using [admin/admin].3.Click on "Alerts". .4.Click "Action"-> "Manage Notification".5.Add sample alert for service.6. In above example you can use Gmail username and password to test connection..7. After adding sample alert close the screen..8.From the ambari ui now stop one of the zookeeper service..9.You should be able to see alert indication in ambari ui for the zookeeper service stopped..10. Same time if you tail for ambari-alerts.log you will be able to see "Connection Failed" log messages for zookeeper service..2016-05-27 14:57:31,541 [CRITICAL] [ZOOKEEPER] [zookeeper_server_process] (ZooKeeper Server Process) Connection failed: [Errno 111] Connection refused to node1.example.com:2181.11. Check your email and make sure you receive the alert..12. If you don’t see alerts the enable debugging for Alerts in ambari log4j.properties as shown below..13. Login to ambar

We are pleased to announce the availability of Hortonworks Data Flow(HDF) v3.0.3 for IBM Power Systems on RHEL 7.2. This is an important release as it is the second Hortonworks product ported to IBM Power Systems, specifically Power8 processors, in 2017.This release is a win-win for both IBM and Hortonworks customers as HDF v3.0.3 is the next generation of our open source data-in-motion platform and enables customers to collect, curate, analyze and act on all data in real-time, across the enterprise. Combined with the Hortonworks Data Platform (HDP) currently available for IBM Power Systems, we are improving the experience for customers by simplifying how they create and deploy streaming analytics applications to deliver real time analytics while benefiting from the flexibility, cost of operation and performance of IBM Power8 processors.For additional information, please refer to:Hortonworks Data Flow v3.0.3 Documentation (link)Hortonworks Data Platform v2.6.3 / Ambari v2.6 Documenta