Resources

This article will go over the concepts of security in an Hbase cluster. More specifically we will concentrate on ACL based security and how to apply at the different levels of granularity on an Hbase model.From an overall security perspective and acces control list , or ACL, is a list of permissions associated with an objectACLs focus on the access rules pattern.ACL logicHbase access contol lists are granted on different levels of data abstractions and cover types of operations.Hbase data layoutBefore we go further let us clear out the hierarchical elements that compose the datastorage HbaseCELL : All values written to Hbase are stored in a what is know as a CELL. (Cell can also be refered to as KeyValue).Cells are identified by a multidimensionnal key {row, column, qualifier, timestamp}. In the example above : CELL => Rowkey1,CF1,Q11,TS1COLUMN FAMILY : A column Family groups together arbitrary cells.TABLE : All Cells belong to a Column family and are organized into a table.NAMESPA

while running the sqoop command from the java program with -verbose option can result into race condition during obtaining lock on the console appender.we can workaround this with the help of SSHXCUTE framework which will create java program and sqoop command context separately.ENV: HDP 2.4Java Version : JDK-8Step 1: download sshxcute jar from https://sourceforge.net/projects/sshxcute/Step 2: Create RunSqoopCommand.javaimport net.neoremind.sshxcute.core.SSHExec;import net.neoremind.sshxcute.core.ConnBean;import net.neoremind.sshxcute.task.CustomTask;import net.neoremind.sshxcute.task.impl.ExecCommand;public class RunSqoopCommand {public static void main(String args[]) throws Exception{ ConnBean cb = new ConnBean("localhost", "root","hadoop"); SSHExec ssh = SSHExec.getInstance(cb); ssh.connect(); CustomTask sqoopCommand = new ExecCommand("sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true -Dmapred.job.na

SYMPTOMRunning java action via Oozie workflow fails with below error: Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], main() threw exception, Could not find Yarn tags property (mapreduce.job.tags) java.lang.RuntimeException: Could not find Yarn tags property (mapreduce.job.tags) at org.apache.oozie.action.hadoop.LauncherMainHadoopUtils.getChildYarnJobs(LauncherMainHadoopUtils.java:52) at org.apache.oozie.action.hadoop.LauncherMainHadoopUtils.killChildYarnJobs(LauncherMainHadoopUtils.java:87) at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:44) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38) at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:36) .ROOT CAUSE Missing or Yarn related jar file conflict in Oozie sharelib..RESOLUTION Complete the following steps as oozie user in the Oozie node:1. Recreate Oozie sharelib using below command/usr/hdp/<hdp-version>/oozie/bin/oozie-setup.sh sharelib create -localli

SYMPTOM Beeline fails with below error:$ beeline --verbose Beeline version 0.14.0.2.2.6.0-2800 by Apache Hive beeline> !connect jdbc:hive2://prodnode1.crazyadmins.com:10000/default;principal=hive/prodnode1.crazyadmins.com@CRAZYADMINS.COM scan complete in 8ms Connecting to jdbc:hive2://prodnode1.crazyadmins.com:10000/default;principal=hive/prodnode1.crazyadmins.com@CRAZYADMINS.COM Enter username for jdbc:hive2://prodnode1.crazyadmins.com:10000/default;principal=hive/prodnode1.crazyadmins.com@CRAZYADMINS.COM: kuldeepk Enter password for jdbc:hive2://prodnode1.crazyadmins.com:10000/default;principal=hive/prodnode1.crazyadmins.com@CRAZYADMINS.COM: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.2.6.0-2800/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.2.6.0-2800/hive/lib/hive-jdbc-0.14.0.2.2.6.0-2800-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: S

SYMPTOM: Access Audit logs show 6 hours behind from Central Timezone. This is related to https://issues.apache.org/jira/browse/RANGER-336ERROR: Ranger Access Audit logs show 6 hours behind from Central Timezone. ROOT CAUSE: This is bug - https://issues.apache.org/jira/browse/RANGER-336RESOLUTION: Create a file name "ranger-admin-env-javaopts.sh" with below entry in path "/usr/hdp/current/ranger-admin/conf/" - export JAVA_OPTS=" ${JAVA_OPTS} -Duser.timezone=UTC"Save the file and restart Ranger admin service.

SYMPTOM: Ambari agent not able to register with Ambari server. ERROR: Below is the error logs - ERROR 2016-12-19 10:17:54,387 Controller.py:194 - Unable to connect to: https://oser402529.host.com:8441/agent/v1/register/oser402566.host.comTraceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 137, in registerWithServer data = json.dumps(self.register.build(self.version)) File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 230, in dumps return _default_encoder.encode(obj) File "/usr/lib/python2.6/site-packages/ambari_simplejson/encoder.py", line 200, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python2.6/site-packages/ambari_simplejson/encoder.py", line 260, in iterencode return _iterencode(o, 0)UnicodeDecodeError: 'utf8' codec can't decode byte 0xac in position 928: invalid start byteERROR 2016-12-19 10:17:54,388 Controller.py:195 - Error:'utf8' codec can't decode byte 0x

These are the steps to build and run spark streaming application, it was built and tested on HDP-2.5setup:ENV: HDP2.5scala : 2.10.4sbt: 0.13.11mkdir spark-streaming-example cd spark-streaming-example/ mkdir -p src/main/scala cd src/main/scalasample code:import org.apache.spark.rdd.RDDimport org.apache.spark.sql.{SQLContext, SaveMode}import org.apache.spark.streaming.StreamingContextimport org.apache.spark.streaming.Secondsimport org.apache.spark.{SparkConf, SparkContext}import org.apache.spark.streaming.Time;import org.apache.spark.SparkConfimport org.apache.spark.SparkContextimport org.apache.spark.rdd.RDDimport org.apache.spark.streaming.{Time, Seconds, StreamingContext}import org.apache.spark.util.IntParamimport org.apache.spark.sql.SQLContextimport org.apache.spark.storage.StorageLevelobject SqlNetworkWordCount { def main(args: Array[String]) { val sparkConf = new SparkConf().setAppName("SqlNetworkWordCount") val ssc = new StreamingContext(spark

Pre-requisitesHortonworks Data Platform 2.5 on CentOS 7.2Python distribution that comes with HDP 2.5 - Python 2.7.5Download and install pip#wget https://bootstrap.pypa.io/get-pip.pyInstall add-on package#pip install requestsStart Python CLI (default version)#pythonImport pre-reqs>>>import requests>>>import json>>>import sysEnvironment VariablesSet Ambari domain variable to the IP address or FQDN of your Ambari node.>>>AMBARI_DOMAIN = '127.0.0.1'Set Ambari port, Ambari user and password variables to match your specifics.>>>AMBARI_PORT = '8080'>>>AMBARI_USER_ID = 'admin'>>>AMBARI_USER_PW = 'admin'Set the following variable to the IP address or FQDN of your ResourceManager node.>>>RM_DOMAIN = '127.0.0.1'Set Resource Manager port variable>>>RM_PORT = '8088'Ambari REST API Call ExamplesLet's find Cluster Name, Cluster Version, Stack and Stack Version:>>>restAPI='/api/v1/clusters'>>>url="http:/

Drones / unmanned aerial vehicle (UAV) Ingest We have several options on ingesting data from Drones depending on the model, location, networking and needs. MQTT is a popular option if you are on the same WIFI or network. See my article on MQTT https://community.hortonworks.com/articles/72420/ingesting-remote-sensor-feeds-into-apache-phoenix.html. Many UAVs will let you push up to a cloud MQTT server and asynchronously consume this data from your enterprise HDF system. They can also be access via TCP, UDP, REST and offline download. For my example I am parsing a directory full of downloaded BEBOP Parrot 2 Drone images. NIFI with Four Parallel Flows from the Ingested File Step 1: GetFile from directory of images. Step 2: PutHDFS Store FlowFile Raw images to HDFS. Step 3: Update Attribute add an attribute (API KEY) to pass to Microsoft Cognitive Services. Step 4: ReplaceText to convert body to a JSON message with image Step 5: InvokeHTTP to call https://api.projectoxford.ai/vision/v1.0/

During debugging a problem with delete topic,I dig into Kafka code to know how delete command works, this the sequence of event occurred during command execution1. TopicCommand issues topic deletion /usr/hdp/current/kafka-broker/bin/kafka-run-class.sh kafka.admin.TopicCommand --zookeeper rkk3.hdp.local:2181 --delete --topic sample 2. which create a new admin path /admin/delete_topics/<topic> 3. The controller listens for child changes on /admin/delete_topic and starts topic deletion for the respective topics 4. The controller has a background thread that handles topic deletion. A topic's deletion can be started only by the onPartitionDeletion callback on the controller.5. Once DeleteTopicsThread invoked it looks for topicsToBeDeleted and for each topic it deregister partition change listener on the deleted topic, This is to prevent the partition change listener firing before the new topic listener when a deleted topic gets auto created. 6. Controller will remove this replica from

This article is a follow-up on my original article about the visualization of a cluster and its services/components ( https://community.hortonworks.com/articles/2010/visualizing-hdp-cluster-service-allocation.html). In the first part I am particularly focusing on a new feature that enables users to build and plan a cluster by using a drag-n-drop Web UI.Build a ClusterUntil now visualizing a cluster and its service allocation either meant exporting the information from Ambari or writing a JSON file that outlines the details of the nodes. Planning and deploying a cluster should be easier, right?I'll introduce, Build a Cluster 🙂This simple Web UI is based on different drag-n-drop functionalities and allows the creation of a new cluster by simply dragging Hadoop components from the elements list to the indivdual nodes.Lets go over the different features....User InterfaceThe UI is divided into two sections Elements & Settings and Cluster:Elements & Settings (left): Contain the avai

BackgroundTungsten became the default in Spark 1.5 and can be enabled in earlier versions by setting spark.sql.tungsten.enabled to true (or disabled in later versions by setting this to false). Even without Tungsten, Spark SQL uses a columnar storage format with Kryo serialization to minimize storage cost.GoalThe goal of Project Tungsten is to improve Spark execution by optimizing Spark jobs for CPU and memory efficiency (as opposed to network and disk I/O which are considered fast enough).ScopeTungsten focuses on the hardware architecture of the platform Spark runs on, including but not limited to JVM, LLVM, GPU, NVRAM, etc. Optimization FeaturesOff-Heap Memory Management using binary in-memory data representation aka Tungsten row format and managing memory explicitly,Cache Locality which is about cache-aware computations with cache-aware layout for high cache hit rates,Whole-Stage Code Generation (aka CodeGen).Design ImprovementsTungsten includes specialized in-memory data structures

ENV: HDP 2.4.2STEP 1: Setting up MySQL SSL# Create clean environmentshell> rm -rf newcertsshell> mkdir newcerts && cd newcerts# Create CA certificateshell> openssl genrsa 2048 > ca-key.pemshell> openssl req -new -x509 -nodes -days 3600 \ -key ca-key.pem -out ca.pem# Create server certificate, remove passphrase, and sign it# server-cert.pem = public key, server-key.pem = private keyshell> openssl req -newkey rsa:2048 -days 3600 \ -nodes -keyout server-key.pem -out server-req.pemshell> openssl rsa -in server-key.pem -out server-key.pemshell> openssl x509 -req -in server-req.pem -days 3600 \ -CA ca.pem -CAkey ca-key.pem -set_serial 01 -out server-cert.pem# Create client certificate, remove passphrase, and sign it# client-cert.pem = public key, client-key.pem = private keyshell> openssl req -newkey rsa:2048 -days 3600 \ -nodes -keyout client-key.pem -out client-req.pemshell> openssl rsa -in client-key.pem -out client-key

while debugging kafka producer slowness, I observed following steps Kafka producer take while producing a single record to kafka broker.ENV: HDP 2.52 node kafka cluster, single producer producing a record to 'testtopic' which has 2 partitions with 2 replicasKafka Producer start with the configured settings it start adding matrices sensors.update cluster metadata version which includes cluster information like broker nodes and partitions, assign version id to this cluster metadata version.Updated cluster metadata version 1 to Cluster(nodes = [Node(-2, rkk2, 6667), Node(-1, rkk1, 6667)], partitions = [])Set up and start Kafka producer I/O thread aka Sender thread.Request metadata update for topic testtopic.Producer's NetworkClient metadata request to one of the broker which consist of api_key,api_version,correlation_id and client_id.Sending metadata request ClientRequest(expectResponse=true, callback=null, request=RequestSend(header={api_key=3,api_version=0,correlation_id=0,client_id=pro

To send data to my firewall protected internal Hadoop cluster, I have my remote Raspberry Pi 3 with attached Sense-Hat use a cron job to send MQTT messages to a cloud hosted MQTT Broker. NiFi then subscribes to that queue and pulls down the messages asynchronously. I use JSON as my packaging format as that is very easy to work with and process in NiFi and elsewhere.The Sense-Hat connects to your non-Pi Zero and provides a number of easy to access sensors including ones for temperature, humidity, X,Y, Z, Roll and Pitch. It also has a nice light display that can be used to display graphical messages. For data capture needs it also has standard Linux reports like CPU temperature that we can also grab. You could also grab memory usage and disk space. Since the API to access Sense-Hat is Python, it makes sense to keep my access program in Python. So I have a small Python program that reads and formats the Sense Hat sensor values, puts them into a JSON document and sends them up

Yarn exposes cluster application REST API to get the application stats.With the Applications API, you can obtain a collection of resources, each of which represents an application. When you run a GET operation on this resource, you obtain a collection of Application Objects. In this example, we will analyze application stats using Spark-SQL.Env: HDP 2.5Spark Version: Spark-2Data Preparation: get the apps.json while invoking REST API using curl commandcurl -v -X GET -H "Content-Type: application/json" http://`hostname`:8088/ws/v1/cluster/apps > /tmp/apps.jsonnext step is to invoke spark shell and execute the sequence of steps as following[spark@rkk1 ~]$ spark-shell SPARK_MAJOR_VERSION is set to 2, using Spark2Setting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel).16/12/17 16:01:52 WARN SparkContext: Use an existing SparkContext, some configuration may not take effect.Spark context Web UI available at http://172.26.81.127:4040Spark context available a

Many times we find that the operations (like start/stop/restart ...etc) are failing from ambari UI. In such cases if we want to troubleshoot what Ambari UI did to perform that operation or How the commands were executed. Then we can manually execute those same operations from the individual host with the help of "/var/lib/ambari-agent/data/command-xxx.json" file.- When we perform any operation from ambari UI like (Starting / Stopping Datanode) then we will notice that ambari shows the operation progress in the UI. There we can see basically the following two files.Example: stderr: /var/lib/ambari-agent/data/errors-952.txtstdout: /var/lib/ambari-agent/data/output-952.txt- Apart from the above files there is one more important file which ambari agent uses to execute the instructions/commands that are sent by the AmbariServer. We can find that specific file in the ambari-agent's "/var/lib/ambari-agent/data/command-xxx.json" file. /var/lib/ambari-agent/data/command-952.json - Here the "c

[Related Article On Ambari Server Tuning : https://community.hortonworks.com/articles/131670/ambari-server-performance-tuning-troubleshooting-c.html ]-The jcmd utility comes with the JDK and is present inside the "$JAVA_HOME/bin". It is used to send diagnostic command requests to the JVM, where these requests are useful for controlling Java Flight Recordings, troubleshoot, and diagnose JVM and Java Applications. Following are the conditions for using this utility. - 1. It must be used on the same machine where the JVM is running - 2. Only a user who own the JVM process can connect to is using this utility. This utility can help us in getting many details about the JVM process. Some of the most useful information's are as following: Syntax: jcmd $PID $ARGUMENT Example1: Classes taking the most memory are listed at the top, and classes are listed in a descending order./usr/jdk64/jdk1.8.0_60/bin/jcmd $PID GC.class_histogram > /tmp/22421_ClassHistogram.txt Example2: Generate Hea

Origin: http://henning.kropponline.de/2015/05/19/hivesink-for-flume/With the most recent release of HDP (v2.2.4) Hive Streaming is shipped as technical preview. It can for example be used with Storm to ingest streaming data collected from Kafka as demonstrated here. But it also still has some serious limitations and in case of Storm a major bug. Nevertheless Hive Streaming is likely to become the tool of choice when it comes to streamline data ingestion to Hadoop. So it is worth to explore already today.Flume's upcoming release 1.6 will contain a HiveSink capable of leveraging Hive Streaming for data ingestion. In the following post we will use it as a replacement for the HDFS sink used in a previous post here. Other then replacing the HDFS sink with a HiveSink none of the previous setup will change, except for Hive table schema which needs to be adjusted as part of the requirements that currently exist around Hive Streaming. So let's get started by looking into these restrict

IntroductionMany organizations have come to rely on Hadoop for dealing with the ever-increasing quantities of data that they gather. Today, it is clear what problems Hadoop can solve, however, cloud is still not the first choice for Hadoop deployment. Pros and cons for Hadoop in the cloud have been shared across multiple blogs and books, but the question is always coming-up in discussions with enterprises considering Hadoop in the Cloud. Thus, I thought it would be useful to collate together a few pros and cons, as well as mention a pragmatic approach to consider a hybrid cloud for organizations that have made significant investments on-prem. For organizations going to Hadoop for the first time, Cloud is probably a better bet especially if they don't have a lot of IT expertise and a great stream of revenue exists and needs to be exploited immediately.Pro CloudLack of space. You don’t have space to keep racks of physical servers, along with the necessary power and cooling.Flexibility. I

As part of this article we will see that sometimes we get the following very common error while accessing some services (in the following case while running very basic "hdfs" commands).While running the command we see the error as following:[root@kjss1 ~]# su - hdfs[hdfs@kjss1 ~]$ hdfs dfs -ls /user/admin16/12/16 12:28:27 WARN ipc.Client: Exception encountered while connecting to the server : 16/12/16 12:28:27 WARN retry.RetryInvocationHandler: Exception while invoking GetUserMappingsProtocolClientSideTranslatorPB.getGroupsForUser over kjss2.example.com/10.20.169.238:8020. Not retrying because failovers (15) exceeded maximum allowed (15)java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "kjss1.example.com/172.26.69.237"; destination host is: "kjss2.example.com":8020; at org.apa

- In this article we will see that in general how ambari makes use of the Python "urllib2" and "json" (simplejson) module to query and parse the API data. - We will need to install the "simplejson" python module on our system. - Now we can simply write following kind of python script "/tmp/NameNodeMetricsDetails.py"import timeimport jsonimport urllib2import base64import pprint # Ambari Admin credentials.USERNAME = "admin"PASSWORD = "admin"# NameNode metrics.query = "http://kjss1:8080/api/v1/clusters/JoyCluster/services/HDFS/components/NAMENODE"connection_timeout = 30.0def get_json_beans(query, connection_timeout): response = None try: raw = "%s:%s" % (USERNAME, PASSWORD) auth = 'Basic %s' % base64.b64encode(raw).strip() request = urllib2.Request(query) request.add_unredirected_header('Authorization', auth) response = urllib2.urlopen(request, timeout=connection_timeout) data = response.read() data_dict = json.loads(data) return data_dict finally: if response is not None

HDFS Snapshots are read-only point-in-time copies of the file system. Snapshots can be taken on a subtree of the file system or the entire file system.Snapshots are very efficient because they only copy data that are changed. We can restore the data to any previous snapshot. Some common use cases of snapshots are Data backup and Disaster recovery.HDFS Snapshot Extension:Falcon will support HDFS snapshot-based replication through HDFS Snapshot extension. Using this feature,create and manage snapshots on source/target directories.Mirror data from source to target for disaster recovery using these snapshots.Perform retention on the snapshots created on source and target.Snapshot replication will only work from single source directory to single target directory.For snapshot to work, we expect users to do the following­Both source and target clusters must have a version of Hadoop 2.7.0 or higher.The user submitting and scheduling the falcon extension should have permissions on both source a

Introduction Recently worked with use case which required heavy xml processing. Instead of writing complex custom code end up achieved everything easily with NiFi. I thought this will be useful of someone interested for XML processing in NiFi. The document in general covers the following. Base64 Encoding and Decoding of XML message.Character Set conversion from UTF to Ascii ISO-8859-1XML validation against the XSD.Split the XML into smaller chunks.Transform XML to JSON.Extract the content and outputs into unique files based on content. This is very generic XML processing flow which can be leveraged across many business use cases which process xml data. Apache NiFi Flow In the sample demo scenario, External system sends the Base64 encoded XML data in file format which is read through GetFile processor. Next Base64EncodeContent processor decoded the Base64 content.Incoming data in UTF-8 format with leading BOM bytes which gets converted to the ISO-8859-1 format using the ConvertCha

I have a Phoenix PutSQL that worked in the past, but not today. From the error box, not much help. Thought maybe Hbase or Phoenix was done. Turned out one field was missing. How to investigate this issue? Digging through the logs I quickly found it: Caused by: java.sql.SQLException: ERROR 218 (23018): Constraint violation. DRONEDATA1.DATEKEY may not be null That would be nice in the error listed in NiFi.If something seems like an odd error, dig into the logs for more detailed descriptions.Also the Summary Screen is pretty awesome and will let you find a lot of information for debugging and fixing problems.The Connections tab will let you sort by Queue Size and you can see if things are backed up.If you click the system diagnostics link you will see two tabs of useful information. The first has memory usage and garbage collection.The System tab has your usage for FlowFile and Content Repositories, processors and process load.For more details, check out the following reference