Resources

During the upgrade : if you are using postgres DB, you will face a script failure on the following statement Alter table users add constraint "uniq_user_0" Solution : psql -U ambari_user -d NN_HA_ambari alter table ambari.user drop constraint "uniq_user_0"   After the Upgrade : Spark 1-you have to recompile and rebuild your jars using the new dependencies of Spark 1.6.2 Your pom.xml needs to have the new versions ( new jars of HDP 2.5.3 ) 2-Update any custom jars and check all versions used in your Pom.xml i.e <spark.version>1.6.2</spark.version><hbase.version>1.2.1</hbase.version>   3- Remove the argument spark.yarn.jar if you are using it when submitting the job. --conf spark.yarn.jar=hdfs://NN_HA/user/oozie/share/lib/lib_20160515013357/spark/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar. Error noticed when leaving the arg above : java.lang.IllegalArgumentException: Invalid ContainerId: container_e97_1484824056011_0038_02_000001 or jav

In order to secure access to the Zeppelin UI, we will want to enable TLS (as well as authentication) to ensure confidentiality of communication and to assure the identity of the Zeppelin server. Zeppelin uses Jetty as the underlying HTTP server, so we'll refer to Jetty documentation.In this how-to we'll use a self-signed certificate. In Production environments, you will likely obtain a CA-issued certificate or a trusted root certificate from your PKI team specific to your environment.Since self-signed certificates won't be trusted by your browser by default, we'll show how to trust this certificate on OS X 10.11.6 with Chrome version 55.0.2883.95 (other OS/browser combinations are out of the scope of this article).To generate the self-signed certificate, we'll use the openssl and keytool utilities as follows (see this Jetty doc for reference):openssl genrsa -des3 -out zeppelin.keyopenssl req -new -x509 -key zeppelin.key -out zeppelin.crtkeytool -keystore keystore -import -alias zeppeli

There are quite a few different reasons (already posted) in which i've seen this error obtained. Nonetheless I ran into this issue with a client recently in which none of the posted articles accounted for. The customer was attempting to load the Ambari UI and received the "unable to connect to Ambari server. Confirm ambari server is running and you can reach ambari server from this machine" error. We checked Iptables which were turned on, but when disabled the issue persisted.We checked Ambari Server permspace but the amount specified was sufficient.We checked SElinux and it was disabled.We were finally able to load into the Ambari Server after checking "tripwire":ps -ef |grep tripwireOnce tripwire was disabled, we were successfully able to login. More information regarding tripwire can be found here:https://en.wikipedia.org/wiki/Open_Source_Tripwire

Hadoop can be deployed on a variety of scales. The requirements at each of these will be different. Hadoop has a large number of tunable parameters that can be used to influence its operation. Furthermore, there are a number of other technologies which can be deployed with Hadoop for additional capabilities. Performance Monitoring In HadoopMultiple tools exist to monitor large clusters for performance and troubleshooting. This section briefly highlights two such tools.Ganglia is a performance monitoring framework for distributed systems. Ganglia provides a distributed service which collects metrics on individual machines and forwards them to an aggregator which can report back to an administrator on the global state of a cluster.Ganglia is designed to be integrated into other applications to collect statistics about their operation. Hadoop includes a performance monitoring framework which can use Ganglia as its backend. Instructions are available on the Hadoop wiki as to how to enable

This article describe important configuration steps to consider for securing the Hortonworks Data Platform (HDP) Hadoop clusters that are created from Azure marketplace images of HDP. The current HDP offering on Azure utilizes a template that locks down all ports and protocols by default, except for port 22, for security reasons. This article explains the methods to secure and limit access to the service endpoints (for example those of Ambari) to known IP addresses from the Internet from this default configuration.Here are the steps to create an Azure IaaS cluster:Login to your Azure portal and bring up the Marketplace listings:2. Find the Hortonworks listing.3. Fill in the information to create a cluster.4. Fill in the details for the security information. We recommend using SSH keys instead of passwords for direct access to the nodes, and a strong password for Ambari. 5. Accept the terms and launch the cluster.6. Wait for the cluster deployment to complete.7. If you need to access th

DESCRIPTION:CREATE TABLE EXAMPLE_REPORT1 > ( > key string, > claim_type_code string, > yearservice string, > monthservice string > ) > STORED BY "org.apache.hadoop.hive.hbase.HBaseStorageHandler" > WITH SERDEPROPERTIES("hbase.columns.mapping" = ":key,claim_type_code:claim_type_code, > yearservice:yearservice,monthservice:monthservice") > TBLPROPERTIES("hbase.table.name"="EXAMPLE_REPORT1”);Exception received was as follows:FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208)at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:326) at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:301) at org.apache.hadoop.hbase.client.ClientScanner.initializeScanner

HDB 2.1.1Reference: http://hdb.docs.pivotal.io/211http://hdb.docs.pivotal.io/211/hdb/releasenotes/HAWQ211ReleaseNotes.htmlhttp://hdb.docs.pivotal.io/211/hdb/install/install-ambari.htmlDownload HDB from Hortonworks at http://hortonworks.com/downloads/ or directly from Pivotal at https://network.pivotal.io/products/pivotal-hdb (You need to create a pivotal account)What to look out for If you use only only 1 Master Node, you cannot have a Hawq Master and StandbyIf I install Hawq Master on Same node with Ambari need to change PostGres Port from 5432 on InstallPrep Ensure that httpd is installedyum install httpdsudo service httpd statussudo service httpd startGet and Install repo Log onto Pivotal and download hdb-2.1.1.0-7.tar/* On Ambari Node */1. mkdir /staging2. chmod a+rx /staging3. scp -i <<your key>> -o 'StrictHostKeyChecking=no' hdb-2.1.1.0-7.tar root@<<ambarinode>>:~/staging4. tar -zxvf hdb-2.1.1.0-7.tarcd /staging/hdb-2.1.1.0./setup_repo.sh/* You should s

Apache Spark has ignited an explosion of data exploration on very large data sets. Spark played a big role in making general purpose distributed compute accessible. Anyone with some level of skill in Python, Scala, Java, and now R, can just sit down and start exploring data at scale. It also democratized Data Science by offering ML as a series of black boxes. Training of Artificial Intelligence is now possible for those of us who do not have PHDs in Statistics and Mathematics. Now Spark SQL is also helping to bring data exploration to the business unit directly. In partnership with Apache Hive, Spark has been enabling users to explore very large data sets using SQL expression. However, in order to truly make Spark SQL available for ad-hoc access by business analysts using BI tools, fine grain security and governance are necessary. Spark provides strong authentication via Kerberos and wire encryption via SSL. However, up to this point, Authorization was only possible via HDFS ACLs. This

On RHEL/CentOS you might encounter an exception when trying to stop or restart Oozie :resource_management.core.exceptions.Fail: Execution of 'cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-stop.sh' returned 1. -bash: line 0: cd: /var/tmp/oozie: No such file or directoryThis is likely because of a shell crontab /etc/cron.daily/tmpwatch which delete files/directories unmodified for 30d+[root@local ~]# cat /etc/cron.daily/tmpwatch#! /bin/shflags=-umc/usr/sbin/tmpwatch "$flags" -x /tmp/.X11-unix -x /tmp/.XIM-unix \ -x /tmp/.font-unix -x /tmp/.ICE-unix -x /tmp/.Test-unix \ -X '/tmp/hsperfdata_*' 10d /tmp/usr/sbin/tmpwatch "$flags" 30d /var/tmpfor d in /var/{cache/man,catman}/{cat?,X11R6/cat?,local/cat?}; do if [ -d "$d" ]; then /usr/sbin/tmpwatch "$flags" -f 30d "$d" fidoneJust recreate the directory and you're good to go[root@local ~]# mkdir /var/tmp/oozie[root@local ~]# chown oozie:hadoop /var/tmp/oozie[root@local ~]# chmod 755 /var/tmp/ooz

Story: From the documentation I was able to add a service to existing stack definition in Ambari. Issue: But I was either not able to stop the service or delete the service just in case.https://cwiki.apache.org/confluence/display/AMBARI/Defining+a+Custom+Stack+and+ServicesHow did I solve the problem? 1.Create and Add Stack: cd /var/lib/ambari-server/resources/stacks/HDP/2.4/services 2.Create a directory that contains the service definition for SAMPLESRVmkdir /var/lib/ambari-server/resources/stacks/HDP/2.4/services/SAMPLESRVcd /var/lib/ambari-server/resources/stacks/HDP/2.4/services/SAMPLESRV 3.Create a metainfo.xml as show in the link above. 4.With this we have a service name as SAMPLESRV and it contains SAMPLESRV_MASTER, SAMPLESRV_SLAVE and SAMPLESRV_CLIENT 5.Next we need to create the command scriptsmkdir –p /var/lib/ambari-server/resources/stacks/HDP/2.4/services/SAMPLESRV/package/scriptscd /var/lib/ambari-server/resources/stacks/HDP/2.4/services/SAMPLESRV/package/scripts6.Browse t

If you have ever tried to spawn multiple cloudbreak shells you may have run into an error. That is because the default "cbd util cloudbreak-shell" uses docker containers. The fastest work around of this is to use the Jars directly. These Jars can be remotely run from your personal machine or run on the cloudbreak machine itself.Prepping the cloudbreak machine(only needed if running jars locally on the AWS image)Log into your cloudbreak instance and go to /etc/yum.repos.dRemove the Centos-Base.repo file (this is a redhat machine and this can cause conflicts)Install java-8 (yum install java-1.8.0*)Change directory back to /home/cloudbreakDownloading the JarSet a global variable equal to your cloudbreak version (export CB_SHELL_VERSION=1.6.1)Download the jar (curl -o cloudbreak-shell.jar https://s3-eu-west-1.amazonaws.com/maven.sequenceiq.com/releases/com/sequenceiq/cloudbreak-shell/$CB_SHELL_VERSION/cloudbreak-shell-$CB_SHELL_VERSION.jar)Using the JarInteractive mode (java -jar ./cl

When using the Oozie Proxy job submission API for submitting the Oozie Hive, Sqoop and Pig actions. To pass any configuration to the action is required to be in below format.For Hive action:oozie.hive.options.size : The number of options you'll be passing to Pigoozie.hive.options.n : An argument to pass to Hive, the 'n' should be an integer starting with 0 to indicate the option number<property> <name>oozie.hive.options.1</name> <value>-Doozie.launcher.mapreduce.job.queuename=hive</value></property><property> <name>oozie.hive.options.0</name> <value>-Dmapreduce.job.queuename=hive</value></property><property> <name>oozie.hive.options.size</name> <value>2</value></property>For Pig Action: oozie.pig.options.size : The number of options you'll be passing to Pigoozie.pig.options.n : An argument to pass to Pig, the 'n' should be an integer starting with 0 to indicate

Today we will show how to interact with a NiFi instance to modify a flow at runtime via API.Pre-requisites NiFi installed and running on a localhost: https://nifi.apache.org/download.htmlGroovy - because its JSON builders and REST DSLs are great. If you are on a Mac, the easiest is to run brew install groovy. To install Homebrew (the superb Mac package manager), visit http://brew.sh/Full script is available on GitHub: https://github.com/aperepel/nifi-rest-api-tutorialNiFi REST API Docs: https://nifi.apache.org/docs/nifi-docs/rest-api/index.htmlHere’s the test flow we will be working with today:Prepare the test flow: Add a PutFile component to the canvas.Rename the processor to Save File (right-click -> Configure -> Settings -> Name field). We will be using this name to look up the processor later via API.Add a GetHTTP processor and create a connection from GetHTTP to ‘Save File’. GetHTTP settings can be ignored for now, the Save File processor simply needs an input connected.S

Raspberry PIs and other small devices often have cameras or can have camera's attached. Raspberry Pi's have cheap camera add-ons that can ingest still images and videos (https://www.raspberrypi.org/products/camera-module/). Using a simple Python script we can ingest images and then ingest them into our central Hadoop Data Lake. This is a nice simple use case for Connected Data Platforms with both Data in Motion and Data at Rest. This data can be processed in-line with Deep Learning Libraries like TensorFlow for image recognition and assessment. Using OpenCV and other tools we can process in-motion and look for issues like security breaches, leaks and other events. The most difficult part is the Python code which reads from camera, adds a watermark, converts to bytes, sends to MQTT and then ftps to an FTP server. I do both since networking is always tricky. You could also add if it fails to connect to either, store to a directory on a mapped USB drive. Once network returns

GoalThe goal of of this article is to get familiar with the most convenient method of accessing HAWQ Database i.e. using psql native clientAssumptions HAWQ Database is available and activePSQL Native Client (Available for download @ https://network.pivotal.io)PSQL or "psql" (the binary command) is the ideal choice to connect to HAWQ database/warehouse because HAWQ itself is based on Postgres database. Steps to configure psql on Client/Edge node: Make a note of the Public IP address for the Client/Edge Node, this will be needed later. Access to HAWQ is available by default from HAWQ master host, make a note of HAWQ database name and user you would want to use for enabling database access. To identify name of database (psql -l) from the gpadmin user prompt in linuxTo identify available user names "psql -c \du" Download the HAWQ installation binaries from the following site https://network.pivotal.io/products/pivotal-hdb#/releases/3466/file_groups/520Follow this guide to setup the HAWQ r

Thanks to deep collaboration between Hortonworks and Isilon engineering teams, we have come up with the below table that shows the status of Ambari ViewsThrough Ambari 2.2.2 Ambari Version Kerberos Enabled WebHDFS* Enabled Ambari View Works? Remarks Through Ambari 2.2.2 Y Y No Does not work with Kerberos enabled because OneFS implementation of WebHDFS does not support chunked encoding. However we believe that if we use HTTPFS in front , it should work. This has not been officially tested though. Through Ambari 2.2.2 Y N No Does not work in this combination . This is default behavior of Ambari views which does not support Kerberos without WebHDFS. Through Ambari 2.2.2 N Y Yes If the default fs.defaultfs is set to "hdfs" in core-site.xml, then it works. The default value in Ambari is: hdfs:// <rpc address, so it works by default. if you need to use custom Hive View Cluster

livy documentation states to add jars with the interpreter key value livy.spark.jars. This is applicable when running livy interpreter in yarn-cluster mode only (livy.spark.master). This can be done in local or yarn-client mode as well however by adding the jars to the /usr/hdp/<version>/livy/repl-jars directory. This will add the jars to the spark submit command run by livy when executing within a notebook in modes other than yarn-cluster.

Some people say I must have a bot to read and reply to email at all crazy hours of the day. An awesome email assistant, well I decided to prototype it. This is the first piece. After this I will add some Spark machine learning to intelligently reply to emails from a list of pretrained responses. With supervised learning it will learn what emails to send to who, based on Subject, From, Body Content, attachments, time of day, sender domain and many other variables. For now, it just reads some emails and checks for a hard coded subject. I could use this to trigger other processes, such as running a batch Spark job. Since most people send and use HTML email (that's what Outlook, Outlook.com, Gmail do), I will send and receive HTML emails as to make it look more legit. I could also run my fortune script and return that as my email content. Making me sound wise, or pull in a random selection of tweets about Hadoop or even recent news. Making the email very current and fresh. Snippet Example

Protect Your Cloud Big Data AssetsStep 1: Do not put anything into the cloud unless you have a CISO, Chieft Security Architect, Certified Cloud Administrator, full understanding of your PII and private data, a Lawyer to defend you against the coming lawsuits, full understanding of Hadoop, Hadoop Certified Administrators, a Hadoop premier support contract, a security plan, full understanding of your Hadoop architecture and layout. Step 2: Study all running services in Ambari. Step 3: Confirm and check all of your TCP/IP ports. Hadoop has a lot of them!Step 4: if you are not using a service, do not run it.Step 5: By default, disable all access to everything, always. Only open ports and access when something and someone critical cannot access them. Step 6: SSL, SSH, VPN and Encryption Everywhere.Step 7: Run Knox! Set it up correctly.Step 8: Run Kali and audit all your IPs and ports.Step 9: Use Kali hacking tools to attempt to access all your web ports, shells and other

Protect Your Cloud Big Data AssetsStep 1: Do not put anything into the cloud unless you have a CISO, Chieft Security Architect, Certified Cloud Administrator, full understanding of your PII and private data, a Lawyer to defend you against the coming lawsuits, full understanding of Hadoop, Hadoop Certified Administrators, a Hadoop premier support contract, a security plan, full understanding of your Hadoop architecture and layout. Step 2: Study all running services in Ambari. Step 3: Confirm and check all of your TCP/IP ports. Hadoop has a lot of them!Step 4: if you are not using a service, do not run it.Step 5: By default, disable all access to everything, always. Only open ports and access when something and someone critical cannot access them. Step 6: SSL, SSH, VPN and Encryption Everywhere.Step 7: Run Knox! Set it up correctly.Step 8: Run Kali and audit all your IPs and ports.Step 9: Use Kali hacking tools to attempt to access all your web ports, shells and othe

Objective This tutorial is intended to walk you through the process of troubleshooting why events are missing from the Metron Kibana dashboard when using the quick-dev-platform. Because of the constrained resources and added complexity of the virtualized environment, sometimes the vagrant instance doesn't come up in a healthy state. Prerequisites You should have already cloned the Metron git repo. Metron You should have already deployed the Metron quick-dev-platform. quick-dev-platform The expectation is you have already started Metron once and shut it down via a vagrant halt. Now you are trying to start Metron again via vagrant up, but things don't seem to be working properly. These troubleshooting steps are also be helpful during the initial start of the VM via the run.sh script. Scope Theses troubleshooting steps were performed in the following environment: Mac OS X 10.11.6 Vagrant 1.8.6 VirtualBox 5.1.6 Python 2.7.12 (Anaconda distribution) Steps Start Metron quick-dev-platform

The Apache Phoenix QueryServer (PQS) is a relatively new (early 2015) component that has been added. Previously, users of Phoenix had at least two things in common: the client was written in Java and the Phoenix client libraries were included in that client. As Phoenix continues to grow as a dominant player in the "Hadoop OLTP" space, the need to eliminate these two burdens became apparent. PQS was the solution chosen to address these problems.PQS specifically refers to an HTTP server which can interact with a custom JDBC driver that translates JDBC API calls to HTTP requests. These components are provided by an upstream Apache project, Apache Calcite, specifically the Calcite component referred to as Avatica. Originally, the HTTP requests were JSON; however recent work (mid-2015) has also shown that serialization using Protocol Buffers is also feasible.The excitement behind PQS is two-fold as alluded. Java-based clients now have two means of running queries: a typical JDBC client with