Community Articles

This is a detailed walk-through of configuring a service account in Google Cloud Platform and a cloud credential in Cloudbreak. Once these are done, Cloudbreak can spin up your clusters in GCP easily and quickly.High Level Steps Enable Compute Engine API in GCP projectCreate service account with required roles in GCP projectCreate credential for GCP service account in CloudbreakYou will need a GCP account that you have full rights to administer service accounts and a Cloudbreak instance. The Cloudbreak can be running anywhere, as long as it has network access to GCP. My example is running on an internal OpenStack cluster.Google Cloud Platform ConsoleLog into the GCP Console at https://console.cloud.google.comOn the main dashboard page, you will find the Project ID. You will need this to define your credential in Cloudbbreak in a later step.GCP - APIs DashboardGo to the Service Accounts screen by (1) clicking the menu in the top left, (2) hovering over APIs and Services, and (3) Cl

The two most important aspect of security is1. Authentication 2. AuthorizationAuthentication : the process of ascertaining that somebody really is who he claims to be. (who are you) Authorization : the process of verifying that you can access to something. (Are you allowed to access the resource)Lets take example of a user logging into a Linux machine (ssh / terminal login). One needs to authenticate himself username and password thus verifying he is the person who he claims to be . The same user might not be authorized to access a file as he dosent have enough permissions to read/write on the file.The idea can be extended to even other services like, one can login (authenticate ) into booking.com if he has a profile, but is not authroized to change the prices of the flights, only admins are allowed to do that. Hence authentication and authorization play a key role in determining the security aspects of a service.Lets see How authentication and authorization are implemented in L

ISSUE : In Ambari 2.4.x, Falcon Web UI is inaccessible(HTTP 503 error) and Ambari Service Check for Falcon fails: "ERROR: Unable to initialize Falcon Client object" Snippet of error from falcon logs2017-01-04 14:08:56,390 INFO - [main:] ~ Initializing graph db (MetadataMappingService:143) 2017-01-04 14:08:56,438 ERROR - [main:] ~ Failed startup of context org.mortbay.jetty.webapp.WebAppContext@96a75da{/,/usr/hdp/current/falcon-server/server/webapp/falcon} (log:87) Caused by: org.apache.falcon.FalconException: java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory]. Caused by: java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory]. Caused by: java.lang.reflect.InvocationTargetException Caused by: java.lang.NoClassDefFoundError: com/sleepycat/je/LockMode Caused by: java.lang.ClassNotFoundException: com.sleepycat.je.LockMode 2017-01-04

Installing Hue can be tricky as the latest version Hue 4.1.0 is available only as source code and Compiling it requires a lot of dependencies and development libraries for which there is very little information available on the InternetThe best document that I could find as reference was Hue 4.1.0 Installation Guide, though there are a few packages missing in the docAfter series of trial and error came up with the below steps to Compile / Install Hue successfully Install JDK, Hue dependencies, and development Libraries$ sudo yum -y update$ sudo yum install -y git gcc libxml2-devel libxslt-devel cyrus-sasl-devel cyrus-sasl-gssapi mysql-devel python-devel python-setuptools sqlite-devel ant gcc-c++ *sasl* krb5-devel libtidy openldap-devel wget libffi-devel gmp-devel java-1.8.0-openjdk Install Apache Maven from public Fedora repository$ sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo$ sudo sed -i s/\$releas

BackgroundAfter Kerberos HDP/HDF cluster, some native admin UI will return "401 Authentication required".For example, Ambari Infra Solr Admin UIThis is because the web UI is configured for SPNEGO (Simple and Protected GSSAPI Negotiation Mechanism) to extend Kerberos to HTTP. Client must have a valid Kerberos ticket and send by browser.Mac Kerberos Client ConfigurationKerberos software is installed by default in Mac OS, but need to add configure file to access your KDC server.$ sudo vi /Library/Preferences/edu.mit.Kerberos[domain_realm] .hortonworks.com = "HORTONWORKS.COM" hortonworks.com = "HORTONWORKS.COM"[libdefaults] default_realm = "HORTONWORKS.COM" dns_fallback = "yes" noaddresses = "TRUE"[realms]HORTONWORKS.COM = { admin_server = "ad.hortonworks.com" default_domain = "hortonworks.com" kdc = "ad.hortonworks.com" }Firefox Configurationtype about:configsearch negotiate-authOption1 Command Linecommand line to init a kerberos ticket$ kinit hadoopadmin@HORTONWORKS.COMhadoop

Issue: When running hive shell in a docker you will be getting a message "mbind: Operation not permitted" printed on the console but the operations will pass..Root Cause: mbind syscall is used for NUMA (non-uniform memory access) operations which is blocked by docker by default. But in hive opts there is an option which specifies '+UseNUMA'..Resolution:Go to Ambari -> Hive -> Configs -> Advanced 1) Remove '-XX:+UseNUMA' from 'hive.tez.java.opts'.2) Remove '-XX:+UseNUMA' from hive-env template.Hope this helps 🙂

Lets start with having a quick understanding of a file saved in Linux OS1. vim /etc/fstab to know the filesystem of the disk.2. create and save a file using vim. 3. filefrag -v filename Facts to appreciate. 1. Filefrag commands will give you an idea of how many blocks( OS blocks formed by grouping sectors) the file occupies. block is the minimum data that can be read by the OS . Data is fetched from hard disk in multiples of the blocks. 2. The blocks might not be contiguous( can you corelate something to HDFS).3. The filesystem has no idea about a record saved in the file or the format of the file. For the filesystem file is a sequence of bytes. 4. Number of blocks occupied on Linux FileSystem is filesize/filesystem block size , which mostly is file size/ 4096 for ext3/4. 5. Record is a logical entity which, only the reader and writer of the file and can understand. Filesystem has no clue about the same.6. We have standardized the process of identifying a record with file format

IntroductionThis post enumerates the steps necessary to setup Atlas development environment using IntelliJ on Mac and Windows.This setup uses the BerkeleyDB for backend and embedded Solr as index engine.Setup with other backend and index engine variations are similar but will involve additional setup.PrerequisitesThese should be present on your machine before you begin: Git for cloning repository. The command Git Shell is useful if you are switching between Mac and Windows.Maven for performing command line build.IntelliJ Community Edition or higher.BerkeleyDB as backend.Code Base SetupDownload code base from GitHub location. Clone it under c:\work\Apache\atlas on Window and ~/Apache/atlas on Mac.Change directory to the location above and initiate a build (using mvn clean install package).Deploy Directory SetupCreate a directory say Deploy (say c:\work on Windows or ~/work on Mac) with a structure below it: conf Copy atlas-application.properties, users-credentials.properties, policy-st

Summary: The release of HDF 3.1 brings about a significant number of improvements in HDF: Apache Nifi 1.5, Kafka 1.0, plus the new NiFi registry. In addition, there were improvements to Storm, Streaming Analytics Manager, Schema Registry components. This article shows how you can use ambari-bootstrap project to easily generate a blueprint and deploy HDF clusters to both either single node or development/demo environments in 5 easy steps. To quickly setup a single node setup, a prebuilt AMI is available for AWS as well as a script that automates these steps, so you can deploy the cluster in a few commands. Steps for each of the below option are described in this article:A. Single-node prebuilt AMI on AWSB. Single-node fresh installC. Multi-node fresh install A. Single-node prebuilt AMI on AWS:Steps to launch the AMI 1. Launch Amazon AWS console page in your browser by clicking here and sign in with your credentials. Once signed in, you can close this browser tab. 2. Select

In this article, we will have a walk through of integrating LDAP with NiFi Registry. The precondition for LDAP to work with NiFi-Registry is that SSL need to be enabled. This article also covers the "How To" enable SSL for NiFI Registry. For the sake of simplicity, I am using self signed certificates (JKS, using keytool). Steps for creating self signed certificates are as below :-Generate KeyStorekeytool -genkey -keyalg RSA -validity 3650 -alias <alias_name> -keypass <pwd> -storepass <pwd> -dname "cn=hostname, ou=home, o=ak, c=us" -keystore nifi_reg_keystore.jks Export a certificate with the public keykeytool -export -alias <alias_name> -file nifi_reg.cer -storepass <pwd> -keystore nifi_reg_keystore.jksGenerate TrustStorekeytool -import -noprompt -alias nr-c0 -file nifi_reg.cer -storepass changeitchangeit -keystore nifi_reg_truststore.jks Below, is a representation of the NiFi Registry UI with default http and anonymous user login. Now we will login to Am

From Ambari 2.4, there is a built-in service in Ambari called Ambari Infra, which is used to create infrastructure (Solr) for Ranger, Atlas or Log Search.In case of real time indexing or sub second responses is not a must, Ambari Infra Solr collections can be configured to store their indices on HDFS. In order to achieve that you will need the following steps:1. Create a folder on hdfs with the proper permissions:hdfs dfs -mkdir /user/infra-solrhdfs dfs -chown infra-solr:hdfs /user/infra-solr2. Edit solrconfig.xml of your collection where you want to use HDFS to store the indices, change the following section:<directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"></directoryFactory>to:<directoryFactory name="DirectoryFactory" class="solr.HdfsDirectoryFactory"> <str name="solr.hdfs.home">hdfs://<namenode_host>:8020/user/infra-solr</str> <str name="solr.hdfs.confdir">/etc/hadoop/conf</str

In previous article of the sereies, Adding a New Telemetry Data Source to Apache Metron, we walked through how to add a new data source squid to Apache Metron. The inevitable next question is how I can enrich the telemetry events in real-time as it flows through the platform. Enrichment is critical when identifying threats or as we like to call it "finding the needle in the haystack". The customers requirement are the following The proxy events from Squid logs needs to ingested in real-time.The proxy logs has to be parsed into a standardized JSON structure that Metron can understand.In real-time, the squid proxy event needs to be enriched so that the domain named are enriched with the IP informationIn real-time, the IP with in the proxy event must be checked against for threat intel feeds.If there is a threat intel hit, an alert needs to be raisedThe end user must be able to see the new telemetry events and the alerts from the new data source.All of this requirements will need to be

PROBLEM: Created a Phoenix Table on top of existing Hbase table, the phoenix table is not working as expected or is not needed now, need to delete Phoenix table but not the existing Hbase table as it has valid data.CONSTRAINT : Simple dropping of table from Phoenix shell would delete Hbase table too. SOLUTION : Step 1: Delete Phoenix table entry from SYSTEM.CATALOG as follows :delete from SYSTEM.CATALOG where TABLE_NAME='table_name';Step 2 : Restart the region server which hosts SYSTEM.CATALOG to clear metadata cache which may still not allow creating same named table from Phoenix. Note : This approach may not work well if there are indexes or views on top of this table as we would end up corrupting catalog table.You may also try snapshotting existing Hbase table , dropping the table completely from Phoenix and then cloning / restoring the Hbase table from snapshot.

Short Description Storm Service Check fails with a Kerberos error after performing an Ambari upgrade in a cluster that had Metron installed. ArticleIn a cluster where Metron was removed before upgrading Ambari server, the following error can occur when performing a Service Check in Storm service: Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: The TGT found is not renewable at org.apache.storm.security.auth.kerberos.AutoTGT.populateCredentials(AutoTGT.java:1) at org.apache.storm.StormSubmitter.populateCredentials(StormSubmitter.java:3) at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:748) at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:590) at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:239) at storm.starter.WordCountTopology.main(WordCountTopology.java:492) Caused by: java.lang.RuntimeException: The TGT found is not renewable at org.apache.storm.security.auth.kerberos.AutoTG

Apache NiFi provides a great depth of information about each component on the canvas. Stats are collected for the number of FlowFiles in, the number of FlowFiles out, how many bytes were read or written, how long it took, etc. However, these stats that are displayed here are all "stock metrics." These metrics are the same for each processor. There are times when we may want to expose other metrics for a custom processor.One way that we can do this in NiFi is through the use of counters. The counters are a lesser-known feature of NiFi, and until recently have been pretty limited in their usefulness. This is due to the fact that upon startup, NiFi sets all counters to 0 and as the system runs, these counters are monotonically increasing. They count how many times something has happened since NiFi started. To see these counters, we can go to the drop-down menu in the top-right corner of the NiFi UI and click "Counters."We then are provided with a list of counters that have been registered

PROBLEM: There are multiple issues connecting Windows clients such as Squirrel client to Phoenix / Hbase on a cluster enabled with Kerberos. SOLUTION: Please follow below working steps. 1. Create Jaas configuration file hbase.jaas on client machine.   Client { com.sun.security.auth.module.Krb5LoginModule required   useKeyTab=false useTicketCache=true   renewTicket=true serviceName="zookeeper"; keytab="" principal="" }; 2. Copy kerberos config file from cluster nodes to local client machine (krb5.conf/krb5.ini) 3. Add below java options in Squirrel launcher script. [/Applications/SQuirreLSQL.app/Contents/MacOS/squirrel-sql.sh ] -Djava.security.auth.login.config="/Users/gsharma/hbase/hbase.jaas" -Djava.security.krb5.conf="/Users/gsharma/hbase/krb5.conf" e.g $JAVACMD -Xmx256m -cp "$CP" -Djava.security.auth.login.config="/Users/gsharma/hbase/hbase.jaas" -Djava.security.krb5.conf="/Users/gsharma/hbase/krb5.conf" $MACOSX_SQUIRREL_PROPS -splash:"$SQUIRREL_SQL_HOME/icons/splash.jpg" n

In this post, we will see how to configure node labels on YARN.Before we go for the configuration part, let’s understand what is node label in YARN.Node labels allows us to divide our cluster in different parts and we can use those parts individually as per our requirements. More specifically, we can create a group of node-managers using node labels, for example group of node managers which are having high amount of RAM and use them to process only critical production jobs! This is cool, isn’t it? So lets see how we can configure node labels on YARN..Types of node labels:Exclusive – In this type of node labels, only associated/mapped queues can access the resources of node label.Non Exclusive(sharable) – If resources are not in use for this node label then it can be shared with other running applications in a cluster..Configuring node labels:.Step 1: Create required directory structure on HDFSNote – You can run below commands from any of the hdfs client.sudo su hdfshadoop fs -mkdir -p

This article describes to store your zeppelin notes in a GitHub repo.1) Create a GitHub repo where you want to store your zeppelin notes. I have created a repo named 'zeppelin-notes'.Select ssh for cloning. We will be using ssh key to perform git operations.2) Copy the ssh public key from the node where Zeppelin is installed and add it in git hub. The command displays the public key and screenshot tells how to add the ssh key to github# su zeppelin# cat ~/.ssh/id_rsa.pubClick on "New SSH Key" , give a title for the key and paste the content and click save.3) Clone the git repo in the node where zeppelin is installed and set proper permissions to the foldersu zeppelincd /usr/hdp/current/zeppelin-server/git clone git@github.com:cvr-aditya/zeppelin-notes.git chown zeppelin:zeppelin zeppelin-noteschmod -R 777 zeppelin-notes4) Change the zeppelin notebook directory path. Go to Ambari -> Zeppelin -> Configs -> Advanced zeppelin-config and set the value for zeppelin.notebook.dir to t

resource_management.core.exceptions.Fail: Failed to download file from http://ambari1.hdp.hadoop:8080/resources//mysql-connector-java.jar due to HTTP error: HTTP Error 404: Not FoundBy the following command, the problem is fixed:Ambari Server:(Ref: https://discuss.pivotal.io/hc/en-us/articles/115001611807-Hive-Services-Fail-to-Start-giving-Error-HTTP-Error-404-Not-Found- )sudo yum install mysql-connector-java*ls -al /usr/share/java/mysql-connector-java.jarcd /var/lib/ambari-server/resources/ln -s /usr/share/java/mysql-connector-java.jar mysql-connector-java.jar

For 2018, some awesome new in-demand features have come to my favorite Swiss Army Knife of IoT and Enterprise development, Apache NiFi. Speaking of knives, for fun say, “Apache NiFi” to Google Assistant. Okay, back to the awesome new release of Apache NiFi.So there are a couple of new Processors that I want to highlight first. I am liking the new CountText processor, useful for counting elements of text documents like words and lines. My example flow is using it and I see some useful metrics gathering there. I also think some of these could be used as file validation checks to feed to machine learning algorithms. My files of type X are usually of this # of lines and words, but not this time. I have come across a couple of use cases on file ingest in the past that could use this. In one example a company was reading personnel files from an sFTP. The first step in validation was checking to see they received the proper number of lines, since one person per line. Another occasio

One of the really cool features of the new Apache NiFi Registry UI are the deep links. This capability enhances collaboration by providing shareable URL’s to various perspectives within the NiFi Registry web application. The following deep links are available within the ‘administration’ perspective:/nifi-registry/administration/workflow – This ‘administration’ perspective displays a sortable table of all buckets the current user is authorized to view. /nifi-registry/administration/workflow(sidenav:manage/bucket/:bucketId) – This administration’ perspective side panel displays the requested bucket’s details and a sortable table of all of the policies created for this :bucketId./nifi-registry/administration/users ** – This ‘administration’ perspective displays a sortable table of all users and groups the current user is authorized to view. /nifi-registry/administration/users(sidenav:manage/user/:userId) ** – This administration’ perspective side panel displays the requested user’s deta

The HDF 3.1 release supports single-sign on to NiFi using KnoxSSO. This article will assume you've already completed setting up KnoxSSO, as discussed in part I and part II of this series.We'll further assume that NiFi has been configured for baseline security, as documented in this HCC article.Once the websso topology has been defined in your Knox configuration, the steps to make NiFi a participating application in KnoxSSO are straightforward. A couple of notes: 1) make sure nifi.security.user.login.identity.provider is blank, since you'll be using the KnoxSSO topology's authentication method--i.e., a JWT-based Federation Provider--to gain access to the NiFi UI, and 2) make sure the value for knoxsso.token.ttl is reasonable, the default is 30000 ms, or 30 s. A larger value like 36000000--or 10 hours--likely makes sense for production environments.Once the websso topology has been defined for KnoxSSO, as discussed in part I and part II of this series, we'll need to grab the Knox server'

Hello,I've compiled this article after I struggled quiet a bit to have the ingestion working.Credits to @Kuldeep Kulkarni, @Rahul Pathak and @awhitter for helping me figuring out these steps.Feel free to contact me. I'll make sure to keep this up-to-date.Sqoop is the tool of choice for ingesting OLTP data to HDFS of Hive. Incremental ingestion requires to keep track of the last row ingested in order to know where to start the next round from. Using a shared Sqoop metastore and sqoop job will let sqoop take care of keeping track of the ingestion.Oozie will be setup to call the sqoop job at the required interval of time. For this a coordinator will be required,1 Sqoop1a. Shared MetastoreCreate a Sqoop metastore on a node (preferably not on a data/worker node). - Create a folder to save the metastore DB file (e.g. "/var/lib/sqoop/metastore)- configure the Metastore from Ambari > Sqoop > Configssqoop.metastore.server.location "/var/lib/sqoop/metastore/shared.db"sqoop.metastore.c

Apache Deep Learning 101 SeriesThis is for people preparing to attend my talk on Deep Learning at DataWorks Summit Berling 2018 (https://dataworkssummit.com/berlin-2018/#agenda) on Thursday April 19, 2018 at 11:50AM Berlin time.You can easily run Apache MXNet on an OSX machine or a Linux workstation utilizing a Python script. I have forked the standard Apache MXNet Wine Detector Tutorial (http://mxnet.incubator.apache.org/tutorials/embedded/wine_detector.html) to read our local OSX webcam (you may need to change your OpenCV WebCam port from 0 to 1 or to 2, depending on your number of webcams and which one you want to use. I am running this on an OSX laptop connected to a monitor that has a built in webcam, so I use that one which is 1. The webcam numbering starts at 0. If you only have one, then use 0.Let's get this installed!git clone https://github.com/apache/incubator-mxnet.gitThe installation instructions at Apache MXNet's website (http://mxnet.incubator.apache.org/install/i

Sqoop# Sandbox 2.3.2# using postgres ambari database# edit pg_hba.conf file in /var/lib/pgsql/data# Add the following line as the first line of pg_hba.conf. It allows access to all databases for all users with an encrypted password:********************************# TYPE DATABASE USER CIDR-ADDRESS METHODhost all all 0.0.0.0/0 md5********************************### POSTGRES ###su postgrespsql\c ambari#list all tables\dt ambari.*select * from ambari.hosts;### SQOOP ####check sqoop versionsqoop version# get list of commandssqoop help# more command specificsqoop help import### DEMO #### Download Postgres Drivercurl -L 'http://jdbc.postgresql.org/download/postgresql-9.2-1002.jdbc4.jar' -o postgresql-9.2-1002.jdbc4.jar# put in the sqoop librarysudo cp postgresql-9.2-1002.jdbc4.jar /usr/hdp/current/sqoop-client/lib/# list tables in ambari database / username ambari password bigdatasqoop list-tables --connect jdbc:postgresql://127.0.0.1/ambari --username ambari -P# sqoop ambari hosts table into