Resources

Often lines of business, individual users or shared teams will use online Google Sheets to share spreadsheet and tabular data amongst teams or without outside vendors. It's quick and easy to add sheets and store your data in Google Drive as spreadsheets.Often you will want to consolidate, federate, analyze, enrich and use this data for reporting and dashboards throughout your organization.An easy way to do that is to read in the data using Google's Sheet API. This is a standard SSL HTTP REST API that returns clean JSON data. I created a simple Google Sheet to test ingesting a Google Sheet with HDF. You will need to enable Google Sheets API in the Google APIs Console. You must be logged into Google and have a Google Account (use the one where you created your Google Spreadsheets).Google DocumentationGoogle provides a few Quick starts that you can use to ingest this data: https://developers.google.com/sheets/quickstart/jsor https://developers.google.com/sheets/quickstart/python.

This topic describes installing and configuring Splice Machine on a Hortonworks Ambari-managed cluster. Follow these steps: Verify Prerequisites Download and Install Splice Machine Configure Hadoop Services Make any needed Optional Configuration Modifications Restart the Cluster Verify your Splice Machine InstallationVerify PrerequisitesBefore starting your Splice Machine installation, please make sure that your cluster contains the prerequisite software components:A cluster running HDP 2.4.2HBase installedHDFS installedYARN installedZooKeeper installedDownload and Install Splice MachinePerform the following steps on each node in your cluster:1.Create the splice installation directory:sudo mkdir -p /opt/splice2.Download the Splice Machine package into the splice directory on the node:sudo curl 'https://s3.amazonaws.com/20snapshot/installer/2.0.1.23/hdp2.4.2/SPLICEMACHINE-2.0.1.23-SNAPSHOT.hdp2.4.2.p0.108.tar.gz' -o /opt/splice/SPLICEMACHINE-2.0.1.23-SNAPSHOT.hdp2.4.2.p0.108

As part of a live Drone ingest, I want to identify what in the image. The metadata provides a ton of information on GPS, altitude and image characteristics, but not what's in the image.IBM, Microsoft and Google all have APIs that do a good job of this and they for the most part of "free" tiers. I wanted to run something locally using libraries installed on my cluster. For my first option, I used TensorFlow Inception-v3 Image Recognition. In future articles I will cover PaddlePaddle, OpenCV and some other Deep Learning and non-deep learning options for Image Recognition. Also I will show the entire Drone to Front-End flow including Phoenix, Spring Boot, Zeppelin, LeafletJS and more. This will be done as part of a meetup presentation with a certified drone pilot.To Run My TensorFlow Binary From HDF 2.0 I use the ExecuteStreamCommand to run a shell script containing the information below:source /usr/local/lib/bazel/bin/bazel-complete.bashexport JAVA_HOME=/opt/jdk1.8.0_101/hdfs df

If you haven't already done so first download the Sandbox from the Hortonworks Website and import it to Virtualbox/VMware. Once you start the Virtual Machine you are directed to the landing page for your new sandbox. http://localhost:8888/Here you have 2 Paths to choose from. If you are completely new to HDP I encourage you to browse the "New to HDP" section. Click on Advanced HDP to get to the links for the components we will be using in this tutorial. If you look at the Atlas link you see the Note: Off by default. Enable by logging in as raj_ops into AmbariYou will need to start in Ambari as user ... raj_opsGo ahead and click the link for Ambari and login. If you see a page that looks like this you are doing great! Atlas has dependencies on other Apache projects. We need to make sure HDFS, Hbase, Kafka and Ambari Infra are all started. Click on each service and then under the service actions drop down select start. After verifying all components are up and running go ahead and follo

Enable a Ranger Plugin and audit to HDFS for a Hadoop component say in this case HiveServer2.Audit files will be stored in folder structure defined in the audit configuration file of the respective ranger plugin. Default format is /ranger/audit/<hadoop-component>/<YYYYMMDD>/<component>_ranger_audit_<hosname>.<count>.logExample audit log for HiveServer2 ranger plugin:hdfs dfs -ls -R /ranger/audit/hiveServer2 /ranger/audit/hiveServer2/20160315/ranger/audit/hiveServer2/20160315/hive_ranger_audit_.1.log/ranger/audit/hiveServer2/20160315/hive_ranger_audit_.2.log/ranger/audit/hiveServer2/20160316/ranger/audit/hiveServer2/20160316/hive_ranger_audit_.1.log/ranger/audit/hiveServer2/20160316/hive_ranger_audit_.2.log/ranger/audit/hiveServer2/20160317/ranger/audit/hiveServer2/20160317/hive_ranger_audit_.1.log/ranger/audit/hiveServer2/20160317/hive_ranger_audit_.2.logProcedure to Create Store Ranger Audit Log in HIVE Create a Hive External table with a dummy location

In HDF 2.0, administrators can secure access to individual NiFi components in order to support multi-tenant authorization. This provides organizations the ability to create least privilege policies for distinct groups of users. For example, let's imagine we have a NiFi Team and a Hadoop Team at our company and the Hadoop Team can only access dataflows they've created, whereas the NiFi Team can access all dataflows. NiFi 1.0 in HDF 2.0 can use different authorizers, such as file-based policies (managed within NiFi) and Ranger-based policies (managed within Ranger), as well as custom, pluggable authorizers. In this example, we'll use Ranger. For more detail on configuring Ranger as the authorizer for NiFi, please see this article. To separate the different teams' dataflows, we'll create separate process groups for each team. In NiFi, access policies are inheritable, supporting simpler policy management with the flexibility of overriding access at the component level. This means tha

Administrator Operations The operations described in this section require superuser privileges. Allow Snapshots: Allowing snapshots of a directory to be created. If the operation completes successfully, the directory becomes snapshottable. Command: hadoop dfsadmin -allowSnapshot $path Arguments: path – The path of the snapshottable directory. See also the corresponding Java API void allowSnapshot(Path path) in HdfsAdmin. Disallow Snapshots: Disallowing snapshots of a directory to be created. All snapshots of the directory must be deleted before disallowing snapshots. Command: hadoop dfsadmin -disallowSnapshot $path Arguments: path – The path of the snapshottable directory. See also the corresponding Java API void disallowSnapshot(Path path) in HdfsAdmin. User Operations The section describes user operations. Note that HDFS superuser can perform all the operations without satisfying the permiss

HDFS Snapshots Overview HDFS Snapshots are read-only point-in-time copies of the file system. Snapshots can be taken on a subtree of the file system or the entire file system. Some common use cases of snapshots are data backup, protection against user errors and disaster recovery. The implementation of HDFS Snapshots is efficient in the following ways: 1) Snapshot creation is instantaneous. The cost is O(1) excluding the inode lookup time. 2) Additional memory is used only when modifications are made relative to a snapshot. Memory usage is O(M), where M is the number of modified files/directories. 3) Blocks in datanodes are not copied. The snapshot files record the block list and the file size. 4) Snapshots do not adversely affect regular HDFS operations, and there is a minor performance impact from accessing snapshotted data depending on the number of modifications. The snapshot data is computed by subtracting the modifications from the current dat

Objective Given the limited resources available in a virtualized sandbox, you may choose to turn specific services on or off. You may choose to enable or disable security, such as Kerberos. Depending on your scenario, you may have a need to switch between these configurations frequently. For reproducible demos, you likely do not want to make these changes between one demo and the next. If you are like me, you may want to have different copies of HDP sandboxes to cover different demo scenarios. With VirtualBox or VMWare sandboxes, you can easily import or clone a sandbox to have multiple, distinct copies. Each copy is unique with no sharing of configuration or data. However, this approach is not quite as intuitive when using the Docker sandbox. If you tried to create multiple containers on a Docker image thinking they would be separate copies, you likely have found they are not completely separate! This tutorial will guide you through the process of using a single sandbox image, with

Do the UpgradePreparation ● For each target cluster find out as much details as possible by reviewing the following: ● Hardware configuration, operating systems and network topology ● Current deployment, cluster topology, configuration of each component ● Security configuration ● User access and management ● Current data ingestion process (if applicable) ● Current running applications and clients connecting into the cluster ● Find the applicable Upgrade Guide from docs.hortonworks.com ● Select validation applications ● Prepare an Upgrade Log to keep track of any upgrade issues and their workarounds ● [Optional but recommended] Prepare one or more Lab (virtual) clusters and install the current HDP stack and Ambari. Use these clusters for mock upgrades and rollbacks to troubleshoot any upgrade issues.Upgrading Clusters ● To upgrade a single cluster use the Upgrade Procedure given below. ● [Optional but recommended] Mock (lab) cluster upgrade: Attempt an upgrade on a Lab

Plan and Assess This is a purely planning step. The expected deliverable is an Upgrade Plan. Gather all details about existing environment to plan for the upgrade path and associated upgrade tasks. 1) Determine Upgrade Path Based on the current and target version of the HDP stack, and whether Ambari is used or not, select the supported upgrade guide from Hortonworks documentation site. Identify key requirement if Namenode HA or other HA needs to be disabled or Security needs to be disabled. Current version: ● HDP Stack version ● Ambari version (if Ambari is used) ● OS Version Target version: ● HDP Stack version ● Ambari version (if Ambari is used) Below are some useful links HDP Stacks Managed by Different Ambari Versions: https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.0.0/bk_ambari-installation/content/determine_stack_compatibility.html Upgrading to Ambari 2.4: https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.0.1/bk_ambari-upgrade/c

With NiFi 1.00 I ingested a lot of image data from drones, mostly to get metadata like geo. I also ingested a resized version of the image in case I wanted to use it. I found a use for it.I am pulling this out very simply with Spring for a simple HTML page. So I wrote a quick Java program to pull out fields I stored in Phoenix (from the metadata) and I wanted to display the image. I could have streamed it out of HDFS using HDFS libraries to read the file and then stream it. sql = "select datekey, fileName, gPSAltitude, gPSLatitude, gPSLongitude, orientation,geolat,geolong,inception from dronedata1 order by datekey asc";out.append(STATIC_HEADER);PreparedStatement ps = connection.prepareStatement(sql);ResultSet res = ps.executeQuery();while (res.next()) { try { out.append("<br><br>\n<table width=100%><tr><td valign=top><img src=\""); out.append("http://tspanndev10.field.hortonworks.com:50070/webhdfs/v1/drone/"). append(res.getSt

In the previous articles, we showed how to deploy an HDF 2.x/3.0 cluster, enable SSL for Nifi and setup the Ranger Nifi plugin. Here we will build on the same cluster and show how to enable kerberos using Active Directory.Summary To achieve this, the high level steps we will follow are: Setup certificate trust for HDF nodesRun Ambari security wizardCreate Ranger policy for nifiadmin userDelete certificateLogin to Nifi using AD principal credentialsPre-requisites You have correctly setup AD as described here Active Directory setup with domain: CLOUD.HORTONWORKS.COMAD already preconfigured with LDAPSCertificate (.crt) used to enable LDAPS is availableOU created where HDF principals will be createdhadoop user has permission to write principals to above OUnifiadmin user created in AD (optionally synced over to Ranger) Test to ensure you can access AD over LDAPS using hadoopadmin user succeeds:ldapsearch -H ldaps://sme-security-ad03.cloud.hortonworks.com:636 -D hadoopadmin@cloud.hortonworks

1. Acquire an EDI File (GetFile, GetFTP, GetHTTP, GetSFTP, Fetch...) 2. Install open source nifi-edireader on NIFI 1.0.0 Download https://github.com/BerryWorksSoftware/edireader Maven Install Berry Works EDIReader Download https://github.com/mrcsparker/nifi-edireader-bundle Maven packge nifi-edireader (must be Maven 3.3 or newer - may have to download and install separately from standard linux package) cp nifi-edireader-nar/target/nifi-edireader-nar-0.0.1.nar to your NIFI/lib Restart NiFi Service 3. Add EdiXML Processor and connect from EDI File input 4. Add extra processing, conversion or routing (TransformXML with XSLT or EvaluateXPATH) to convert to JSON 5. Land to HDFS (PutHDFS) 6. Used the Web Form linked below to generate a test EDI file. ISA*00* *00* *ZZ*SENDER ID *ZZ*RECEIVER ID *010101*0101*U*00401*000000001*0*T*!GS*IN*SENDER ID*APP RECEIVER*01010101*01010101*1*X*004010ST*810*0001BIG*20021208*00001**A999N1*ST*Timothy Spann*9*122334455N3*115 xxx av

We recently had some issues with Ambari and Kafka alerts. It all started when in HDP 2.3.4, every time we changed the kafka listener port from 6667 to any other port, Ambari would complain and give us an error saying that it couldn’t reach port 6667 even though the broker service is actually running in another port. Here’s the exact error: “Connection failed: [Errno 111] Connection refused to sandbox.hortonworks.com:6667” This can be quite annoying especially if you have multiple brokers and they’re all reporting CRITICAL and you just can’t seem to get rid of it. To cut the long story short, here are the steps we did to get rid of the problem. In the section below, we’ll jump into some troubleshooting tips: 1. Get the ID of the kafka broker > curl -u admin:admin -H 'X-Requested-By: ambari' -X GET "http://localhost:8080/api/v1/clusters/Sandbox/alert_definitions" 2. Get the definitions and save it locally > curl -u admin:admin -H 'X-Requested-By: ambari' -X GET "

If you've received the error exitCode=7 after enabling Kerberos, you are hitting this Jira bug. Notice the bug outlines the issue but does not outline a solution. The good news is the solution is simple, as I'll document below.Problem:If you've enabled Kerberos through Ambari, you'll get through around 90-95% of the last step "Start and Test Services" and then receive the error:16/09/26 23:42:49 INFO mapreduce.Job: Running job: job_1474928865338_002216/09/26 23:42:55 INFO mapreduce.Job: Job job_1474928865338_0022 running in uber mode : false16/09/26 23:42:55 INFO mapreduce.Job: map 0% reduce 0%16/09/26 23:42:55 INFO mapreduce.Job: Job job_1474928865338_0022 failed with state FAILED due to: Application application_1474928865338_0022 failed 2 times due to AM Container for appattempt_1474928865338_0022_000002 exited with exitCode: 7For more detailed output, check application tracking page:http://master2.fqdn.com:8088/cluster/app/application_1474928865338_0022Then, click on links to lo

Introduction This article will address some of the most least documented issues you may find while importing data using oozie-sqoop-hcatalog and respective solutions. Heart beat, follow by read-timed-out with thrift server SYMPTOM While trying to execute an oozie workflow, on the stdout of the oozie sqoop job you can see: hive.metastore (HiveMetaStoreClient.java:open(382)) - Trying to connect to metastore with URI thrift://host:port Heart beat Heart beat Heart beat ERROR [main] hive.log (MetaStoreUtils.java:logAndThrowMetaException(1221)) - Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed outorg.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out ROOT CAUSE Missing hive-site.xml for the sqoop action caused the errorRESOLUTION Save the hive-site.xml on the hdfs and reference from the sqoop action file tag: <file>${nameNode}/user/${wf:user()}/${appRoot}/env/${cluster}/hiv

Most user access control is accomplished on the group level. One of the example mentioned in Ranger release is to control each doctor only able to view the patients of his/her own. However most example provided up-to-date are mostly static filter condition that will not change for individual user. In this tutorial, we would demonstrate how to create a row level filter that applies to a group but takes different effect on the user level by leveraging UDF. The use case is as following: a patient table contains patient information as well as the id of the doctor treating the patient (this is a simplified example, in really world the relation may be a many-to-many relation, but it could a handled by the same approach). We also have a doctor reference table where the doctor login credential could be found. We will use the login credential to identify the doctor and only show the patient that associated with the doctor.To setup the environment, first download the HDP2.5 Sandbox form Hortonwo

I ran the same flow myself and examined the AVRO file in HDFS using AVRO Cli.Even though I didn't specify SNAPPY compression, it was there in the file.java -jar avro-tools-1.8.0.jar getmeta 23568764174290.avrolog4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).log4j:WARN Please initialize the log4j system properly.log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.avro.codec snappyavro.schema {"type":"record","name":"people","doc":"Schema generated by Kite","fields":[{"name":"id","type":"long","doc":"Type inferred from '2'"},{"name":"first_name","type":"string","doc":"Type inferred from 'Gregory'"},{"name":"last_name","type":"string","doc":"Type inferred from 'Vasquez'"},{"name":"email","type":"string","doc":"Type inferred from 'gvasquez1@pcworld.com'"},{"name":"gender","type":"string","doc":"Type inferred from 'Male'"},{"name":"ip_address","type":"string","doc":"Type inferred from '32.8.254.252'"},{"n

I had a few hours in the morning before the conference schedule kicked in, so I decided to write a little HDF 2.0 flow to grab all the tweets about Strata Hadoop conference. First up, I used GetTwitter to read tweets and filtered on these terms:strata,stratahadoop,strataconf,NIFI,FutureOfData,ApacheNiFi,Hortonworks,Hadoop,ApacheHive,HBase,ApacheSpark,ApacheTez,MachineLearning,ApachePhoenix,ApacheCalcite,ApacheStorm,ApacheAtlas,ApacheKnox,Apache Ranger, HDFS, Apache Pig, Accumulo, Apache Flume, Sqoop, Apache FalconInput:InvokeHttp: I used this to download the first image URL from tweets.GetTwitter: This is our primary source of data and the most important. You must have a twitter account, a twitter developer account and create a twitter application. Then you can access the keywords and hashtags above. So far I've ingested 14,211 tweets into Phoenix. This included many times I've shut it down for testing and moving things around. I've had this run live as I've added pieces. I d

In many cases, it's sufficient or even desirable to use self-signed, or company-internal signed certificates to provide TLS encryption, for example between your users' browsers and Ambari and also between Ambari and an LDAP service (LDAPS). If you're not careful with your management of the certificates for each component you're working with, it's possible that you can end up with a broken Ambari views installation. In such cases, when you try to load Ambari views, you might instead see a stack trace which on the surface may appear cryptic, with an error code such as "RA040". If you do see javax.net.ssl.SSLHandshakeException in the trace somewhere, it's likely that your problem lies with the custom trust store in Ambari. Consider this scenario: 1. You set up HTTPS for Ambari using a private CA - e.g. your corporate IT team to sign your certificates so that IT-managed browsers which already have imported the CA certificates do not generate warnings. You used "ambari-server setup-s

There are many situations in which running benchmarks for certain workloads on Apache Phoenix can provide meaningful insight into an installation. Commonly, such a benchmark is very useful to understand the baseline characteristics of a new installation of Apache Phoenix. Alternatively, the ability to re-run the same benchmark after changing a configuration property change is extremely useful in understanding the effect of that change. Many approaches exist to test systems that have a SQL interface, many of them focused on a specific type of workload. The following approaches aim to describe a few benchmarks which users can run on their own and tweak to a workload which makes sense for their cluster.Apache JMeter Automation Apache JMeter is a tool which was initially designed to test web applications; however, it also has the ability to execute SQL queries against some JDBC Driver. JMeter allows us to define queries to execute against a Phoenix table. An instance of JMeter can

One key feature in Apache Atlas is the ability to track data lineage in your Data Lake visually. This allows you to very quickly understand the lifecycle of your data and answer questions about where the data originated from and how it relates to other data in the Data Lake. To Illustrate this we will use our own twitter data to perform sentiment analytics on our tweets in Hive and see how this is reflected in Apache Atlas. By now you should have a working sandbox or HDP environment up and running with Atlas enabled. If not please take a look at the following tutorial to help get you started: Getting Started with Atlas in HDP 2.5First we need to gather the data sets we will use in this tutorial. Log into twitterClick on your account settings at the top rightSelect "Your Twitter Data" from the list on the left side of your screen. Now enter your password and in the "Other Data" section at the bottom select "Twitter Archive" This might take a little time but you will get a link to downlo

The Security Guide was extensively updated to reflect new Ranger and Knox features for HDP 2.5. This new content includes: Ranger-Atlas integration using tag-based services and policies.Tags and policy evaluation.Atlas Ranger Authorization plugin.Ranger-Hive row-filtering and column-masking.Migration script to convert from Audit-to-DB to Audit-to-Solr.Ranger Reports page improvements.Knox SSO.Ranger Usersync enhancements.Ranger High Availability (added to Apache Hadoop High Availability guide).Updates to reflect the new Ambari Infra Solr instance.The Data Governance guide was completely rewritten for HDP 2.5 to reflect new Apache Atlas capabilities, along with a new Atlas UI. This new content includes: Atlas-Ranger integration and Tag synch. Installation and Configuration.Configuring Atlas Security.Configuring Atlas High Availability.Searching and viewing Atlas Assets.Working with Atlas tags.Enabling and Managing the Atlas Business Taxonomy (Technical Preview).Updated REST API resou

HDP 2.3 Index Life Cycle(Client events are in orange)-Columns Names in SYSTEM.CATALOG table:-Query:-“select TABLE_NAME,DATA_TABLE_NAME,INDEX_TYPE,INDEX_STATE,INDEX_DISABLE_TIMESTAMP from system.catalog where INDEX_TYPE is not null;”TABLE_NAME:-Name of the indexDATA_TABLE_NAME:-Name of the parent data table of indexINDEX_TYPE:-GLOBAL(1)LOCAL(2)INDEX_STATE:-Index States (Abbreviation stored in SYSTEM.CATALOG, states in bold are available to client too):-BUILDING("b") USABLE("e")UNUSABLE("d")ACTIVE("a")INACTIVE("i")DISABLE("x")REBUILD("r")(Below paragraph is from phoenix site)DISABLE will cause the no further index maintenance to be performed on the index and it will no longer be considered for use in queries.REBUILD will completely rebuild the index and upon completion will enable the index to be used in queries again. BUILDING will partially rebuild the index from the last disabled timestamp and upon completion.INACTIVE/UNUSABLE will cause the index to no longer be considere