Resources

This tutorial is a companion to this article:https://community.hortonworks.com/articles/72182/bring-spark-to-the-business-with-fine-grain-securi.htmlThe article outlines the use cases and potential benefits to the business that Spark fine grain security with LLAP may yield. This article also has a second part that covers how to apply Tag based security for Spark using Ranger and Atlas in combination.Tag Based (Meta Data) Security for Apache Spark with LLAP, Atlas, and RangerGetting StartedInstall an HDP 2.5.3 Cluster via Ambari. Make sure the following components are installed: HiveSparkSpark Thrift ServerAmbari InfraRanger Enable LLAPNavigate to the Hive Configuration Page and click Enable Interactive Query. Ambari will ask what host group to put the Hiveserver2 service into. Select the Host Group with the most available resources.With Interactive Query enabled, Ambari will display new configurations options. These options provide control of resource allocation for the LLAP service.

To secure the Spark Thrift server first we need to change the mode from binary to http then secure the channel with the certificates.Login to Ambari-> Spark(2)-> Configs -> Custom spark-hive-site-override:Set the following parameters :hive.server2.transport.mode : http hive.server2.thrift.http.port : 10015 / 10016 ( in case of spark 2)hive.server2.http.endpoint : cliservice#Enabling the SSL modehive.server2.use.SSL : true hive.server2.keystore.path : </path/to/your/keystore/jks>hive.server2.keystore.password : <keystorepassword>in case of server certs are not available process to create self-signed certs (from Hive Wiki page)Setting up SSL with self-signed certificatesUse the following steps to create and verify self-signed SSL certificates for use with HiveServer2: Create the self-signed certificate and add it to a keystore file using: keytool -genkey -alias example.com -keyalg RSA -keystore keystore.jks -keysize 2048 Ensure the name used in the

The KnoxShell release artifact provides a small footprint client environment that removes all unnecessary server dependencies, configuration, binary scripts, etc. It is comprised of a couple different things that empower different sorts of users. A set of SDK type classes for providing access to Hadoop resources over HTTPA Groovy based DSL for scripting access to Hadoop resources based on the underlying SDK classesToken based Sessions to provide a CLI SSO session for executing multiple scriptWhile testing the KnoxShell examples for the 0.14.0 Apache Knox release, I realized that using the KnoxShell for access to HiveServer2 was not easily done.This is due to the fact that we are leveraging the knoxshell executable jar which makes is difficult to add additional classes and jars to the classpath for the executing script.I needed to create a launch script that called the main class of the executable jar while also being able to set the classpath with additional jars for Apache Hive client

An example of using KnoxShell to develop a simple SQL client for access to HiveServer2 through Apache Knox Gateway.

IntroductionMy Apache Knox Dynamic Service Endpoint Discovery article describes some exciting new functionality available in the 0.14.0 release of Apache Knox. The gateway is now able to dynamically determine the endpoint URLs of cluster services to proxy from Ambari. The associated benefits are described in that article.Another benefit of this new functionality, which is not mentioned in that article, is the added ability to dynamically respond to cluster configuration changes that affect generated Knox topologies by re-generating and re-deploying those topologies. Without this, deployed topologies can be easily disabled when any of the proxied Hadoop services' configuration changes in the cluster. Cluster MonitoringWhen Knox deploys a simple topology descriptor, and generates a corresponding topology based on discovered cluster configuration details, it subsequently has the ability to monitor that cluster configuration for changes. When it discovers a change, it updates all of i

In this article we will go over how we can use nifi to ingest PDFs and while we ingest we will use a custom groovy script with executescript processor to extract images from this PDF. The images will be tagged with PDF filename, page num and imagenum , so that it can be indexed in hbase/solr for quick searching and doing some machine learning or other analytics. The Groovy code below depends on the following jars. Download and copy them to a folder on your computer. I my case they were under /var/pdimagelib/. pdfbox-2.0.7.jar fontbox-2.0.7.jar jai_imageio.jar commons-logging-1.1.1.jar Copy the code below to a file on your computer, in my case the file was under /var/scripts/pdimage.groovyimport java.nio.charset.*;import org.apache.commons.io.IOUtils;import java.awt.image.BufferedImage;import java.awt.image.RenderedImage;import java.io.File;import java.io.FileOutputStream;import java.util.Iterator;import java.util.List;import java.util.ArrayList;import java.util.Map;import javax.i

Using Hive in Oozie can be challenging. There are two available actionsHiveAction and Hive2ActionTh Hive action uses the hive client and needs to set a lot of libraries and connections. I ran into a lot of issues especially related to security. Also the logs are not available in the Hive server and hive server settings are not honoured.The Hive2 action is a solution for this. It runs a beeline command and connects through jdbc to the hive server. The below assumes that you have used LDAP or PAM security for your hive server. <action name="myhiveaction"> <hive2 xmlns="uri:oozie:hive2-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <jdbc-url>jdbc:hive2://server:10000/default</jdbc-url> <password>${hivepassword}</password> <script>/data/sql/dayly.sql</script> <param>database=${database}</param> <param>day=${day}</param> </hive2&

This article list down the steps involved to enable SSL for Livy Server in HDP 2.6.1.0This setup instruction is for example purpose. Production cluster setup may be different.Prerequisite:Firstly, prepare the keystore and truststore files for enabling SSL in Livy server. The steps followed to generate keystore and truststores are for testing purpose only. 1) create keystore files, certificates, and truststore files for each host in cluster. Generate Keystore file for Livy serverkeytool -genkey -alias <host> -keyalg RSA -keysize 1024 –dname CN=<host>,OU=hw,O=hw,L=paloalto,ST=ca,C=us –keypass <keyPassword> -keystore <keystore_file> -storepass <storePassword> Create a certificatekeytool -export -alias <host> -keystore <keystore_file> -rfc –file <cert_file> -storepass <StorePassword> Create Truststore filekeytool -import -noprompt -alias <host> -file <cert_file> -keystore <truststore_file> -storepass <truststorePas

This article provides a step-by-step guide on how to leverage the Workflow Manager (WFM) View within Apache Ambari to connect, automate, and monitor big data processing tasks. It explains how to build a defined workflow engine to schedule and execute Apache Hive queries, integrating seamlessly with the Hortonworks Data Platform (HDP) alongside other Hadoop ecosystem jobs like Sqoop, Pig, MapReduce, and Spark. More details in the attached PDF.

Setting Up a Data Science Platform on HDP using AnacondaBuilding a Data Science Platform using Anaconda needs to be able to Launch PySpark jobs on the clusterSynchronize python libraries from vetted public repositoriesIsolate environments with specific dependencies to run production jobs using an older version of a package whilst simultaneously running new version of the packageLaunching notebooks and PySpark jobs using different kernels such as Python_2.7, Python_3.x, R, ScalaFramework of the Data Science Platform Private Repo ServerEdge NodesDevTestProdAnsibleGitJenkinsBuilding blocks of the Data Science Platform AnacondaAnsibleGitJenkins

In Apache NIFI, flow configuration allows users to reference environmental specific or system variables using Expression Language (EL). Prior to NIFI 1.4 release, a file-based variable registry was available. However, the management of those variables can be challenging due to the lack of a user interface. Additionally, those variables in the file based variable registry cannot be updated at NIFI runtime, hence breaking the continuous processing whenever NIFI needs to be restarted to update variables.In Apache NIFI 1.4, a UI driven variable registry was made available. This would help simplify the configuration management of flows in flow migration use cases across environments (works great in conjunction with the Flow Registry that is being discussed/worked on in the community).More importantly, similar to how controller services are handled by the NIFI framework, variables are now persisted in the flow definition file (flow.xml). There is a ‘variable' section nested under each proces

Modern Web-Browsers come with few inbuilt defenses for common web attacks but we need to enable our web applications to use them. Recently support for many such HTTP response headers were added to Zeppelin to thwart common attacks like Cross-site scripting, ClickJacking, Man-in-the-Middle and SSL Downgrade attacks which Browsers can use to enable client-side security features.We need to configure the properties in zeppelin-site.xml listed below to enable the supported security headers. 1. The "zeppelin.server.xxss.protection" property needs to be updated in the zeppelin-site.xml in order to set X-XSS-PROTECTION header. The HTTP X-XSS-Protection response header is a feature of Internet Explorer, Chrome and Safari that stops pages from loading when they detect reflected cross-site scripting (XSS) attacks. When value is set to "1; mode=block", the browser will enable XSS filtering and prevent rendering of the page if an attack is detected.When value is set to "0", it turns off the protect

ISSUEWhile trying to connect to the HBase cluster from an edge node or a client HBase API we get the expection "Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations " SYMPTOM The exact stack that we encounter is as below:log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:312) at org.apache.hadoop.hbase.client.ScannerCallableWi

Below are the steps to run hive query in a shell script using Oozie shell action .1. Configure job.propertiesExample:#*************************************************# job.properties#*************************************************nameNode=hdfs://<namenode-fqdn>:8020jobTracker=<resourcemanager-host-fqdn>:8050queueName=defaultexamplesRoot=examplesoozie.use.system.libpath=trueoozie.wf.application.path=${nameNode}/user/${user.name}/.2. Configure Workflow.xmlExample:<workflow-app xmlns="uri:oozie:workflow:0.3" name="shell-wf"><credentials> <credential name='my_auth' type='hcat'> <property> <name>hcat.metastore.uri</name> <value>thrift://<hive-metastore-hostname>:9083</value> </property> <property> <name>hcat.metastore.principal</name> <value>hive/_HOST@HWX.COM</value> </property> </credential></credentials> <start to="shell-node"

IntroductionThe 0.14.0 release of Apache Knox includes the ability to dynamically determine topology endpoints for Hadoop services in Ambari-managed clusters. Prior to this release, users had to determine each of these endpoint URLs by navigating the Ambari UI (or combing through the various cluster configuration files), and explicitly add them to their topology descriptors; there was a lot of potential for human error.Support for a new, simplified topology descriptor has been added to leverage this dynamic endpoint discovery and facilitate provider configuration sharing across topologies.This is a dramatic improvement in the usability of Knox. Simplified DescriptorsSimplified descriptors are a means to facilitate provider configuration sharing and service endpoint discovery. Rather than editing an XML topology descriptor, it’s now possible to create a simpler descriptor that declaratively specifies the desired contents of a topology, which will ultimately yield a full topology de

Provisioning clusters with MiNiFi C++ Bootstrap. This article references code from https://github.com/apache/nifi-minifi-cpp/pull/218 MiNiFi C++ has a new bootstrap process that enables you to select dependencies with a menu guided approach. In this article, we’ll discuss using the bootstrap script to bypass the menu and install all dependencies that your system is capable of installing. The major advantage of this is that you will be able to run MiNiFI C++, which doesn’t incur a startup cost and can be run within a very small memory footprint, on any device you can connect to using SCP to distribute the source and SSH to run the boot strap command.Overview of the boot strap process. The bootstrap script is a simple set of bash scripts that run the dependency installation, CMAKE setup, and build. You will then use boot strap to run a make install. The bootstrap.sh script normally uses prompts to ensure agreement at every step, but for the purposes of this article, we’ll be supplyin

Ambari server typically gets to know about the service availability from Ambari agent and using the '*.pid' files created in /var/run. Following covers couple of scenarios to troubleshoot:Scenario 1: Ambari Agent is not communicating appropriately with Ambari ServerIf all the services are shown to be down in a given node, then it is most likely an Ambari agent issue.Following steps could be used to troubleshoot Ambari agent issues# ambari-agent status Found ambari-agent PID: 19715 ambari-agent running. Agent PID at: /var/run/ambari-agent/ambari-agent.pid Agent out at: /var/log/ambari-agent/ambari-agent.out Agent log at: /var/log/ambari-agent/ambari-agent.logand check if the pid indeed exist by doing a ps -ef. In case the pid doesn't exist, also run ps -ef | grep 'ambari_agent' to see if a stale process is around. Eg,# ps -ef | grep "ambari_agent" root 18626 13528 0 04:45 pts/0 00:00:00 grep ambari_agent root 19707 1 0 Feb17 ? 00:00:00 /usr/bin/python2 /usr/lib/p

1. Configure the FTP client (Below is a screenshot from Cyberduck)2. Refer the below link for the HDP stack version http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-installation/content/hdp_stack_repositories.htmlFor ex: I'm interested in this repohttp://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.5.0.0/hdp.repo3. Navigate to the above path manually, right click the respective hdp.repo and select HTTPS URL. In the below screen print I have showed the same and you'll download the tar file in HTTPS. Hope this helps!Thanks

Getting started with MiNiFiIn this tutorial, we will learn how configure MiNiFi to send data to NiFi: Installing HDFInstalling MiNiFiSetting up the Flow for NiFiSetting up the Flow for MiNiFiPreparing the flow for MiNiFiConfiguring and starting MiNiFiEnjoying the data flow!ReferencesFor a primer on HDF, you can refer to the tutorials here Tutorials and User GuidesInstalling HDF If you do not have NiFi installed, please follow the instructions found hereNOTE: The above installation guide is for HDF 1.2.0.1, this is the version that matches Apache MiNiFi 0.0.1. Although HDF 2.0 may work, for this exercise -- it is not recommended at this time. Installing MiNiFiNow that you have NiFi up and running it is time to download and install MiNiFi. Open a browser. Download the MiNiFi Binaries from Apache MiNiFi Downloads Page. There are two options: tar.gz a format tailored to Linux and a zip file more compatible with Windows. If you are using a Mac either option is just fine. Figure 1. MiNiFi d

For the Devops crowd, you can take it a step further and automate provisioning of Ambari server. For that consider using community contributions like https://supermarket.chef.io/cookbooks/ambari if you were to use Chef, for example. For an exhaustive tour of the REST API, consult the docs https://github.com/apache/ambari/blob/trunk/ambari-views/docs/index.mdThis recipe assumes you have an unsecured HDP cluster with Namenode HA. Tested on Ambari 2.4.2.# list all available views for the current version of Ambaricurl --user admin:admin -i -H 'X-Requested-By: ambari' -X GET http://localhost:8080/api/v1/views/# get Files View onlycurl --user admin:admin -i -H 'X-Requested-By: ambari' -X GET http://localhost:8080/api/v1/views/FILES# get all versions of Files View for the current Ambari releasecurl --user admin:admin -i -H 'X-Requested-By: ambari' -X GET http://localhost:8080/api/v1/views/FILES/versions# get specific version of FILES view curl --user admin:admin -i -H 'X-Requested-By: ambari'

Building schemas is tedious work and fraught with errors. The InferAvroSchema processor can get you started. It generates a compliant schema for use. There is one caveat, you have to make sure you are using Apache Avro safe field names. I have a custom processor that will clean your attributes if you need them Avro-safe. See processor listed below.Example Flow Utilizing InferAvroSchemaInferAvroSchema DetailsStep 0: Use Apache NiFi to Convert Data to JSON or CSVStep 1: Send JSON or CSV Data to InferAvroSchemaI recommend setting output destination to flowfile-attribute, input content type to json, pretty avro output to true.Step 2: The New schema is now in attribute: inferred.avro.schema.inferred.avro.schema{ "type" : "record", "name" : "schema1", "fields" : [ { "name" : "table", "type" : "string", "doc" : "Type inferred from '\"schema1.tableName\"'" } ] } This schema can then be used for conversions directly or stored in Hortonworks Schema Registry or Apache NiFi Built-

Hello everyone!I am trying to post a question about an issue with Flume in the CDH but it seems the post has been automatically deleted. Can I ask why is this happening?

Overview There are a couple of different options for importing data from Teradata into Hadoop: Sqoop and the Teradata JDBC driver (documentation) Hortonworks Connector for Teradata (documentation) Teradata Connector for Hadoop (TDCH) (download README here or get the PDF doc) This article will explore examples of each of the three above along with some of the pros/cons. The aim of this document is help understand when to use each option while keeping the following in mind: Goal is to create a schema which is as close to Teradata as possibleHive tables will usually end up as ORC but may need intermediate tables stored as TEXTFILE. The examples below use textfile.I have not tested all possible column types. If you have more information, please let me know.Versions tested with: HDP 2.3.2, Sqoop 1.4.6, TDCH 1.4.1 or 1.4.4 All of the examples will use the following common environment variables which you would define at the top of a bash script. The variables are plugged in to the v

In one of the previous articles Extending SAM with Custom Processors & User Defined Functions we saw how SAM can be extended to incorporate any custom processing logic using Custom Processors & User Defined Functions. That is a very useful feature if one is writing application logic from scratch. However, there are cases where such code is already written and one wants to reuse that code. A good example of this is an existing Apache Storm Spout or Bolt that is interacting with a proprietary system. How would one be able to plug in such a Spout or Bolt in SAM? This is what this article will attempt to explain.For people who are more hands on and want to try out adding a Custom source to SAM and run it as part of a SAM application we have created an example at SAM Custom Source. Please follow the instructions in README.md to execute the steps on your existing SAM installation. For people who want to first get in to the details, continue reading on the rest of this article. You

1) Enable librdkafka extensionPlease reference https://github.com/apache/nifi-minifi-cpp/blob/master/README.mdcmake -DENABLE_LIBRDKAFKA ..2) Configure PublishKafka Processor with below propertiesKnown Brokers: A comma-separated list of known Kafka Brokers in the format <host>:<port> which map to kafka configuration bootstrap.serversTopic Name: The Kafka Topic of interestDelivery Guarantee: Specifies the requirement for guaranteeing that a message is sent to Kafka like all/1/0 which map to kafka configuration request.required.acksMax Request Size: Maximum Kafka protocol request message size which map to kafka configuration message.max.bytesRequest Timeout: The ack timeout of the producer request in milliseconds which map to request.timeout.msClient Name: Client Name to use when communicating with Kafka which map to kafka configuration client.idBatch Size: Maximum number of messages batched in one MessageSet which map to kafka configuration batch.num.messagesQueue Buffering M