Resources

IntroductionHBase replication provides with a way of replicating HBase data from one cluster to another by adding the remote Zookeeper quorum as remote peer.Configuration on the clusterFirst of all, it is necessary to set thehbase.replication property to true. Then add the remote peer through hbase shell. The peer id can be any short name:For example:add_peer '1', "hdpdstzk01.machine.domain, hdpdstzk02.machine.domain, hdpdstzk03.machine.domain:2181:/hbase-secure"(If using Kerberos then the right JAAS configuration needs to be used, or it would be required to have the hbase service keytab in the cache to authenticate correctly against Zookeeper through SASL).Configuration on the tablesReplication is set at table and column family level by setting the propertyREPLICATION_SCOPE to ‘1’. The default value that tables get created with if not specified is ‘0’, which means no replication. If applying on already existing tables, then they need to be disabled, then the property added through alt

After completing this tutorial you will understand how to: leverage Spark to infer a schema on a CSV dataset and persist it to Hive without explicitly declaring the DDL deploy the Spark Thrift Server on the Hortonworks Sandbox connect and ODBC tool (Tableau) to the Spark Thrift Server via the Hive ODBC driver, leveraging caching for ad-hoc visualization Assumption 1: It is assumed that you have downloaded and deployed the Hortonworks sandbox, installed the Hive ODBC driver on your host machine, and installed Tableau (or your preferred ODBC-based reporting tool). Assumption 2: Please ensure that your host machine's /etc/hosts file has the appropriate entry mapping sandbox.hortonworks.com to the IP of your sandbox (e.g., 172.16.35.171 sandbox.hortonworks.com sandbox).Deploying the Spark Thrift Server Within Ambari, click on the Hosts tab and then select the sandbox.hortonworks.com node from the list. Now you can click “Add” and choose Spark Thrift Server from the

Automated deployment of a fresh HDP cluster that includes Zeppelin (via blueprints)Background:The Zeppelin Ambari service has been updated and now supports installing latest Zeppelin version (0.5.5) on HDP using blueprints to automate creation of 'data science' cluster. SequenceIQ team have a datascientist blueprint that installs Zeppelin, but based on my conversations with @lpapp, its based on an older version of the Ambari service so does not install the latest or support as many options.Below is a writeup of how to deploy Zeppelin via blueprints. Note that if you already have a cluster running, you should just use the Add service wizard in Ambari to deploy Zeppelin using the steps on the githubPurpose: Sample steps below for installing a 4-node HDP cluster that includes Zeppelin, using Ambari blueprints and Ambari bootstrap scripts (by @Sean Roberts)Pre-reqs: Bring up 4 VMs imaged with RHEL/CentOS 6.5 or later (e.g. called node1-4 in this case). Note that the VMs should not already

First you need to get the ID of the alert definition in your system:curl -H "X-Requested-By: ambari" -u admin:<password> -X GET http://<ambari_host>:<ambari_port>/api/v1/clusters/<cluster_name>/alert_definitions?AlertDefinition/name=ambari_agent_disk_usageGet the alert definition with the id from prevoius step:curl -H "X-Requested-By: ambari" -u admin:<password> -X GET http://<ambari_host>:<ambari_port>/api/v1/clusters/<cluster_name>/alert_definitions/61Set the new thresholds by changing the JSON alert definition:curl -H "X-Requested-By: ambari" -u admin:<password> -i -X PUT -d '{ "AlertDefinition":{ "cluster_name":"xyz", "component_name":"AMBARI_AGENT", "description":"This host-level alert is triggered if the amount of disk space used on a host goes above specific thresholds. The default values are 85% for WARNING and 90% for CRITICAL.", "enabled":true, "id":61, "ignore_host":false, "interval":1

Lab OverviewIn this lab, we will learn to: Configure Solr to store indexes in HDFSCreate a solr cluster of 2 solr instances running on port 8983 and 8984Index documents in HDFS using the Hadoop connectorsUse Solr to search documentsPre-Requisite The lab is designed for the HDP Sandbox. Download the HDP Sandbox here, import into VMWare Fusion and start the VMLABStep 1 - Log into Sandbox After it boots up, find the IP address of the VM and add an entry into your machines hosts file e.g.192.168.191.241 sandbox.hortonworks.com sandbox Connect to the VM via SSH (root/hadoop), correct the /etc/hosts entryssh root@sandbox.hortonworks.com If running on an Ambari installed HDP 2.3 cluster (instead of sandbox), run the below to install HDPsearchyum install -y lucidworks-hdpsearchsudo -u hdfs hadoop fs -mkdir /user/solrsudo -u hdfs hadoop fs -chown solr /user/solr If running on HDP 2.3 sandbox, run belowchown -R solr:solr /opt/lucidworks-hdpsearch Run remaining steps as solrsu solrStep 2 - Co

See Slides & video for the session.

This was tested on Yosemite 10.10.5 1) Install NiFi on your MacOS: http://docs.hortonworks.com/HDPDocuments/HDF1/HDF-1.2/bk_HDF_InstallSetup/content/ch_HDF_installing.html 2) Setup your machine to foward syslog messages to port 1514. Backup your current syslog configuration. mv /etc/syslog.conf /etc/syslog.conf.bkp Edit your syslog.conf file to send all messages to UDP localhost port 1514 sudo vi /etc/syslog.conf Add the following entry to /etc/syslog.conf *.* @127.0.0.1:1514Restart syslogd sudo launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plist sudo launchctl load /System/Library/LaunchDaemons/com.apple.syslogd.plist Confirm syslogd is running. A result should display a process id (PID) for /usr/sbin/syslogd ps -ef | grep syslogd 3) Test with NiFi. Add a ListenSyslog processor to the canvas with following settings: Protocol: UDP Port: 1514 Local Network Interface: lo0 Connect the ListenSyslog process to an output port and have the r

Setting up Hortonworks Dataflow (HDF) to work with kerberized Kafka in Hortonworks Data Platform (HDP)HDF 1.2 does not contain the same Kafka client libraries as the Apache NiFi version. HDF Kafka libraries are specifically designed to work with the Kafka versions supplied with HDP.The following Kafka support matrix breaks down what is supported in each Kafka version:*** (Apache) refers to the Kafka version downloadable from the Apache website.For newer versions of HDF (1.1.2+), NiFi uses zookeeper to maintain cluster wide state. So the following only applies if this is a HDF NiFi cluster:1. If a NiFi cluster has been setup to use a kerberized external or internal zookeeper for state, every kerberized connection to any other zookeeper would require using the same keytab and principal. For example a kerberized embedded zookeeper in NiFi would need to be configured to use the same client keytab and principal you want to use to authenticate with a say a Kafka zookeeper. 2. If

This is an extension of Starting Spark jobs directly via YARN REST APIAssume the cluster is kerberized and the only access is Knox. Further assume that Knox uses Basic Authentication and we have user and password of the user to start the Spark job.The overall idea is to callcurl with <<user>>:<<password>> as Basic Authentication ==> Knox (verifying user:password against LDAP or AD) ==> Resource Manager (YARN REST API) with kerberos principal1) Create and distribute a keytab The user that should run the spark job also needs to have a kerberos principal. For this principle create a keytab on one machine:[root@KDC-HOST ~]$ kadminkadmin> xst -k /etc/security/keytabs/<<primaryName>>.keytab <<primaryName>>@<<REALM>>Use your REALM and an appropriate primaryName of your principleThen distribute this keytab to all other machines in the cluster, copy to /etc/security/keytabs and set permissions[root@CLUSTER-HOST ~]$ chown <&

Snakebite is a pure python based HDFS client which is much faster than the classic hdfs client operations(hadoop fs / hdfs dfs).Snakebite is developed by Spotify. More details can be found at https://github.com/spotify/snakebiteI am using HDP 2.4 and Ambari 2.2.1.1 on CentOS/1). Install Python Pip. yum install python-pip2).Install Snakebite.pip install snakebiteNow snakebite is installed. You can run below sample commands.snakebite ls /tmpsnakebite ls /Use snakebite help for more options.Happy Snakebiting!

Apache Drill is an open source, low-latency query engine for Hadoop that delivers secure, interactive SQL analytics at petabyte scale. With the ability to discover schemas on-the-fly, Drill is a pioneer in delivering self-service data exploration capabilities on data stored in multiple formats in files or NoSQL databases. By adhering to ANSI SQL standards, Drill does not require a learning curve and integrates seamlessly with visualization tools. SourceTutorial:Grab latest version of drillwget http://getdrill.org/drill/download/apache-drill-1.2.0.tar.gz tar xvfz apache-drill-1.2.0.tar.gz /root/drill/apache-drill-1.2.0[root@node1 apache-drill-1.2.0]# cd bin/Start Drill in distributed mode ( You can start in embedded mode too)[root@node1 conf]# ../bin/drillbit.shstartstarting drillbit, logging to /root/drill/apache-drill-1.2.0/log/drillbit.outDrill Web Console - http://host:8047Enable storage pluginsClick Storage --> Enable hbase,hive,mongoModify storage plugin for Hive and HBase as

The current HDP 2.3.2 Sandbox offering on Azure Marketplace has the user and password: user: azure password: azure The current splash page states: user: adminpass: adminwhich is incorrect. This will be fixed in a future update of the Sandbox.

Original Post Calcite is a highly customizable engine for parsing and planning queries on data in a wide variety of formats. It allows database-like access, and in particular a SQL interface and advanced query optimization, for datanot residing in a traditional database. Apache Calcite is a dynamic data management framework. It contains many of the pieces that comprise a typical database management system, but omits some key functions: storage of data, algorithms to process data, and a repository for storing metadata. Calcite intentionally stays out of the business of storing and processing data. As we shall see, this makes it an excellent choice for mediating between applications and one or more data storage locations and data processing engines. It is also a perfect foundation for building a database: just add data. Source Tutorial https://calcite.apache.org/docs/tutorial.html Demo: Read DEPT and EMPS table Create a test table based on existing csv examp

Dear Colleages, I'm not able to change my community account email address. Can you tell me how does it works, please? Thanks in advance,Butkiz 

Platform Theme Key FeaturesFully Automated Scripted Install of Metron on AWSOne of the largest hurdles we have heard about from the community and customers working with the original OpenSoc code base was that it was nearly impossible to get the application up and running. Hence, our engineering team collaborated with the community to provide a scripted automated install of Metron on AWS. The install only requires the user’s AWS credentials, a set of ansible scripts/playbooks, and Ambari BluePrints / APIs and AWS APIs to deploy the full end to end Metron application. The below table summarizes the steps that occur during the automated install.StepDescriptionComponents Deployed Step 1Spin up EC2 instances where HDP and Metron will be installed and deployed10 m4.xlarge instances Step 2 Spin up an AWS VPC1 AWS VPC Step 3Install Ambari Server and Agents via Ansible ScriptsAmbari Server 2.1.2.1 on master node Ambari Agents on slave nodes Step 4Using Ambari Blueprints and APIS, install 7

Part 1Linkedin PostPart 1 - In case you missed the Introduction to Apache NiFiAssumption - HDP and NiFi installation is in place. You can use HDP Sandbox if you don't have the cluster. NiFi installation - You can follow Blog 1End Goal of this tutorial is to display Tweets related to particular search terms. For example: My twitter id is allaboutbdata and following screenshot shows the tweet sent on Twitter and same tweet in HDFS/Hive and Solr. The whole setup was done using NiFi.Demo:Install HDP search: yum install -y lucidworks-hdpsearchCreate user directory in HDFS & changer permissionssudo -u hdfs hadoop fs -mkdir /user/solr sudo -u hdfs hadoop fs -chown solr /user/solr chown -R solr:solr /opt/lucidworks-hdpsearch/solrSetup Solr su solr cd /opt/lucidworks-hdpsearch/solr/server/solr-webapp/webapp/banana/app/dashboards/ mv default.json default.json.orig wget https://raw.githubusercontent.com/abajwa-hw/ambari-nifi-service/master/demofiles/default.json Important : Must change the ho

Metron User PersonasThere are six user personas for Metron:Persona NameDescriptionSOC AnalystProfile: Beginner, Junior-level analystTools Used: SIEM tools/dashboards, Security endpoint UIs, Email/Ticketing/Workflow Systems Responsibilities: Monitor security SIEM tools, search/investigate breaches, malware, review alerts and determine to escalate as tickets or filter out, follow security playbooks, investigate script kiddie attacks.SOC InvestigatorProfile: More advanced SME in cybersecurity, Experienced security analyst, understands more advanced features of security tools, thorough understanding of networking and platform architecture (routers, switches, firewalls, security), Ability to dig through and understand various logs (Network, firewall, proxy, app, etc..) Tools Used: SIEM/Security tools, Scripting languages, SQL, command line Responsibilities: Investigate more complicated/escalated alerts, investigate breaches, Takes the necessary steps to remove/quarantine the malware, brea

***Take the backup of complete RANGER database before you perform any operation. This can happen if you encounter any issues during the upgrade or required java patches are partially applied 1) Stop Ranger from ambari ui 2) Take fresh db dump 3) Login to the ranger database, review x_policy table and make sure all policies are available (If policies are missing, you may have to restore database from the backup taken before the upgrade and simply restart ranger service) mysql> use ranger;mysql> select * from x_policy;  4) Delete all java patches from x_db_version_h table mysql> select * from x_db_version_h;mysql> create table x_db_version_h_backup as select * from x_db_version_h;mysql> delete from x_db_version_h where version like 'J%';  5) Start ranger service from ambari uiThis would take couple of minutes to re-apply all deleted patches. Once done, plugins will start downloading policies 6) Login to Ranger UI and v

Apache Metron - First Steps in the Cloud Apache Metron integrates a variety of open source technologies to offer a centralized platform for cyber security monitoring, analysis, and threat detection. The platform provides full packet capture indexing, telemetry aggregation and enrichment, advanced behavioral analytics, and the ability to tailor the platform to the specific threats facing your organization. These instruction will take you step-by-step from nothing to a fully-functioning, multi-node Apache Metron cluster running in Amazon's EC2 cloud environment. The platform will capture live network packets from the wire, leverage best-in-class open source tooling as additional sources of telemetry, and perform real-time enrichment of that data. The data is persisted for advanced analytical modeling and presented within a fully customizable, searchable single pane of glass. Getting Started Apache Metron depends on many excellent third-party, open-source components. To see Apache Met

An HTTPS endpoint for receiving data in NiFi requires two processors and two controller services: HandleHttpRequest, HandleHttpResponse, SSLContextService, and HttpContextMap. Note: The HandleHttpRequest processor in NiFi 0.6 does not have functional client authentication, but a fix will be implemented in the next version (see NIFI-1753). SSL Context Service This service can be created during the set up of the HandleHttpRequest processor The following properties should be set: Name = AgentSSLContextServiceKeystore Filename = <Path to Keystore>Keystore Password = <Keystore Password>Keystore Type = JKSSSL Protocol = TLS Since Client Authentication will be disabled in the HandleHttpRequest processor, the Truststore configurations are not necessary. HTTP Context Map This service can be created during the set up of the HandleHttpRequest processor The name should be set to AgentSSLContextMap HandleHttpRequest This processor receives HTTP requests The fol

"Error executing: call add_columns_to_support_audit_log_aggregation();java.sql.SQLException: Incorrect key file for table 'xa_access_audit'; try to repair itSQLException : SQL state: HY000 java.sql.SQLException: Incorrect key file for table 'xa_access_audit'; try to repair it ErrorCode: 10342016-04-11 06:05:59,187 [E] 015-auditlogaggregation.sql import failed!"SOLUTION:Make sure you have enough space in the /tmp directory and take the backup of 'xa_access_audit' as below.Login to MySQL. use ranger_audit; create table xa_access_audit_backup as select * from 'xa_access_audit' ; truncate table xa_access_audit_backup;and retry the upgrade.

Hello from the Metron PM and Eng TeamToday, the Hortonworks Metron product management and engineering team are kicking off a multi-part blog series on Apache Metron, the next gen security analytics application that Hortonworks is building working with the Apache Community. Over the course of the next few weeks, we will release a series of blogs that covers the following topics:Part 1 - Apache Metron Explained - Overview of Apache Metron and traces a security telemetry event as it flows through the platform. Part 2 - Apache Metron User Personas and Why Metron? - Who will be the different users of Apache Metron? What are the core functional themes? What has been the focus for the first release? We will address all 3 of these questions in this blog.Part 3 - Apache Metron Tech Preview 1 - Come and Get It We will walk you through what the Metron community has been working on for the last 3 months. By the end of this blog, you will have a good understanding of what is in Metron Tech Previe

After logging in to the forums, the rest of your website becomes unusable until I log out again.  Tried this in both firefox and chrome. After logging in to the forums all pages including support, documentation, and downloads all redirect here: http://www.cloudera.com/content/cloudera/en/access-is-denied.html.  Many of the other pages such as products, solutions, and partners are still usable. I can get to everything again when i log out.

Prerequisites: Launch Sandbox on Azure VM Size: Minimum of A4 or A5 A Twitter App You'll use the API credentials The "Application Details" don't matter Prepare the Sandbox Connect to SSH & Ambari Connect to the Sandbox using SSH or web console: http://<<ip>>:4200/ Become root: sudo su - Reset the Ambari password: ambari-admin-password-reset Login to Ambari: http://<<ip>>:8080 User: admin Before moving to the next steps, ensure all services on the left are started (green) or in maintenance mode (black). Install NiFi In Ambari, Click "Actions" (bottom left) -> Add Service Choose NiFi and continue through the dialogs. You shouldn't need to change anything NiFi should now be accessible at http:<<ip>>:9090/nifi/ Tune Sandbox The Sandbox is tuned to run on minimal hardware. We need to update the Hive, Tez & YARN configuration for our use case. This could take up to 15 minutes to complete: bash <(curl -sSL https://git.io/v

RequirementsHDP 2.3.x cluster, whether it is a multi-node cluster or a single-node HDP Sandbox.InstallingThe Spark 1.6 Technical Preview is provided in RPM and DEB package formats. The following instructions assume RPM packaging: Download the Spark 1.6 RPM repository:wget -nv http://private-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.4.1-10/hdp.repo -O /etc/yum.repos.d/HDP-TP.repoFor installing on Ubuntu use the following: http://private-repo-1.hortonworks.com/HDP/ubuntu12/2.x/updates/2.3.4.1-10/hdp.list Install the Spark Package: Download the Spark 1.6 RPM (and pySpark, if desired) and set it up on your HDP 2.3 cluster:yum install <strong>spark</strong>_2_3_4_1_10-master -yIf you want to use pySpark, install it as follows and make sure that Python is installed on all nodes.yum install <strong>spark</strong>_2_3_4_1_10-python -yThe RPM installer will also download core Hadoop dependencies. It will create “spark” as an OS user, and it will create the /us