Community Articles

The current HDP 2.3.2 Sandbox offering on Azure Marketplace has the user and password: user: azure password: azure The current splash page states: user: adminpass: adminwhich is incorrect. This will be fixed in a future update of the Sandbox.

Original Post Calcite is a highly customizable engine for parsing and planning queries on data in a wide variety of formats. It allows database-like access, and in particular a SQL interface and advanced query optimization, for datanot residing in a traditional database. Apache Calcite is a dynamic data management framework. It contains many of the pieces that comprise a typical database management system, but omits some key functions: storage of data, algorithms to process data, and a repository for storing metadata. Calcite intentionally stays out of the business of storing and processing data. As we shall see, this makes it an excellent choice for mediating between applications and one or more data storage locations and data processing engines. It is also a perfect foundation for building a database: just add data. Source Tutorial https://calcite.apache.org/docs/tutorial.html Demo: Read DEPT and EMPS table Create a test table based on existing csv examp

Platform Theme Key FeaturesFully Automated Scripted Install of Metron on AWSOne of the largest hurdles we have heard about from the community and customers working with the original OpenSoc code base was that it was nearly impossible to get the application up and running. Hence, our engineering team collaborated with the community to provide a scripted automated install of Metron on AWS. The install only requires the user’s AWS credentials, a set of ansible scripts/playbooks, and Ambari BluePrints / APIs and AWS APIs to deploy the full end to end Metron application. The below table summarizes the steps that occur during the automated install.StepDescriptionComponents Deployed Step 1Spin up EC2 instances where HDP and Metron will be installed and deployed10 m4.xlarge instances Step 2 Spin up an AWS VPC1 AWS VPC Step 3Install Ambari Server and Agents via Ansible ScriptsAmbari Server 2.1.2.1 on master node Ambari Agents on slave nodes Step 4Using Ambari Blueprints and APIS, install 7

Part 1Linkedin PostPart 1 - In case you missed the Introduction to Apache NiFiAssumption - HDP and NiFi installation is in place. You can use HDP Sandbox if you don't have the cluster. NiFi installation - You can follow Blog 1End Goal of this tutorial is to display Tweets related to particular search terms. For example: My twitter id is allaboutbdata and following screenshot shows the tweet sent on Twitter and same tweet in HDFS/Hive and Solr. The whole setup was done using NiFi.Demo:Install HDP search: yum install -y lucidworks-hdpsearchCreate user directory in HDFS & changer permissionssudo -u hdfs hadoop fs -mkdir /user/solr sudo -u hdfs hadoop fs -chown solr /user/solr chown -R solr:solr /opt/lucidworks-hdpsearch/solrSetup Solr su solr cd /opt/lucidworks-hdpsearch/solr/server/solr-webapp/webapp/banana/app/dashboards/ mv default.json default.json.orig wget https://raw.githubusercontent.com/abajwa-hw/ambari-nifi-service/master/demofiles/default.json Important : Must change the ho

Metron User PersonasThere are six user personas for Metron:Persona NameDescriptionSOC AnalystProfile: Beginner, Junior-level analystTools Used: SIEM tools/dashboards, Security endpoint UIs, Email/Ticketing/Workflow Systems Responsibilities: Monitor security SIEM tools, search/investigate breaches, malware, review alerts and determine to escalate as tickets or filter out, follow security playbooks, investigate script kiddie attacks.SOC InvestigatorProfile: More advanced SME in cybersecurity, Experienced security analyst, understands more advanced features of security tools, thorough understanding of networking and platform architecture (routers, switches, firewalls, security), Ability to dig through and understand various logs (Network, firewall, proxy, app, etc..) Tools Used: SIEM/Security tools, Scripting languages, SQL, command line Responsibilities: Investigate more complicated/escalated alerts, investigate breaches, Takes the necessary steps to remove/quarantine the malware, brea

***Take the backup of complete RANGER database before you perform any operation. This can happen if you encounter any issues during the upgrade or required java patches are partially applied 1) Stop Ranger from ambari ui 2) Take fresh db dump 3) Login to the ranger database, review x_policy table and make sure all policies are available (If policies are missing, you may have to restore database from the backup taken before the upgrade and simply restart ranger service) mysql> use ranger;mysql> select * from x_policy;  4) Delete all java patches from x_db_version_h table mysql> select * from x_db_version_h;mysql> create table x_db_version_h_backup as select * from x_db_version_h;mysql> delete from x_db_version_h where version like 'J%';  5) Start ranger service from ambari uiThis would take couple of minutes to re-apply all deleted patches. Once done, plugins will start downloading policies 6) Login to Ranger UI and v

Apache Metron - First Steps in the Cloud Apache Metron integrates a variety of open source technologies to offer a centralized platform for cyber security monitoring, analysis, and threat detection. The platform provides full packet capture indexing, telemetry aggregation and enrichment, advanced behavioral analytics, and the ability to tailor the platform to the specific threats facing your organization. These instruction will take you step-by-step from nothing to a fully-functioning, multi-node Apache Metron cluster running in Amazon's EC2 cloud environment. The platform will capture live network packets from the wire, leverage best-in-class open source tooling as additional sources of telemetry, and perform real-time enrichment of that data. The data is persisted for advanced analytical modeling and presented within a fully customizable, searchable single pane of glass. Getting Started Apache Metron depends on many excellent third-party, open-source components. To see Apache Met

An HTTPS endpoint for receiving data in NiFi requires two processors and two controller services: HandleHttpRequest, HandleHttpResponse, SSLContextService, and HttpContextMap. Note: The HandleHttpRequest processor in NiFi 0.6 does not have functional client authentication, but a fix will be implemented in the next version (see NIFI-1753). SSL Context Service This service can be created during the set up of the HandleHttpRequest processor The following properties should be set: Name = AgentSSLContextServiceKeystore Filename = <Path to Keystore>Keystore Password = <Keystore Password>Keystore Type = JKSSSL Protocol = TLS Since Client Authentication will be disabled in the HandleHttpRequest processor, the Truststore configurations are not necessary. HTTP Context Map This service can be created during the set up of the HandleHttpRequest processor The name should be set to AgentSSLContextMap HandleHttpRequest This processor receives HTTP requests The fol

"Error executing: call add_columns_to_support_audit_log_aggregation();java.sql.SQLException: Incorrect key file for table 'xa_access_audit'; try to repair itSQLException : SQL state: HY000 java.sql.SQLException: Incorrect key file for table 'xa_access_audit'; try to repair it ErrorCode: 10342016-04-11 06:05:59,187 [E] 015-auditlogaggregation.sql import failed!"SOLUTION:Make sure you have enough space in the /tmp directory and take the backup of 'xa_access_audit' as below.Login to MySQL. use ranger_audit; create table xa_access_audit_backup as select * from 'xa_access_audit' ; truncate table xa_access_audit_backup;and retry the upgrade.

Hello from the Metron PM and Eng TeamToday, the Hortonworks Metron product management and engineering team are kicking off a multi-part blog series on Apache Metron, the next gen security analytics application that Hortonworks is building working with the Apache Community. Over the course of the next few weeks, we will release a series of blogs that covers the following topics:Part 1 - Apache Metron Explained - Overview of Apache Metron and traces a security telemetry event as it flows through the platform. Part 2 - Apache Metron User Personas and Why Metron? - Who will be the different users of Apache Metron? What are the core functional themes? What has been the focus for the first release? We will address all 3 of these questions in this blog.Part 3 - Apache Metron Tech Preview 1 - Come and Get It We will walk you through what the Metron community has been working on for the last 3 months. By the end of this blog, you will have a good understanding of what is in Metron Tech Previe

Prerequisites: Launch Sandbox on Azure VM Size: Minimum of A4 or A5 A Twitter App You'll use the API credentials The "Application Details" don't matter Prepare the Sandbox Connect to SSH & Ambari Connect to the Sandbox using SSH or web console: http://<<ip>>:4200/ Become root: sudo su - Reset the Ambari password: ambari-admin-password-reset Login to Ambari: http://<<ip>>:8080 User: admin Before moving to the next steps, ensure all services on the left are started (green) or in maintenance mode (black). Install NiFi In Ambari, Click "Actions" (bottom left) -> Add Service Choose NiFi and continue through the dialogs. You shouldn't need to change anything NiFi should now be accessible at http:<<ip>>:9090/nifi/ Tune Sandbox The Sandbox is tuned to run on minimal hardware. We need to update the Hive, Tez & YARN configuration for our use case. This could take up to 15 minutes to complete: bash <(curl -sSL https://git.io/v

RequirementsHDP 2.3.x cluster, whether it is a multi-node cluster or a single-node HDP Sandbox.InstallingThe Spark 1.6 Technical Preview is provided in RPM and DEB package formats. The following instructions assume RPM packaging: Download the Spark 1.6 RPM repository:wget -nv http://private-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.4.1-10/hdp.repo -O /etc/yum.repos.d/HDP-TP.repoFor installing on Ubuntu use the following: http://private-repo-1.hortonworks.com/HDP/ubuntu12/2.x/updates/2.3.4.1-10/hdp.list Install the Spark Package: Download the Spark 1.6 RPM (and pySpark, if desired) and set it up on your HDP 2.3 cluster:yum install <strong>spark</strong>_2_3_4_1_10-master -yIf you want to use pySpark, install it as follows and make sure that Python is installed on all nodes.yum install <strong>spark</strong>_2_3_4_1_10-python -yThe RPM installer will also download core Hadoop dependencies. It will create “spark” as an OS user, and it will create the /us

SYMPTOM: Attempting to submit a Pig job via WebHCat that defines parameter(s) for substitution as command-line arguments results in an "incorrect usage" message and job does not run. Doing the same through Hue results in an "undefined parameter" message and job does not run. ROOT CAUSE: If the parameter is passed to curl as a single argument (-d 'arg=-param paramName=paramValue') it is interpreted incorrectly by Pig. Submitting the parameter via Hue as a single argument has the same unwanted effect.WORKAROUND: Pass the parameter as two arguments: curl -d file=myScript.pig -d 'arg=-param' -d 'arg=paramName=paramValue' 'http://<server>:50111/templeton/v1/pig'To achieve the same using Hue, pass two arguments in sequence (refer to attached image for an example).RESOLUTION: WebHCat works as designed. This issue is a limitation of curl. The Hue workaround is good for a single parameter, however multiple parameters may not work.

Geo Distance calculations in Hive and Java March 23, 2016 Eran Orgad Geographical distance is the distance measured along the surface of the earth. The distances between points are defined by geographical coordinates (Geo Location cords) in terms of latitude and longitude. Each pair represent a location as: Lexington MA: Latitude : 42.4428, Longitude: -71.2317 Mountain View CA: Latitude: 37.405990600586, Longitude: -122.07851409912 Distance: 2676.09225228497 Miles To be able to calculate the distance on Hortonworks HDP using Hive (and Tez) let’s generate some sample data that includes some pairs of Geolocation coordinates. Data file called: distance1.csv 42.28,-71.87,42.28,-71.86 42.00,-71.87,42.28,-71.11 42.28,-71.87,42.28,-71.86 42.28,-71.87,42.28,-71.22 42.00,-71.87,42.28,-72.33 42.28,-71.87,42.28,-70.44 42.00,-71.87,42.28,-71.55 42.28,-71.87,41.28,-71.66 42.00,-71.87,43.28,-71.77 42.28,-71.87,44.28,-71.88 42.00,-71.87,45.28,-71.99 42.28,-71.87,46.1

Issue: In 2.3.4 Ranger Hive Repo Test Connection not working with HS2 HTTP TransportMode configuration. Workaround: 1) Move httpclient-4.2.5.jar httpcore-4.2.5.jar from /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/lib to a tmp folder. 2) fetch the httpclient-4.4.jar and httpcore-4.4.jar from /usr/hdp/current/hive-server2/lib/ and place it in /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/lib folder 3) Restart ranger admin and configure hive repo with http config and test the HTTP connection and lookup.

IntroductionIf you are new to Metron or the Metron Tech Preview 1, the following links should provide some good information to review before walking through the installation: Intro to Apache MetronWhat is in Apache Metron Tech Preview 1Build InstructionsThe following steps provide instructions on how to install a full working Metron application on a single node VM with Vagrant. This deployment option is ideal for experimenting and playing with the Metron application. While these instructions should work on most development environments, these instructions were tested on Mac OS X El Capitan.PrerequisitesOn your Macintosh: Install the latest version of Virtual Box. Install the latest version of Vagrant. Install Maven if you don't have it, and define associated environmental variables. For example, add the following to your ~/.bash_profile file:export MAVEN_HOME=/Users/rmckissick/Documents/Files/apache-maven-3.3.9export PATH=$MAVEN_HOME/bin:$PATHInstall JAVA 1.8 if you don't have it, and

If you don’t have important data in Cloudbreak, you can drop the db using root and do a...cbd deleteSwitch back to the cloudbreak user and Edit the ‘Profile’ file in your cloud break deployment directory (i.e. /var/lib/cloudbreak-deployment). Then export the username and password you would like to use as the default user. export UAA_DEFAULT_USER_EMAIL=admin@example.comexport UAA_DEFAULT_USER_PW=mypassThis will override the default settings. Then...cbd initcbd startOn startup, your default settings should be displayed as being changed.

IntroductionSecurity best practices when using Ranger dictate that Hive jobs should ideally run as user 'hive' so that only Ranger Hive policies apply for end user access to data, and letting 'hive' own all the directory/file structure for Hive on HDFS. This is achieved by using hive.server2.enable.doAs set to 'false'. It also allows to improve performance as it enables container pre-warming for Tez, as it is only applicable for those jobs started by 'hive', and not by other end users.ProblemThe problem introduced by doAs = false is that, if YARN Capacity Scheduler queue mappings have been defined on a user/group basis, the mappings will not apply since all the jobs will be started as the same user (i.e. 'hive'), making the queue definitions completely useless.SolutionOne solution could be to use a Hive hook that could detect the real user that started the query so that we could submit the job to the right queue even if it still runs as user 'hive'. Then, the hook could find the list o

Pragmatic Kafka Security 0.9, Setup and Java Producer Jump Start Kafka Security implementation.SSL for Authentication, ACL for AuthorizationThe below steps will provide 4 vms with Kafka Zookeeper installed on all of them via VagrantSSL authentication will be enabled between the Consumers and BrokersACL is also enabledPrerequisites Vagrant : Vagrant installation https://www.vagrantup.com/downloads.htmlMavenBasic Java / Kafka Understanding.gitNote New Console Consumer adds Group.id by default.Main Commands. Vagrant suspend, Vagrant resume --no-provision , Vagrant destroyTo clean up all, just run vagrant destroy -f (everything will get cleaned)ImageInstallation and Running (install commands)Step1 git clone https://github.com/Symantec/kafka-security-0.9.gitcd kafka-security-demoRun Start.sh and go for coffee or just read along documentation ( 10 - 15 min) (internally runs sh /vagrant/data/step1-all.sh => update software, install java, kafka, zoo)(internally runs sh /vagrant/data/step2s

Pre-reqs are to download and start the EsgynDB sandbox: https://community.hortonworks.com/content/kbentry/23695/esgyndb-based-on-apache-trafodion-and-hdp-23-sandb.html1. Download Trafodion clients from https://esgyn.com/product/download2. Untar JDBCT4.zip file and extract jdbcT4.jar file3. Copy jdbcT4.jar file to zeppelin install folder/lib and restart zeppelin services4. Launch the zeppelin localhost:99955. Go to the interpreter tab and click on +Create6. In create interpreter, for Name fieldtype, TrafodionFor Interpreter drop down box, select psqlIt will show up the properties boxfor postgresql.driver.nameorg.trafodion.jdbc.t4.T4DriverFor postgresql.passwordssFor postgresql.urljdbc:jdbcT4://<yourTrafodion server>:23400/:For postgresql.userssThen click SaveThe click restart (far right)7. Go to Notebook and create notebook by specifying a nameOn the far right, click on "Interpreter Binding" Bind Trafodion interpreter to your notebookMove Trafodion to the top and unselect the othe

Download the OVA:https://www.dropbox.com/sh/fqq63zjdaqjg9hn/AABusSGCjXA_Z7ffzBhPHD9pa?dl=0 Instructions to Start: Import SandboxStartLog on directly: <Alt+f5> (user: root password traf123! NOTE: It will say Password "hadoop" it is not)or user putty (127.0.0.1 Port 2222 user root password traf123!)Open GUI via browser: 127.0.0.1:8080 (admin/admin)Start All Start HDFS Zookeeper HBase in that order independentlyhbase shell -> status 'detailed' (should get a detailed status and no errors)From putty: sudo su trafodionsqcheckcstat (make sure no process running)sqstartInstructions to Stop/Save: From Trafodion sqstopckillallFrom Ambari GUI Stop AllStop HBase Stop ZookeeperStop HDFSFrom putty do a jps and make sure no HBase processes are running. Stop and Save session

As of HDP 2.3 Sandbox, there is now a another way to determine the version of Sandbox being used. Previously, comparing md5 checksums was one way to compare versions of Sandboxes. Now, one can use the command sandbox-versionRunning the command in the terminal will output a result such as[root@sandbox azure]# sandbox-version Sandbox information: Created on: 27_10_2015_15_18_06 for vmware Hadoop stack version: Hadoop 2.7.1.2.3.2.0-2950 Ambari Version: 2.1.2

Please find below a potential design for Disk & RAID configuration for a typical 12 disk Server running NiFi. This design is intended for a simple log ingestion use case, where customer needs very little provenance records, but would also like reliability on the storage layer.FlowFile repo: 2 drives setup as RAID 1 Provenance repo: 2 Drives RAID 1 Content repo setup either: 4 drives (RAID 10) /cont_repo1 4 drives (RAID 10) /cont_repo2 or 2 drives (RAID 1) /cont_repo1 2 drives (RAID 1) /cont_repo2 2 drives (RAID 1) /cont_repo3 2 drives (RAID 1) /cont_repo4 Thanks @mpayne & @Andrew Grande for guidance!

Download connector http://hortonworks.com/hdp/addons/**** Extract tar file ******** Copy jar into sqoop-client/lib ***cp *.jar /usr/hdp/current/sqoop-client/lib/**** Create tables in Teradata ****We will be importing data from /tmp/test , HDFS location**** Sqoop ****sqoop export --connect jdbc:teradata://Terdatahost/Database=DBName --connection-manager org.apache.sqoop.teradata.TeradataConnManager --username user --password passwd --table test --export-dir /tmp/test/ --batch

In this article, we will install Apache Hawq 2.0.0.0_beta in a cluster composed by:2 masters (1 active, 1 standby)3 segments (slaves)On each node on the cluster install repository to fetch libhdfs3:[root@hdpmaster01~]# curl -s -L "https://bintray.com/wangzw/rpm/rpm" -o /etc/yum.repos.d/bintray-wangzw-rpm.repo[root@hdpmaster01~]#Install epel repository:[root@hdpmaster01~]# yum -y install epel-releaseInstall missing dependencies:[root@hdpamaster01~]# yum -y install man passwd sudo tar which git mlocate links make bzip2 net-tools autoconf automake libtool m4 gcc gcc-c++ gdb bison flex cmake gperf maven indent libuuid-devel krb5-devel libgsasl-devel expat-devel libxml2-devel perl-ExtUtils-Embed pam-devel python-devel libcurl-devel snappy-devel thrift-devel libyaml-devel libevent-devel bzip2-devel openssl-devel openldap-devel protobuf-devel readline-devel net-snmp-devel apr-devel libesmtp-devel xerces-c-devel python-pip json-c-devel libhdfs3-devel apache-ivy java-1.7.0-openjdk-devel openssh