Resources

SYMPTOM: Attempting to submit a Pig job via WebHCat that defines parameter(s) for substitution as command-line arguments results in an "incorrect usage" message and job does not run. Doing the same through Hue results in an "undefined parameter" message and job does not run. ROOT CAUSE: If the parameter is passed to curl as a single argument (-d 'arg=-param paramName=paramValue') it is interpreted incorrectly by Pig. Submitting the parameter via Hue as a single argument has the same unwanted effect.WORKAROUND: Pass the parameter as two arguments: curl -d file=myScript.pig -d 'arg=-param' -d 'arg=paramName=paramValue' 'http://<server>:50111/templeton/v1/pig'To achieve the same using Hue, pass two arguments in sequence (refer to attached image for an example).RESOLUTION: WebHCat works as designed. This issue is a limitation of curl. The Hue workaround is good for a single parameter, however multiple parameters may not work.

Geo Distance calculations in Hive and Java March 23, 2016 Eran Orgad Geographical distance is the distance measured along the surface of the earth. The distances between points are defined by geographical coordinates (Geo Location cords) in terms of latitude and longitude. Each pair represent a location as: Lexington MA: Latitude : 42.4428, Longitude: -71.2317 Mountain View CA: Latitude: 37.405990600586, Longitude: -122.07851409912 Distance: 2676.09225228497 Miles To be able to calculate the distance on Hortonworks HDP using Hive (and Tez) let’s generate some sample data that includes some pairs of Geolocation coordinates. Data file called: distance1.csv 42.28,-71.87,42.28,-71.86 42.00,-71.87,42.28,-71.11 42.28,-71.87,42.28,-71.86 42.28,-71.87,42.28,-71.22 42.00,-71.87,42.28,-72.33 42.28,-71.87,42.28,-70.44 42.00,-71.87,42.28,-71.55 42.28,-71.87,41.28,-71.66 42.00,-71.87,43.28,-71.77 42.28,-71.87,44.28,-71.88 42.00,-71.87,45.28,-71.99 42.28,-71.87,46.1

Issue: In 2.3.4 Ranger Hive Repo Test Connection not working with HS2 HTTP TransportMode configuration. Workaround: 1) Move httpclient-4.2.5.jar httpcore-4.2.5.jar from /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/lib to a tmp folder. 2) fetch the httpclient-4.4.jar and httpcore-4.4.jar from /usr/hdp/current/hive-server2/lib/ and place it in /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/lib folder 3) Restart ranger admin and configure hive repo with http config and test the HTTP connection and lookup.

IntroductionIf you are new to Metron or the Metron Tech Preview 1, the following links should provide some good information to review before walking through the installation: Intro to Apache MetronWhat is in Apache Metron Tech Preview 1Build InstructionsThe following steps provide instructions on how to install a full working Metron application on a single node VM with Vagrant. This deployment option is ideal for experimenting and playing with the Metron application. While these instructions should work on most development environments, these instructions were tested on Mac OS X El Capitan.PrerequisitesOn your Macintosh: Install the latest version of Virtual Box. Install the latest version of Vagrant. Install Maven if you don't have it, and define associated environmental variables. For example, add the following to your ~/.bash_profile file:export MAVEN_HOME=/Users/rmckissick/Documents/Files/apache-maven-3.3.9export PATH=$MAVEN_HOME/bin:$PATHInstall JAVA 1.8 if you don't have it, and

If you don’t have important data in Cloudbreak, you can drop the db using root and do a...cbd deleteSwitch back to the cloudbreak user and Edit the ‘Profile’ file in your cloud break deployment directory (i.e. /var/lib/cloudbreak-deployment). Then export the username and password you would like to use as the default user. export UAA_DEFAULT_USER_EMAIL=admin@example.comexport UAA_DEFAULT_USER_PW=mypassThis will override the default settings. Then...cbd initcbd startOn startup, your default settings should be displayed as being changed.

IntroductionSecurity best practices when using Ranger dictate that Hive jobs should ideally run as user 'hive' so that only Ranger Hive policies apply for end user access to data, and letting 'hive' own all the directory/file structure for Hive on HDFS. This is achieved by using hive.server2.enable.doAs set to 'false'. It also allows to improve performance as it enables container pre-warming for Tez, as it is only applicable for those jobs started by 'hive', and not by other end users.ProblemThe problem introduced by doAs = false is that, if YARN Capacity Scheduler queue mappings have been defined on a user/group basis, the mappings will not apply since all the jobs will be started as the same user (i.e. 'hive'), making the queue definitions completely useless.SolutionOne solution could be to use a Hive hook that could detect the real user that started the query so that we could submit the job to the right queue even if it still runs as user 'hive'. Then, the hook could find the list o

Pragmatic Kafka Security 0.9, Setup and Java Producer Jump Start Kafka Security implementation.SSL for Authentication, ACL for AuthorizationThe below steps will provide 4 vms with Kafka Zookeeper installed on all of them via VagrantSSL authentication will be enabled between the Consumers and BrokersACL is also enabledPrerequisites Vagrant : Vagrant installation https://www.vagrantup.com/downloads.htmlMavenBasic Java / Kafka Understanding.gitNote New Console Consumer adds Group.id by default.Main Commands. Vagrant suspend, Vagrant resume --no-provision , Vagrant destroyTo clean up all, just run vagrant destroy -f (everything will get cleaned)ImageInstallation and Running (install commands)Step1 git clone https://github.com/Symantec/kafka-security-0.9.gitcd kafka-security-demoRun Start.sh and go for coffee or just read along documentation ( 10 - 15 min) (internally runs sh /vagrant/data/step1-all.sh => update software, install java, kafka, zoo)(internally runs sh /vagrant/data/step2s

Pre-reqs are to download and start the EsgynDB sandbox: https://community.hortonworks.com/content/kbentry/23695/esgyndb-based-on-apache-trafodion-and-hdp-23-sandb.html1. Download Trafodion clients from https://esgyn.com/product/download2. Untar JDBCT4.zip file and extract jdbcT4.jar file3. Copy jdbcT4.jar file to zeppelin install folder/lib and restart zeppelin services4. Launch the zeppelin localhost:99955. Go to the interpreter tab and click on +Create6. In create interpreter, for Name fieldtype, TrafodionFor Interpreter drop down box, select psqlIt will show up the properties boxfor postgresql.driver.nameorg.trafodion.jdbc.t4.T4DriverFor postgresql.passwordssFor postgresql.urljdbc:jdbcT4://<yourTrafodion server>:23400/:For postgresql.userssThen click SaveThe click restart (far right)7. Go to Notebook and create notebook by specifying a nameOn the far right, click on "Interpreter Binding" Bind Trafodion interpreter to your notebookMove Trafodion to the top and unselect the othe

Download the OVA:https://www.dropbox.com/sh/fqq63zjdaqjg9hn/AABusSGCjXA_Z7ffzBhPHD9pa?dl=0 Instructions to Start: Import SandboxStartLog on directly: <Alt+f5> (user: root password traf123! NOTE: It will say Password "hadoop" it is not)or user putty (127.0.0.1 Port 2222 user root password traf123!)Open GUI via browser: 127.0.0.1:8080 (admin/admin)Start All Start HDFS Zookeeper HBase in that order independentlyhbase shell -> status 'detailed' (should get a detailed status and no errors)From putty: sudo su trafodionsqcheckcstat (make sure no process running)sqstartInstructions to Stop/Save: From Trafodion sqstopckillallFrom Ambari GUI Stop AllStop HBase Stop ZookeeperStop HDFSFrom putty do a jps and make sure no HBase processes are running. Stop and Save session

As of HDP 2.3 Sandbox, there is now a another way to determine the version of Sandbox being used. Previously, comparing md5 checksums was one way to compare versions of Sandboxes. Now, one can use the command sandbox-versionRunning the command in the terminal will output a result such as[root@sandbox azure]# sandbox-version Sandbox information: Created on: 27_10_2015_15_18_06 for vmware Hadoop stack version: Hadoop 2.7.1.2.3.2.0-2950 Ambari Version: 2.1.2

Please find below a potential design for Disk & RAID configuration for a typical 12 disk Server running NiFi. This design is intended for a simple log ingestion use case, where customer needs very little provenance records, but would also like reliability on the storage layer.FlowFile repo: 2 drives setup as RAID 1 Provenance repo: 2 Drives RAID 1 Content repo setup either: 4 drives (RAID 10) /cont_repo1 4 drives (RAID 10) /cont_repo2 or 2 drives (RAID 1) /cont_repo1 2 drives (RAID 1) /cont_repo2 2 drives (RAID 1) /cont_repo3 2 drives (RAID 1) /cont_repo4 Thanks @mpayne & @Andrew Grande for guidance!

Download connector http://hortonworks.com/hdp/addons/**** Extract tar file ******** Copy jar into sqoop-client/lib ***cp *.jar /usr/hdp/current/sqoop-client/lib/**** Create tables in Teradata ****We will be importing data from /tmp/test , HDFS location**** Sqoop ****sqoop export --connect jdbc:teradata://Terdatahost/Database=DBName --connection-manager org.apache.sqoop.teradata.TeradataConnManager --username user --password passwd --table test --export-dir /tmp/test/ --batch

In this article, we will install Apache Hawq 2.0.0.0_beta in a cluster composed by:2 masters (1 active, 1 standby)3 segments (slaves)On each node on the cluster install repository to fetch libhdfs3:[root@hdpmaster01~]# curl -s -L "https://bintray.com/wangzw/rpm/rpm" -o /etc/yum.repos.d/bintray-wangzw-rpm.repo[root@hdpmaster01~]#Install epel repository:[root@hdpmaster01~]# yum -y install epel-releaseInstall missing dependencies:[root@hdpamaster01~]# yum -y install man passwd sudo tar which git mlocate links make bzip2 net-tools autoconf automake libtool m4 gcc gcc-c++ gdb bison flex cmake gperf maven indent libuuid-devel krb5-devel libgsasl-devel expat-devel libxml2-devel perl-ExtUtils-Embed pam-devel python-devel libcurl-devel snappy-devel thrift-devel libyaml-devel libevent-devel bzip2-devel openssl-devel openldap-devel protobuf-devel readline-devel net-snmp-devel apr-devel libesmtp-devel xerces-c-devel python-pip json-c-devel libhdfs3-devel apache-ivy java-1.7.0-openjdk-devel openssh

Here my findings on how to increase sandbox disk space. Please if anyone has a easier solution, update this:1- Backup sandox files2- Convert .vmdk disk to .vdi (virtuabox does not allow resizing on vmdk images): VBoxManage clonehd "/Users/gbraccialli/VirtualBox VMs/HDP 2.3_1.2_B/Hortonworks Sandbox with HDP 2.3-disk1.vmdk" "/Users/gbraccialli/VirtualBox VMs/HDP 2.3_1.2_B/Hortonworks Sandbox with HDP 2.3-disk1.vdi" --format VDI --variant Standard3- Increase .vdi sizeVBoxManage modifyhd "/Users/gbraccialli/VirtualBox VMs/HDP 2.3_1.2_B/Hortonworks Sandbox with HDP 2.3-disk1.vdi" --resize 4000004- Open virtualbox, remove .vmdk image and add new .vdi image file5- Download gparted iso imagehttp://sourceforge.net/projects/gparted/?source=typ_redirect6- Add gparted iso image as cdrom drive on virtualbox, make cdrom as first device on boot order and boot virtualbox using gparted image7- Using gparted gui, resize the partition, save changes, shutdown virtualbox and remove gparted cdrom iso image

To disable SSLv3 for old Hue version, do the following:1.) open file "/usr/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py"2.) add the following line, after line no. 1669: ctx.set_options(SSL.OP_NO_SSLv2 | SSL.OP_NO_SSLv3)3.) restart HueTo check if Hue replies on SSLv3 or not you can use this one:openssl s_client -connect <hue-server-name>:<hue-port> -ssl3Here the corresponding GitHub patch, thanks to @cdraper:https://github.com/cloudera/hue/commit/0060abf9aae0049c082c9948658eea7df848ab6e

1. If using Ambari versions before 2.1.2, Disable HBase per region metrics. On Ambari server host. Edit the following files under /var/lib/ambari-server/resources/:common-services/HBASE/0.96.0.2.0/package/templates/HBASE/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2common-services/HBASE/0.96.0.2.0/package/templates/HBASE/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2and add the following lines at the end before '% endif %}':*.source.filter.class=org.apache.hadoop.metrics2.filter.GlobFilterhbase.*.source.filter.exclude=*Regions*Do a rolling restart of the HBase RegionServers. Note: This does not disable RS metrics. It just disables the per region / per table metrics collected at Region level. This is disabled by default from Ambari 2.1.2. 2. Tune AMS configs: Find out the Heap available to AMS collector host. Change the following settings based on available memory:ams-hbase-env_ :: hbase_master_heapsize = 8192m (Or 16384m if available)ams-hbase-env_ :: hbase_master_xmn_size =

Tutorial LinkSandboxVersion: HDP 2.3.2AmbariVersion2.1.2Hadoop stack version:Hadoop2.7.1.2.3.2.0-2950Issue 1: Error initializing SparkContext when executing spark-shell commandWhen you issue the command as rootspark-shell --master yarn-client --driver-memory 512m --executor-memory 512mYou would would receive the error:ERROR SparkContext: Error initializing SparkContext.org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user/root/.sparkStaging/application_1451921066894_0001":hdfs:hdfs:drwxr-xr-x Resolutionsudo su - hdfshdfs dfs -mkdir /user/roothdfs dfs -chown root:hdfs /user/rootexit--------------------------------------------------------Issue 2

Tutorial LinkSandbox Version: HDP 2.3.2Ambari Version 2.1.2Hadoop stack version: Hadoop 2.7.1.2.3.2.0-2950Issue 1: Permission DeniedCopy the data over to HDFS on Sandbox with the following command will result in a Permission denied error.hadoop fs -put ~/Hortonworks /user/guest/HortonworksResolution: See Hands-on Spark Tutorial: Permission Denied------------------------------

With recent improvements in Ambari, upgrading can be done easily either using Rolling Upgrade or Express Upgrade. But there are times when upgrade/downgrade gets stuck either since all the precautions were not followed or due to product issues. When upgrade gets stuck, it is typically in an Upgrade Paused status or there aren't any status as such. At this juncture, care needs to be taken to ensure that the ambari-server is not restarted without consulting with Technical Support.Current status of the upgrade can be checked using multiple method: Ambari Log files Ambari API URL's Ambari Databases1. Ambari Log files Review Ambari Log files for any errors / exception when the upgrade is in progress. 2. Use Ambari URL's to find what are the failures http://<ambari-server>:8080/api/v1/clusters/c1/upgrades For eg http://vcert1.novalocal:8080/api/v1/clusters/VCertify/upgrades This would show all the upgrade or downgrade attempts. for eg if this is a upgrade failure, identify the latest

Getting started with Nifi expression language and custom Nifi processors on HDP sandboxThis tutorial is part of a webinar for partners on Hortonworks DataFlow. The recording will be made available at http://hortonworks.com/partners/learn/Background For a primer on HDF, you can refer to the materials here to get a basic backgroundA basic tutorial on using Nifi on HDP sandbox is also available hereGoals Build Nifi flow to analyze Nifi's network traffic using tcpdump. Use Expression Language to extract out source/target IPs/portsBuild and use custom tcpdump processor to filter Nifi's source/target IPs/ports on HDP sandboxNote that: Nifi can be installed independent of HDPThe custom processor also can be built on any machine where Java and eclipse are installedSandbox is being used for demo purposes, to have everything in one placePre-Requisites: Install Nifi on sandbox The lab is designed for the HDP Sandbox. Download the HDP Sandbox here, import into VMWare Fusion and start the VMAfter

This document is an informal guide to setting up a test cluster on Amazon AWS, specifically the EC2 service. This is not a best practice guide nor is it suitable for a full PoC or production install of HDP.Please refer to Hortonworks documentation online to get a complete set of documentation.Create InstancesCreated the following RHEL 6.3 64bit instances:m1.medium ambarimasterm1.large hdpmaster1m1.large hdpmaster2m1.medium hdpslave1m1.medium hdpslave2m1.medium hdpslave3Note: when instantiating instances, I increased the root partition to 100Gb on each of them. For long term use, you may want to create separate volumes for your each of the datanodes to store larger amounts of data. Typical raw storage per node is 12-24Tb per slave node.Note: I edit the Name column in the EC2 Instances screen to the names mentioned above so I know which box I’m dealing withConfigure Security GroupsUsed the following security group rules: ICMP Port (Service) Source Action ALL sg-79c54511 (had

Apache JIRA HDFS-9552 is a documentation patch that clarifies exactly what kinds of permission checks are enforced by HDFS for each kind of user operation on a file system path. This documentation is not yet live on hadoop.apache.org. I am posting the same content here in the interim until HDFS-9552 ships in an official Apache release.Permission ChecksEach HDFS operation demands that the user has specific permissions (some combination of READ, WRITE and EXECUTE), granted through file ownership, group membership or the other permissions. An operation may perform permission checks at multiple components of the path, not only the final component. Additionally, some operations depend on a check of the owner of a path.All operations require traversal access. Traversal access demands the EXECUTE permission on all existing components of the path, except for the final path component. For example, for any operation accessing /foo/bar/baz, the caller must have EXECUTE permission on /, /foo and

By default, Ranger uses the log4j DailyRollingFileAppender to manage rotation of log files. This is the appender used by many HDP components, and it does not have a concept of max number of files to keep around. The advantage of using this appender is that it is easier to locate a log entry since files are split up on date boundaries. The main disadvantage of this appender is that there is no way to limit the amount of log data that is kept.There are two common options for dealing with this.1. Keep the DailyRollingFileAppender, and trim logs using a cron script. The following script would only keep 30 days worth of logs:#!/bin/bashfind /var/log/ranger -mtime +30 | xargs --no-run-if-empty rmYou can change the +30 to be +number_of_days.This script could be installed, for example, in a file called /etc/cron.daily/trim_ranger_logs and it would run each morning and remove logs older than 30 days.2. Change to the RollingFileAppender. To do this you need to modify the log4j.xml file for the a

Q: What are the use cases for Centrify?A: Integrate AD with Linux and there are no local user deployment similar to how SSSD is configured. The size of the cluster and/or domains is big enough that it's hard to manage with SSSD. Centrify greatly simplifies the management of this type of environment. Centrify ldapproxy also abstracts the complexity around integrating with other filers/appliances like Isilon, NetApp, etc. ldapproxy fronts Hadoop and doesn't expose internals of AD but only give information about a zone. It is usually used for machine to machine type authentication. Q: If there are multiple domains in a forest, how does Centrify know which domain controller to use to authenticate a user?A: Centrify walk the forest tree and figure out what domain controller to use to authenticate the user. It utilizes Domain Controller Service to perform this action. It doesn't use krb5.conf file. The Centrify agent knows what forest or domain controller it belongs to. It is PAM and sit

The release of HDP 2.3.4 offers a newer version of the ATS which is version 1.5. Part of configuring this new ATS version includes providing the following property if Tez is used:<property> <name>yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes</name> <value>org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl</value> <description>Plugins that can translate a timeline entity read request into a list of timeline cache ids, separated by commas. </description> </property>If this property is included but the tez-client is not present, a message will be seen in the ATS log stating ClassNotFound and the ATS server will fail shortly after issuing a startup.The full documentation for configuring ATS 1.5 is located here:http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_yarn_resource_mgt/content/ch08s02.htmlA documentation bug has been filed as the only reference to this in the configuration section