Resources

Objectives Upon completion of this tutorial, you should have a 3 node cluster of NiFi running on CentOs 7.2 using Vagrant and VirtualBox.Prerequisites You should already have installed VirtualBox 5.1.x. Read more here: VirtualBox You should already have installed Vagrant 1.8.6. Read more here: Vagrant NOTE: Version 1.8.6 fixes an annoying bug with permissions of the authorized_keys file and SSH. I highly recommend you upgrade to 1.8.6. You should already have installed the vagrant-vbguest plugin. This plugin will keep the VirtualBox Guest Additions software current as you upgrade your kernel and/or VirtualBox versions. Read more here: Vagrant vbguest plugin You should already have installed the vagrant-hostmanager plugin. This plugin will automatically manage the /etc/hosts file on your local mac and in your virtual machines. Read more here: Vagrant hostmanager pluginScope This tutorial was tested in the following environment: Mac OS X 10.11.6 (El Capitan) VirtualB

Observed in Teradata's distribution of Hortonworks, when ACID is enabled via Ambari, following error can show up.Error: java.lang.AbstractMethodError: Method org/postgresql/jdbc3/Jdbc3ResultSet.isClosed()Z is abstract (state=,code=0)org.apache.hive.service.cli.HiveSQLException: java.lang.AbstractMethodError: Method org/postgresql/jdbc3/Jdbc3ResultSet.isClosed()Z is abstract at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:258) at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:244) at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:247) at org.apache.hive.beeline.Commands.execute(Commands.java:848) at org.apache.hive.beeline.Commands.sql(Commands.java:713) at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:983) at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:718) at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:767) at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:485) at org.apache.hive.beeline.BeeLine.main(Be

SYMPTOM: Hive query through knox fails quickly with a a 500 error. ERROR: You will find something like this in the gateway-audit.logdispatch|uri|http://<hostname>:10000/cliservice?doAs=dav|success|Response status: 500 ROOT CAUSE: Size of query being sent is larger than default replayBufferSize. RESOLUTION: Change your topology to contain the following parameter in the service that is having the issue.<param><name>replayBufferSize</name><value>16</value></param> For example if I had this issue in hive it would look something like:<service> <role>HIVE</role> <url>$HTTP_SCHEME://$HIVE_HOST:10001/cliservice</url> <param><name>replayBufferSize</name><value>16</value></param> </service> The Value of the replayBufferSize is in KB and has a default of 8KB. Make sure the value is larger than your largest query size in order to get past this issue.

SysDigSysDig (Github) is an open source tool that allows for the exploration, analysis and trouble shooting of Linux systems and containers. It is well documented and very easy to install and use. It can be used for container and Linux system diagnostics, security analysis, monitoring and basic system information capture. Remember that sysdig can produces thousands of lines of messages and can continue doing so forever depending on the options selected. Check out the examples and read through all the options, you can monitor a ton of data really fast and also check for security anomalies.NiFi Ingesting SysDigSysdig can produce amazing amounts of logs. I chose to ingest 1 second chunks as ASCII JSON. I selected those options and listed them below. The results are arrays of JSON. I decided it's best to save them as a large JSON files for now and convert them to ORC later for Hive analysis in Zeppelin. You could also split them into individual JSON rows and process those.

A few weeks ago, we announced the availability of Hortonworks Connected Data Cloud for AWS Technical Preview. We have just updated the Technical Preview (to RC#36) with the following new features:Ability to manage cluster templates to make re-using cluster configurations a whole lot easierAbility to set custom properties on cluster create to fine-tune your cluster configurationCheckout the Release Notes for more details:http://hortonworks.github.io/hdp-aws/releasenotes/Get Started and Launch Hortonworks Cloud in your AWS environment:http://hortonworks.github.io/hdp-aws/Check it out. You can provide feedback and ask questions here at the Hortonworks Community Connection:http://hortonworks.github.io/hdp-aws/feedback/#how-to-create-a-questionHave fun!

Below are the steps for Oozie database migration from Derby to Mysql. Step 1 - Have MySQL server installed and ready to be configured. .Step 2 - Stop Oozie service from Ambari UI. .Step 3 - On Ambari Server, run below command ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar Note - Please pass appropriate driver if /usr/share/java/mysql-connector-java.jar does not exists. . .Step 4 - Login to Mysql server as root user and create a blank 'oozie' database and grant required permissions to the 'oozie' user. #create database oozie; #grant all privileges on oozie.* to 'oozie’@‘<oozie-server>' identified by 'oozie’; Note - 'oozie' is your oozie database password, if you want you can change it in above command . .Step 5 - Add mysql database server details in Oozie configuration via Ambari UI..Step 6 - Make sure that you have mysql connector jar under Oozie libext (/usr/hdp/<version>/oozie/libext/mysql-connector-java.jar), if not then cop

We just updated Hortonworks Data Cloud for AWS to Technical Preview #3. This update includes the following new features: Ability to use an Existing VPC for the Cloud Controller and your clusters.Centrally manage your Hive Metastore database instances, to make re-using the Metastore between clusters much easier.Optionally receive an email notification once your cluster is ready.Checkout the Release Notes for more details:http://hortonworks.github.io/hdp-aws/releasenotes/Get Started and Launch Hortonworks Data Cloud in your AWS environment:http://hortonworks.github.io/hdp-aws/Check it out. You can provide feedback and ask questions here at the Hortonworks Community Connection:http://hortonworks.github.io/hdp-aws/feedback/#how-to-create-a-questionHave fun!

IntroductionHow many times we've heard that a problem can't be solved if it isn't completely understood? However, all of us, even when having a lot of experience and technical skill, we still tend to jump to the solution workspace by-passing key steps in the scientific approach. We just don’t have enough patience and, quite often, the business is on our back to deliver a quick fix. We just want to get it done! We found ourselves like that squirrel in the Ice Age chasing for the acorn just using our experience and minimal experiment, dreaming that the acorn will fall on our lap. This article is meant to re-emphasize a repeatable approach that can be applied to tune Storm topologies starting from understanding the problem, forming a hypothesis, testing that hypothesis, collecting the data, analyzing the data, drawing conclusions based on facts and less on empirical. Hopefully it helps to catch more acorns!Understand the ProblemUnderstand the functional problem resolved by the topology; q

SYMPTOM : Immediately after exporting HDFS directories via NFS , some of the directories start throwing permission denied errors to authorized users added in Ranger policies.ROOT CAUSE : NFS neither honors Ranger policies nor HDFS ACLs. If a directory has HDFS permission bits such as 000 and access is controlled fully via Ranger, this directory won’t be exported at all. Messages such as below can be seen in NFS gateway logs :- 2016-07-27 17:35:19,071 INFO mount.RpcProgramMountd (RpcProgramMountd.java:mnt(127)) - Path /test1 is not shared. 2016-07-27 17:35:37,297 INFO mount.RpcProgramMountd (RpcProgramMountd.java:mnt(127)) - Path /test2 is not shared. 2016-07-27 17:39:34,581 INFO mount.RpcProgramMountd (RpcProgramMountd.java:mnt(144)) - Giving handle (fileId:12345) to client for export /Even if the directory gets exported due to some available permissions, effective permission bits are only from HDFS and not from Ranger policies.

SYMPTOMS : Errors such as "KeeperErrorCode = NoAuth for /config/topics"ROOT CAUSE : Errors such as above are reported while trying to create or delete topic from an ordinary user because only the process owner of Kafka service such as root can write to zookeeper znodes i.e. /configs/topics.Ranger policies do not get enforced when a non privileged user creates a topic is because kafka-topics.sh script talks directly to zookeeper in order to create the topics. It will add entries into the zookeeper nodes and the watchers on the broker side will monitor and create topics accordingly. Due to this process involving zookeeper, the authorization cannot be done through the ranger plugin. NEXT STEPS : If one would want to allow users to be able to create topics, We have a script called kafka-acls.sh which would help allow or deny users on topics and many such options. The details are elaborated in the document mentioned below :- http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_secure-

SizingThere's no simple rule of thumb for this, it's as much an art as it is a science, as it depends on the workloads and how chatty they are with your current ZKs.One way to look at this is:If you have 3 ZKs you can afford to lose one, if you have 5 you can afford to lose two.If your IT is aggressively applying security patches and other upgrades, like firmware, kernel, Java, other packages used by Hadoop tools, and taking nodes down to do the job, then during those upgrades with 3 ZKs, you ZK runs with only two nodes, and if you are unlucky and one of them goes down, then your whole cluster will go down. So, in this case 5 are better.Warning: the more ZK nodes you have, the slower the ZK becomes for writes.PlacementZookeeper is a master node, as such it can be collocated with other master services. Ideally, you would not want to collocate it with an HA service. It is quite light on memory and CPU requirements, but since is disk intensive, don't collocate it with disk-intensive servi

Disable Transparent Huge Pages (THP)Transparent Huge Pages (THP) is a Linux memory management system that reduces the overhead of Translation Lookaside Buffer (TLB) lookups on machines with large amounts of memory by using larger memory pages.However THP feature is known to perform poorly in Hadoop cluster and results in excessively high CPU utilization.Disable THP to reduce the amount of system CPU utilization on your worker nodes. This can be done by ensuring that both proc entries are set to [never] instead of [always].Use Recommended File System TypesSome file systems offer better performance and stability than others. As such, the HDFS dfs.datanode.data.dir and YARN yarn.nodemanager.local-dirs should be configured to use mount points that are not formatted with the most optimal file systems. Take a look at this article on file system choices: https://community.hortonworks.com/articles/14508/best-practices-linux-file-systems-for-hdfs.htmlDisable Host SwappinessThe Linux kernel pr

Linux ACLs are implemented in such a way that setting default ACLs on parent directory shall automatically get inherited to child directories and umask shall have no influence in this behavior. However HDFS ACLs have slightly different approach here, they do take into account umask set in hdfs-site.xml in parameter "fs.permissions.umask-mode" and enforce ACLs on child folders based on these two parameter with umask taking precedence over the other.Let's try and reproduce this case :- [gaurav@test ~]$ fs -mkdir /tmp/acltest [gaurav@test ~]$ fs -setfacl -m default:mask::rwx /tmp/acltest [gaurav@test ~]$ fs -setfacl -m mask::rwx /tmp/acltest [gaurav@test ~]$ fs -setfacl -m default:user:adam:rwx /tmp/acltest [gaurav@test ~]$ fs -setfacl -m user:adam:rwx /tmp/acltest Let's see what ACLs are implemented :- [gaurav@test~]$ fs -getfacl /tmp/acltest # file: /tmp/acltest # owner: gaurav# group: hdfs user::rwx user:adam:rwx group::r-x mask::rwx other::r-x default:user::rwx default:user:adam:rwx d

Below are the steps to run Hive(TEZ) query in a shell script using Oozie shell action.1. Configure job.propertiesExample:#*************************************************# job.properties#*************************************************nameNode=hdfs://<namenode-fqdn>:8020jobTracker=<resourcemanager-host-fqdn>:8050queueName=defaultexamplesRoot=examplesoozie.use.system.libpath=trueoozie.wf.application.path=${nameNode}/user/${user.name}/.2. Configure Workflow.xmlExample:<?xml version="1.0" encoding="UTF-8"?><workflow-app xmlns="uri:oozie:workflow:0.4"name="test-shell-with-kerberos-wf"> <global> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> <property> <na

HiveServer2 does not expose jmx or equivalent metrics but often we require to monitor the Hiveserver2 process to know cpu usage, heap usage and state of various thread along with some other params like no of class loaded by hs2 jvm. it is also very useful if you want to run some cpu/memory sampling to profile hiveserver2.Step 1: login into Ambari and open hive advanced configurationStep 2: expand advanced hive-env settings and add the following configuration in hive-env templateif [ "$SERVICE" = "hiveserver2" ]; then export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8008"fiStep 3: Restart HiveServer2Step 4: on client machine open jvisualvm($JAVA_HOME/bin), once open go to File -> Add JMX Connection -> enter hostname and port no(it's 8008 in this case) and you are good to go.

If you are not using any Hive Authorization like SQL Standard Authorization or Ranger and you don't want other to read/write data and allow the only owner to read & write, and allowing the group to read.Changing the warehouse directory permission.1.From the command line in the Ambari server node, edit the file/var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.pySearch for hive_apps_whs_dir which should go to this block: params.HdfsResource(params.hive_apps_whs_dir, type="directory", action="create_on_execute", owner=params.hive_user, group=params.user_group, mode=0755 ) Modify the value for mode from 0755 to the desired permission, for example, 0777.Save and close the file.Restart the Ambari server to propagate the change to all nodes in the cluster:ambari-server restartIt may take a few seconds to update the file in the Ambari agents on all nodes. To verify if the change has been applied on a particular node, check the content of

I recently ran into a situation where I had enabled HDFS HA and later had to change the value of dfs.nameservices. So basically during HA setup I set the value for dfs.nameservices to "MyHorton", but a couple hours later realized I should have used "MyCluster" instead.This article explains how you can change the dfs.nameservices value after HDFS HA has been enabled already. Background: What is the purpose of dfs.nameservices? Its the logical name of your HDFS nameservice. Its important to remember that there are several configuration parameters that have a key, which includes the actual value of dfs.nameservices, e.g. dfs.namenode.rpc-address.[nameservice id].nn1Preparation: Put your HDFS in safemode and backup the namespace (https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#dfsadmin; dfsadmin -safemode enter; dfsadmin -saveNamespace); Stop Namenode service Backup Hive Metastore (mysqldump hive > /tmp/mydir/backup_hive.sql) Change Configuration

Although it is a simple solution, there is not much reference to this exact problem in the www.We have seen people doing multiple projection, flattening and group by to get the data into shape for storing into the final table using pig after the aggregation and group operations.Source Table:CREATE EXTERNAL TABLE IF NOT EXISTS source(pk1 string,pk2 string,agg1 INT,agg2 INT)STORED AS ORC tblproperties("orc.compress"="ZLIB");Data:Target Table:CREATE EXTERNAL TABLE IF NOT EXISTS target_aggregated(pk1 string,pk2 string,sum_agg1 BIGINT,sum_agg2 BIGINT)STORED AS ORC tblproperties("orc.compress"="ZLIB");Pig Script:--Load the data in pig relationsstaging = LOAD 'DEFAULT.SOURCE' USING org.apache.hive.hcatalog.pig.HCatLoader();--Group the datagroup_staging = group staging BY (pk1,pk2);--Flatten the grouped data and generate aggregates with same attribute names as the target tablecalculate_group_staging = FOREACH group_staging GENERATE FLATTEN(group) AS(pk1,pk2),SUM(staging.agg1) as sum_agg1, SU

Pre-requisite :Remove the incubator-ranger project from eclipse if it's already there and close the eclipse. Clone incubator-ranger repo in the local machine from GitHub. Ignore this step if you have already cloned the required ranger branch. To check out the code from Apache Incubator-ranger GIT repository refer below command. For example :git clone https://github.com/apache/incubator-ranger.git Set JAVA_HOME in the environment variable.For example : export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64 Install maven 3.3.9 or higher. Ignore this step if required version of maven is already installed. To get the installed maven version try command : mvn -versionFor example : cd /usr/local wget http://mirror.fibergrid.in/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz tar xvf apache-maven-3.3.9-bin.tar.gzSet maven command line tool in the system path environment variable:For example : export M2_HOME=/usr/local/apache-maven-3.3.9export M2=$M2_HOME/binexport PATH=$M2:

To start pulling jmx metrics from hadoop you need to first enable it via jvm parameters. Go to ambari-->Yarn-->config-->yarn-envEnable jmx by adding the following params to YARN_RESOURCEMANAGER_OPTS"-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8001"Set the port to whatever port is available on your clusterSave the new config. This will require restart of yarn. Now you have jmx enabled but you need to client to start pulling jmx metrics. Go to your data node and and download (from here http://wiki.cyclopsgroup.org/jmxterm/download.html) the latest jxmterm-xxx-xxx-uber.jar. For this article I use jmxterm-1.0-alpha-4-uber.jarOnce you have the jmxterm client downloaded lets connect to jxm using java -jar jmxterm-1.0-alpha-4-uber.jar -l service:jmx:rmi:///jndi/rmi://localhost:<YOURPORT>/jmxrmiIn this example I am set port to 8012.java -jar jmxterm-1.0-alpha-4-u

Sometimes you have Java messages that you would like to easily ingest into HDFS or perhaps HDFS as raw files, Phoenix, Hive and other destinations. You can do that pretty easy with Apache NiFi 1.0.0 as part of HDF 2.0. For this simple example, I also added a REST gateway for bulk loading, testing and to provide another way to easily send JMS messages. ListenHTTP accepts HTTP POSTS on port 8099, which I made the listener port for that processor. It takes what you send and publishes that to a JMS queue. I am using ActiveMQ.I have a little Python 2.7 script that I found on github that makes fake log records and modified it to send 1,000 JSON messages via REST to our REST to JMS gateway in NIFI for testing. You can easily do this with shell script and CURL, Apache JMeter, Java code, Go script and many other open source REST testers and clients.url = 'http://server.com:8099/contentListener'r = requests.post(url, json={"rtimestamp": timestamp, "ip": random_ip(), "country": country,

To make it easier for you to find the information that you need, we significantly reorganized SmartSense 1.3.0 documentation: SmartSense Installation introduces the new SmartSense 1.3.0 architecture, and guides you through the installation and upgrade process.SmartSense User Guide guides you as you’re using SmartSense functionality, including accessing and using the new Activity Explorer.ReleaseNotes informs you about new features, fixed issues, and known issues.Check out these updates and help us improve the documentation by providing your feedback on HCC.Thanks! Hortonworks Technical Documentation Team

GoalsUse custom kerberos Principal for configuring HAWQ loginVerifying the usage by creating, loading and query an object with non-default principalNOTE HAWQ does not allow these parameters to be changed during the installation as configuration files are only generated after initialization of the HAWQ init process, so this is applicable only when you have a HAWQ cluster up and running with kerberos enabled. Steps Part 1: KerberosEnsure that the HAWQ is up and runningLogin to kerberos server and create a custom headless principalkadmin.local: addprinc -randkey custom-postgres@HORTONWORKS.COMWARNING: no policy specified for custom-postgres@HORTONWORKS.COM; defaulting to no policyPrincipal "custom-postgres@HORTONWORKS.COM" created.Export the key into a keytab file, ensure that the name remains as "hawq.service.keytab" or else this will be required to change kadmin.local: xst -k hawq.service.keytab custom-postgres@HORTONWORKS.COM....kadmin.local: quitCopy over the keytab file over to al

Hello Team, I want to update my profile, I want to change company name, job profile of my registered email id. Can you please share how can I do it? I need to change this as this is required in my company to get access to cloudera support portal. I registered with few wrong information and due to which I am not able to get approvals from our management team. Please suggest how can I do these changes? Thanks Ankur

Configuring HTTPFSTo configure HTTPFS you can refer below post, it clearly explains the steps. We can install httpfs in any host where Hadoop clients are installed and we call it <HTTPFS_HOST>.HTTPFS - Configure and Run with HDPCreate topology fileIn Knox Gateway host under /etc/knox/conf/topologies directory create a new topology file (copy default.xml) say example.xml After copying it looks like below [root@test-3 topologies]# ls -l /etc/knox/conf/topologiestotal 20-rw-r--r--. 1 knox knox 89 Oct 4 11:25 README-rw-r--r--. 1 knox knox 4422 Oct 4 11:25 admin.xml-rw-r--r--. 1 knox knox 3026 Oct 12 10:22 default.xml-rw-r--r--. 1 knox knox 3026 Oct 12 10:35 example.xmlIn example.xml topology file for WEBHDFS role change host to <HTTPFS_HOST> and port to 14000 . By default HTTPFS_HTTP_PORT is 14000 <service> <role>WEBHDFS</role> <url>http://<HTTPFS_HOST>:14000/webhdfs</url> </service&