Community Articles

1. Lets assume you have HDP cluster installed and managed by Ambari.2. When we want to delete a service [either Custom service or HDP service] using api, you generally use below command -curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://<ambari-server>:8080/api/v1/clusters/c1/services/<SERVICENAME>;2. After executing above command you might see below error -$curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://<ambari-server>:8080/api/v1/clusters/c1/services/HBASE{ "status" : 500, "message" : "org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Cannot remove HBASE. Desired state STARTED is not removable. Service must be stopped or disabled." 3. If you see the above error while Removing/Stopping service please use below steps to resolve the issue4. Login to ambari database [In my case its postgresql] and check the values of the service in below tables -# psql -U ambari[Default password is 'bigdata']

Objective Many people work exclusively from a laptop where storage space is typically limited to 500GB of space or less. Over time, you may find your available storage space has become a regular concern. It's not uncommon to use an external hard drive to augment available storage space. The current version of Docker for Mac (1.12.x) does not provide a configuration setting which allows users to change the location where the Docker virtual machine image is located. This means the image, which can grow up to 64GB in size by default, is located on your laptop's primary hard drive. With the HDP 2.5 version of the Hortonworks sandbox available as a native Docker image, you may find a desire to have more room available to Docker. This tutorial will guide you through the process of moving your Docker virtual machine image to a different location, an external drive in this case. This will free up to 64GB of space on your primary laptop hard drive and let you expand the size of the Docker image

As a follow-up to my article on downloading and playing MIDI files (https://community.hortonworks.com/content/kbentry/65154/nifi-1x-for-automatic-music-playing-pipelines.html) This quick article highlights how you can support both MP3 and MIDI. As you can see it's very easy to have multiple channels in, multiple processing paths and process many types of inputs and files in NIFI.So how do we add MP3 playback to our Jukebox?Add another GetHTTP processorhttp://192.168.1.2:8080/extract/url?url=http://www.amclassical.com/piano//&type=mp3Still using my Microservice to convert web pages into links in JSON, this time filtering out MP3s. Fortunately there's a few web pages out there with free classical MP3s.Next after UpdateAttribute, I check file extension in a Route on Attribute.${filename:contains('midi')}If it's MIDI, call the same MIDI player.I added a second PutFile to save to /opt/demo/mp3, so that I keep my music files seperately.Otherwise, call OSX's command line MP3 player.In

Use CaseBefore meetups, I wanted to play some music. NiFi seemed like a great choice for streaming free music through my Mac. MIDI Command Line Player For OSXbrew install timidityThen you can simply play MIDI files with timidity file.mid.Microservice to Extract Links from Web PagesJava 8 Source Code: https://github.com/tspannhw/linkextractorThe Spring Boot REST API accepts a URL, parses out mid files and returns JSON containing linking and descriptions. Example REST Call to Servicecurl -G -v "http://<urL>:8080/extract/url?url=http://www.midiworld.com/classic.htm/&type=mid"Run the Microservice java -Xms512m -Xmx2048m -Djava.net.preferIPv4Stack=true -jar target/linkextractor-0.0.1-SNAPSHOT.jarJava Snippet Using JSoup to extract links from URL (HTML)pLink = new PrintableLink(); pLink.setLink(link.attr("abs:href")); pLink.setDescr(trim(link.text(), 100)); linksReturned.add(pLink); Outputhdfs dfs -ls /music/*.mid-rw-r--r-- 3 tspann hdfs 87 2016-11-0

On Oct 25th, EMC Isilon announced general availability of OneFS 8.0.1. The official announcement can be found here - https://isiblog.emc.com/category/onefs-8-0-1/OneFS 8.0.1 is the first major upgrade to OneFS 8.0 platform that was released in Feb 2016. While this upgrade has many generic enhancements to the storage side of OneFS, there are four major functionalities related to Hadoop Supports Ambari 2.4This is the first major release of OneFS that will have support for Ambari 2.4. Hortonworks had released HDP 2.5 and Ambari 2.4 at end of Aug 2016. Since then Isilon team had been working to integrate Ambari 2.4 with OneFS. With the release one OneFS 801, customers can now use Ambari 2.4 and HDP 2.5For more instructions on how to configure, see the below link : https://community.emc.com/community/products/isilon/blog/2016/10/24/onefs-and-ambari-24-hdp-25 Ambari Metrics and Alerts integration Though OneFS has been supporting Ambari for a few releases, it was not leveraging

Objective In many organizations "search" is a common requirement for a user friendly means of accessing data. When people thing of "search", they often think of Google. Many organizations use Solr as their enterprise search engine. It is commonly used to power public website search from within the site itself. Organizations will often build custom user interfaces to tailor queries to meet their internal or external end-user needs. In most of these scenarios, users are shielded from the complexity of the Solr query syntax. Solr has a long list of features and capabilities; you can read more here Apache Solr Features. Solr 6 has a new feature which allows you to submit SQL queries via JDBC. This opens up new ways to interact with Solr. Using Zeppelin with SQL is now possible because of this new feature. This should make you more productive because you can use a language syntax with which you are already familiar: SQL! This tutorial will guide you through the process of updating the Z

Starting My Hadoop ToolsNiFi can interface directly with Hive, HDFS, HBase, Flume and Phoenix. And I can also trigger Spark and Flink through Kafka and Site-To-Site. Sometimes I need to run some Pig scripts. Apache Pig is very stable and has a lot of functions and tools that make for some smart processing. You can easily augment and add this piece to a larger pipeline or part of the process.Pig SetupI like to use Ambari to install the HDP 2.5 clients on my NiFi box to have access to all the tools I may need.Then I can just do:yum install pigPig to Apache NiFi 1.0.0ExecuteProcessWe call a shell script that wraps the Pig script.Output of script is stored to HDFS: hdfs dfs -ls /nifi-logs Shell Scriptexport JAVA_HOME=/opt/jdk1.8.0_101/ pig -x local -l /tmp/pig.log -f /opt/demo/pigscripts/test.pig You can run in different Pig modes like local, mapreduce and tez. You can also pass in parameters or the script.Pig Scriptmessages = LOAD '/opt/demo/HDF/centos7/tars/nifi/nifi-1.0.0.2.0

IntroductionThis is a continuation of Apache Storm Tuning Approach for Squirrels article that I published couple weeks ago.A topology is going to have to coexist on a Storm cluster with a variety of other topologies. Some of those topologies will burn CPU doing heavy calculations, others will consume large amounts of network bandwidth.No one sane can claim to provide a silver bullet for how to set up your cluster for the best performance. However, I’d like to share a few recipes and guidelines for dealing with issues as they arise. I listed below 6 categories of contention, describing common problems and strategies to alleviate or eliminate resource contention. I hope that this article at least provides a good map for a deep dive onto Storm tuning. Each resource contention is probably worth it an article.Worker Processes in a ClusterA Storm cluster is installed with a fixed number of available worker processes across all worker nodes. Each time you deploy a new topology to the cluster,

The Ranger KMS logs are present cd /usr/hdp/current/ranger-kms/ewslrwxrwxrwx 1 kms kms 19 sep 23 20:33 logs-> /var/log/ranger/kmsWe would like to change ranger KMS logs location from /var/log/ ranger/kms to /hadoop/log/ranger/kms.mkdir /hadoop/log/ranger/kmschmod 755 /hadoop/log/ranger/kmschown kms:kms /hadoop/log/ranger/kmsrm symbolic link for logs cd /usr/hdp/current/ranger-kms/ vi ranger-kmssearch for word "TOMCAT_LOG_DIR" and replace with new location.TOMCAT_LOG_DIR=/hadoop/log/ranger/kms In Ambari webUI update log directory for ranger KMS kms_log_dir property. Restart KMS

To setup an Ambari managed HDP cluster, below services’ components require database for their respective metastore databases. Ambari ServerHive OozieRanger While a lab/sandbox environment can be setup with default databases for these components, using the same is strongly not recommended for Dev/QA/UAT/Production clusters. Due diligence and planning must be done to ensure that database selection is appropriate for enterprise standard production cluster. Below are key areas to be taken into consideration while planning to select a database for Ambari and HDP components.Supported DatabasesRefer below for supported databases for Ambari and different HDP components in current state.https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-installation/content/database_requirements.html High Availability SupportIt is strongly recommended to setup High Availability for different HDP components, which can be enabled for High Availability in a production cluster.High Availability is s

Objective Cross Data Center Replication, commonly abbreviated as CDCR, is a new feature found in SolrCloud 6.x. This feature enables Solr to replicate data from one source collection to one or more target collections distributed between data centers. The current version provides an active-passive disaster recovery solution for Solr. Data updates, which include adds, updates, and deletes, are copied from the source collection to the target collection. This means the target collection should not be sent data updates outside of the CDRC functionality. Prior to SolrCloud 6.x you had to manually design a strategy for replication across data centers. This tutorial will guide you through the process of enabling CDCR between two SolrCloud clusters, each with 1 server, in a Vagrant + VirtualBox environment. NOTE: Solr 6 is being deployed as a standalone application. HDP 2.5 provides support for Solr 5.5.2 via HDPSearch which does not include CDCR functionality.Prerequisites You should have alre

Demo Extract data from images and store in HDFS. Documents with size less than 10mb stores into HBase. Document > 10mb lands into HDFS with metadata into HBase Part 1 - https://www.linkedin.com/pulse/cds-content-data-store-nosql-part-1-co-dev-neeraj-sabharwal

Step1: Please allow below ports from your OS firewall for Ambari.https://ambari.apache.org/1.2.5/installing-hadoop-using-ambari/content/reference_chap2_7.html.Step2: Please go though the required component and allow below ports from your OS firewall for HDP.https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_HDP_Reference_Guide/content/accumulo-ports.html.Step 3: In order to allow YARN jobs to run successfully, we need to add custom TCP port range to the YARN configuration.Login to Ambari UI --> Select Mapreduce2 --> Configs --> Customer mapred-site --> Add/Modify below propertyyarn.app.mapreduce.am.job.client.port-range=32000-65000 Notes: 1. 32000-65000 is the port range which will be used by Application Master to connect to Node Managers. .2. You can increase the number of ports based on job volume. .How to add exception in Centos7 firewall?Example for Step 3.#firewall-cmd --permanent --zone=public --add-port=32000-65000/tcp#firewall-cmd --reload Please comment i

HDP Sandbox comes pre-installed with SparkR and R . First let's setup the R Studio on the HDP Sandbox:   To download and install RStudio Server open a terminal window and execute the commands corresponding to get the 64-bit version wget https://download2.rstudio.org/rstudio-server-rhel-0.99.893-x86_64.rpmsudo yum install --nogpgcheck rstudio-server-rhel-0.99.893-x86_64.rpmsudo rstudio-server verify-installationsudo rstudio-server stopsudo rstudio-server start Now setup a local user on the hdp sandbox, to access the rstudio: useradd alexpasswd xxxx Next launch web-browser and point to : http://sandbox.hortonworks.com:8787/ Login using the local account created earlier above. Next lets initialize the sparkr in "yarn-client" mode Sys.setenv(SPARK_HOME="/usr/hdp/current/spark-client/").libPaths(c(file.path(Sys.getenv("SPARK_HOME"),"R","lib"),.libPaths()))library(SparkR)sc <- SparkR::sparkR.init(master = "yarn-client") Run the code in rstudio , you should get the following output:

Note: I used TinyCert to generate my own certificate authority and certificates. Probably the easiest way to generate new certificates Create your own certificate authority. Fill in your own values. Create 2 new certificates. One is your client certificate (in this case, bbukacek) and a server certificate which will be used for the NiFi keystore.Client CertificateServer Certificate Download the certificate authority, the client and server certificates and upload to your NiFi environment.Note: Easiest way is to download the client and server certificates in PKCS12 format Create a TrustStore We will use the certificate authority to create the TrustStore. Use the following syntax to create the TrustStore:> keytool -import -file <ca_file>.pem -alias cacert -keystore truststore.jks -storepass <password> Configure NiFi Now that the TrustStore is created we can configure NiFi Keystore and TrustStore.Under $NIFI_HOME/conf, open the nifi.properties.Set the web properties First an

ProblemUsing rhive with following parameters to connect to hivelibrary(RHive)Sys.setenv(HADOOP_CONF_DIR="/etc/hadoop/conf")Sys.setenv(RHIVE_HIVESERVER_VERSION="2")Sys.setenv(HIVE_HOME="/usr/hdp/current/hive-client")Sys.setenv(HADOOP_HOME="/usr/hdp/current/hadoop-client")rhive.init()> rhive.connect()2016-10-26 13:06:22,836 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1173)) - fs.default.name is deprecated. Instead, use fs.defaultFS2016-10-26 13:06:23,720 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable2016-10-26 13:06:24,631 WARN [main] shortcircuit.DomainSocketFactory (DomainSocketFactory.java:<init>(117)) - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.2016-10-26 13:06:25,237 INFO [Thread-5] jdbc.Utils (Utils.java:parseURL(309)) - Supplied authorities: 127.0.0.1:10000

Step1). Get all the properties from the below command: (Here : "http://jss1:8080" is the URL where ambari server is running)curl --user admin:admin -i -H 'X-Requested-By: ambari' -X GET http://jss1:8080/api/v1/clusters/JoyCluster/configurations?type=cluster-env\&tag=version1Step2). Save the output from above command to a file. Or just open the URL in browser and save it to a file:http://jss1:8080/api/v1/clusters/JoyCluster/configurations?type=cluster-env&tag=version1.For example here i got the following output from the above URL:{ "href" : "http://jss1:8080/api/v1/clusters/JoyCluster/configurations?type=cluster-env&tag=version1", "items" : [ { "href" : "http://jss1:8080/api/v1/clusters/JoyCluster/configurations?type=cluster-env&tag=version1", "tag" : "version1", "type" : "cluster-env", "version" : 1, "Config" : { "cluster_name" : "JoyCluster", "stack_id" : "HDP-2.4" }, "properties" : { "fetch_nonlocal_groups"

One of the great things about Apache NiFi is its ability to consume and transform many formats of data, however a particular area of complexity has been around the transformation of JSON data received (i.e. JSON-to-JSON transformation). Take the case of the GetTwitter processor that provides access to Twitter data streams (either from a filtered search, sample set, or firehose endpoints). For anyone who has worked with Twitter’s JSON schema it is very rich with detailed information for an individual tweet. There are cases where a lot of this data isn’t required for analytics and data flow managers or analysts are looking to pare it down to the necessities. There are also instances where incoming JSON data simply needs to be formatted or re-labeled for use in another system or repository, such as Hive, HBase, or MongoDB. Outside of NiFi, JSON-to-JSON transformation has been simplified in the Jolt Java library which offers a declarative approach to defining JSON output. Jolt provi

Objectives Upon completion of this tutorial, you should have a 3 node cluster of NiFi running on CentOs 7.2 using Vagrant and VirtualBox.Prerequisites You should already have installed VirtualBox 5.1.x. Read more here: VirtualBox You should already have installed Vagrant 1.8.6. Read more here: Vagrant NOTE: Version 1.8.6 fixes an annoying bug with permissions of the authorized_keys file and SSH. I highly recommend you upgrade to 1.8.6. You should already have installed the vagrant-vbguest plugin. This plugin will keep the VirtualBox Guest Additions software current as you upgrade your kernel and/or VirtualBox versions. Read more here: Vagrant vbguest plugin You should already have installed the vagrant-hostmanager plugin. This plugin will automatically manage the /etc/hosts file on your local mac and in your virtual machines. Read more here: Vagrant hostmanager pluginScope This tutorial was tested in the following environment: Mac OS X 10.11.6 (El Capitan) VirtualB

Observed in Teradata's distribution of Hortonworks, when ACID is enabled via Ambari, following error can show up.Error: java.lang.AbstractMethodError: Method org/postgresql/jdbc3/Jdbc3ResultSet.isClosed()Z is abstract (state=,code=0)org.apache.hive.service.cli.HiveSQLException: java.lang.AbstractMethodError: Method org/postgresql/jdbc3/Jdbc3ResultSet.isClosed()Z is abstract at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:258) at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:244) at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:247) at org.apache.hive.beeline.Commands.execute(Commands.java:848) at org.apache.hive.beeline.Commands.sql(Commands.java:713) at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:983) at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:718) at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:767) at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:485) at org.apache.hive.beeline.BeeLine.main(Be

SYMPTOM: Hive query through knox fails quickly with a a 500 error. ERROR: You will find something like this in the gateway-audit.logdispatch|uri|http://<hostname>:10000/cliservice?doAs=dav|success|Response status: 500 ROOT CAUSE: Size of query being sent is larger than default replayBufferSize. RESOLUTION: Change your topology to contain the following parameter in the service that is having the issue.<param><name>replayBufferSize</name><value>16</value></param> For example if I had this issue in hive it would look something like:<service> <role>HIVE</role> <url>$HTTP_SCHEME://$HIVE_HOST:10001/cliservice</url> <param><name>replayBufferSize</name><value>16</value></param> </service> The Value of the replayBufferSize is in KB and has a default of 8KB. Make sure the value is larger than your largest query size in order to get past this issue.

SysDigSysDig (Github) is an open source tool that allows for the exploration, analysis and trouble shooting of Linux systems and containers. It is well documented and very easy to install and use. It can be used for container and Linux system diagnostics, security analysis, monitoring and basic system information capture. Remember that sysdig can produces thousands of lines of messages and can continue doing so forever depending on the options selected. Check out the examples and read through all the options, you can monitor a ton of data really fast and also check for security anomalies.NiFi Ingesting SysDigSysdig can produce amazing amounts of logs. I chose to ingest 1 second chunks as ASCII JSON. I selected those options and listed them below. The results are arrays of JSON. I decided it's best to save them as a large JSON files for now and convert them to ORC later for Hive analysis in Zeppelin. You could also split them into individual JSON rows and process those.

A few weeks ago, we announced the availability of Hortonworks Connected Data Cloud for AWS Technical Preview. We have just updated the Technical Preview (to RC#36) with the following new features:Ability to manage cluster templates to make re-using cluster configurations a whole lot easierAbility to set custom properties on cluster create to fine-tune your cluster configurationCheckout the Release Notes for more details:http://hortonworks.github.io/hdp-aws/releasenotes/Get Started and Launch Hortonworks Cloud in your AWS environment:http://hortonworks.github.io/hdp-aws/Check it out. You can provide feedback and ask questions here at the Hortonworks Community Connection:http://hortonworks.github.io/hdp-aws/feedback/#how-to-create-a-questionHave fun!

Below are the steps for Oozie database migration from Derby to Mysql. Step 1 - Have MySQL server installed and ready to be configured. .Step 2 - Stop Oozie service from Ambari UI. .Step 3 - On Ambari Server, run below command ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar Note - Please pass appropriate driver if /usr/share/java/mysql-connector-java.jar does not exists. . .Step 4 - Login to Mysql server as root user and create a blank 'oozie' database and grant required permissions to the 'oozie' user. #create database oozie; #grant all privileges on oozie.* to 'oozie’@‘<oozie-server>' identified by 'oozie’; Note - 'oozie' is your oozie database password, if you want you can change it in above command . .Step 5 - Add mysql database server details in Oozie configuration via Ambari UI..Step 6 - Make sure that you have mysql connector jar under Oozie libext (/usr/hdp/<version>/oozie/libext/mysql-connector-java.jar), if not then cop

We just updated Hortonworks Data Cloud for AWS to Technical Preview #3. This update includes the following new features: Ability to use an Existing VPC for the Cloud Controller and your clusters.Centrally manage your Hive Metastore database instances, to make re-using the Metastore between clusters much easier.Optionally receive an email notification once your cluster is ready.Checkout the Release Notes for more details:http://hortonworks.github.io/hdp-aws/releasenotes/Get Started and Launch Hortonworks Data Cloud in your AWS environment:http://hortonworks.github.io/hdp-aws/Check it out. You can provide feedback and ask questions here at the Hortonworks Community Connection:http://hortonworks.github.io/hdp-aws/feedback/#how-to-create-a-questionHave fun!