Resources

Here are the Requirements:Total Data Size - Uncompressed: 13.5TB; Compressed: 2 TBLarge Virtual Fact Table, View containing a Union All of 3 Large Tables, 11 Billion Records in Total SizeAnother view taking the large virtual fact table, with consecutive Left Out Joins on 8 Dimension Tables, so that no matter what 11 Billion records is always the result.There is timestamp data that you can use to filter rows by.Suppose you were given the following. How would you begin configuring Hortonworks for Hive? Would you focus on storage? How can we configure for compute?Lets assume:Platform: AWSData Node Instance: r3_4xlargeCores: 16RAM: 122 GBEBS Storage: 2 x 1TB DisksSo where do we begin?First Some Quick Calculations:Memory per Core: 122GB/16 = 7.625; Approximate 8 GB per CPU CoreThis means our largest Container Size PER Node per core is 8 GB However we should not reserve all 16 Cores to Hadoop. Some Cores are need for OS and other processes.Let's Assume 14 Cores is reserved for YARN.Memory

It's easier than I would have thought to add images to your SQL results tables in Apache Zeppelin.It's pretty simple to do this in HDP 2.5's version of Apache Zeppelin. You use the %html tag to output HTML instead of text.Use Case: Displaying Image with TensorFlow Inception Image Recognition Results in Same ListExample SQL:SELECT user_name, handle, concat('%html <img width=50 height=60 src="', media_url, '">') as media,substring(inception,0,150) as inception, msg, sentiment, stanfordsentiment, location, timeFROM twitterorc where inception not like '%Not found%' and inception is not null and trim(inception)!= ''

Using Ignition version 7.8.4 (b2016082217) | 64-bitCopy dependencies needed to make a connection to the Apache Phoneix Server werecopied into Ignition\libEdit Igntion\data\ignition.conf to register the dependacy with Ignition's boot strap processRestart Nifi and then begin the process of configuring JDBC drivers by selecting the drivers link in the Databases section of Ignition management consoleSelect create new JDBC Driver on the Drivers Home PageComplete and submit the driver properties page using the following as a guideNow select connections from the data bases menuand create new database connection from the connections home pageComplete and submit the connection properties dialog After successfully submitting the configuration the connection should appear on the connections home pageIf there are any issues with the driver or connection they can be found in the log of the system consoleWith the connection configured we can now added data to the HMI application from queries of the

ErrorRanger Tagsync shows lots of KafkaException in log file, which causes disk space alert in Ambari. Also it used out all of the client port./var/log/ranger/tagsync/tagsync.log10 Nov 2016 11:46:43 ERROR TagSynchronizer [main] - 262 tag-source:atlas initialization failed with javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, b ut the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user kafka.common.KafkaException: fetching topic metadata for topics [Set(ATLAS_ENTITIES)] from broker [ArrayBuffer(BrokerEndPoint(1001,host.domain,6667))] failed at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:73) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:96) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:67) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)

Ambari 2.2.2.18 is released and available for immediate use. The team fixed 60 issues, many of which customers have experienced. No features are included, so an upgrade should be seamless for those using Ambari 2.2.2.0 or are considering upgrading to Ambari 2.2.2.xRefer to the Product Documentation and Release Notes for further information about this release.This product rolls up all issues in private releases less than 18.

ObjectiveIf you are using Docker on CentOS 7, you may have run into issues importing the Docker version of the Hortonworks Sandbox for HDP 2.5. The default configuration of Docker on CentOS 7 limits your Docker virtual machine image to 10GB in size. The Docker HDP sandbox is 13GB in size. This will cause the loading process of the sandbox to fail.This tutorial will guide you through the process of installing Docker on CentOS 7 using Vagrant and modifying the configuration of Docker to move the location of and increase the size of Docker virtual machine image.While we are using Vagrant + Virtualbox, this process should work for any install of CentOS (Amazon, etc) with small changes because VirtualBox and related plugins are not needed.PrerequisitesYou should already have downloaded the Docker HDP 2.5 Sandbox. Read more here: Docker HDP SandboxYou should already have installed VirtualBox 5.1.x. Read more here: VirtualBoxYou should already have installed Vagrant 1.8.6. Read more here: Vag

Introduction Apache Phoenix is a relational database "layer" built on top of Apache HBase, a NoSQL key-value data store. Until recently, the only provided entry-point for users to Phoenix was by their JDBC driver. This implicitly limited client applications to JVM-based languages. The Phoenix QueryServer was built using a sub-project of the Apache Calcite project named Avatica. Avatica is a JDBC-over-HTTP library. It includes both an HTTP server and a JDBC driver. The HTTP server has its own wire API which can be leveraged by clients in any language communicate with the backend JDBC server.Deployment One of the goals of the Phoenix QueryServer was to follow typical deployment strategies of HTTP servers. The primary adopted goal is that the QueryServer aims to be stateless. While this is not entirely the case as QueryServer does maintain internal state, the server and Java client are written in such a way that any missing state in the PQS instance can be automatically rec

1. Lets assume you have HDP cluster installed and managed by Ambari.2. When we want to delete a service [either Custom service or HDP service] using api, you generally use below command -curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://<ambari-server>:8080/api/v1/clusters/c1/services/<SERVICENAME>;2. After executing above command you might see below error -$curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://<ambari-server>:8080/api/v1/clusters/c1/services/HBASE{ "status" : 500, "message" : "org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Cannot remove HBASE. Desired state STARTED is not removable. Service must be stopped or disabled." 3. If you see the above error while Removing/Stopping service please use below steps to resolve the issue4. Login to ambari database [In my case its postgresql] and check the values of the service in below tables -# psql -U ambari[Default password is 'bigdata']

Objective Many people work exclusively from a laptop where storage space is typically limited to 500GB of space or less. Over time, you may find your available storage space has become a regular concern. It's not uncommon to use an external hard drive to augment available storage space. The current version of Docker for Mac (1.12.x) does not provide a configuration setting which allows users to change the location where the Docker virtual machine image is located. This means the image, which can grow up to 64GB in size by default, is located on your laptop's primary hard drive. With the HDP 2.5 version of the Hortonworks sandbox available as a native Docker image, you may find a desire to have more room available to Docker. This tutorial will guide you through the process of moving your Docker virtual machine image to a different location, an external drive in this case. This will free up to 64GB of space on your primary laptop hard drive and let you expand the size of the Docker image

As a follow-up to my article on downloading and playing MIDI files (https://community.hortonworks.com/content/kbentry/65154/nifi-1x-for-automatic-music-playing-pipelines.html) This quick article highlights how you can support both MP3 and MIDI. As you can see it's very easy to have multiple channels in, multiple processing paths and process many types of inputs and files in NIFI.So how do we add MP3 playback to our Jukebox?Add another GetHTTP processorhttp://192.168.1.2:8080/extract/url?url=http://www.amclassical.com/piano//&type=mp3Still using my Microservice to convert web pages into links in JSON, this time filtering out MP3s. Fortunately there's a few web pages out there with free classical MP3s.Next after UpdateAttribute, I check file extension in a Route on Attribute.${filename:contains('midi')}If it's MIDI, call the same MIDI player.I added a second PutFile to save to /opt/demo/mp3, so that I keep my music files seperately.Otherwise, call OSX's command line MP3 player.In

Use CaseBefore meetups, I wanted to play some music. NiFi seemed like a great choice for streaming free music through my Mac. MIDI Command Line Player For OSXbrew install timidityThen you can simply play MIDI files with timidity file.mid.Microservice to Extract Links from Web PagesJava 8 Source Code: https://github.com/tspannhw/linkextractorThe Spring Boot REST API accepts a URL, parses out mid files and returns JSON containing linking and descriptions. Example REST Call to Servicecurl -G -v "http://<urL>:8080/extract/url?url=http://www.midiworld.com/classic.htm/&type=mid"Run the Microservice java -Xms512m -Xmx2048m -Djava.net.preferIPv4Stack=true -jar target/linkextractor-0.0.1-SNAPSHOT.jarJava Snippet Using JSoup to extract links from URL (HTML)pLink = new PrintableLink(); pLink.setLink(link.attr("abs:href")); pLink.setDescr(trim(link.text(), 100)); linksReturned.add(pLink); Outputhdfs dfs -ls /music/*.mid-rw-r--r-- 3 tspann hdfs 87 2016-11-0

On Oct 25th, EMC Isilon announced general availability of OneFS 8.0.1. The official announcement can be found here - https://isiblog.emc.com/category/onefs-8-0-1/OneFS 8.0.1 is the first major upgrade to OneFS 8.0 platform that was released in Feb 2016. While this upgrade has many generic enhancements to the storage side of OneFS, there are four major functionalities related to Hadoop Supports Ambari 2.4This is the first major release of OneFS that will have support for Ambari 2.4. Hortonworks had released HDP 2.5 and Ambari 2.4 at end of Aug 2016. Since then Isilon team had been working to integrate Ambari 2.4 with OneFS. With the release one OneFS 801, customers can now use Ambari 2.4 and HDP 2.5For more instructions on how to configure, see the below link : https://community.emc.com/community/products/isilon/blog/2016/10/24/onefs-and-ambari-24-hdp-25 Ambari Metrics and Alerts integration Though OneFS has been supporting Ambari for a few releases, it was not leveraging

Objective In many organizations "search" is a common requirement for a user friendly means of accessing data. When people thing of "search", they often think of Google. Many organizations use Solr as their enterprise search engine. It is commonly used to power public website search from within the site itself. Organizations will often build custom user interfaces to tailor queries to meet their internal or external end-user needs. In most of these scenarios, users are shielded from the complexity of the Solr query syntax. Solr has a long list of features and capabilities; you can read more here Apache Solr Features. Solr 6 has a new feature which allows you to submit SQL queries via JDBC. This opens up new ways to interact with Solr. Using Zeppelin with SQL is now possible because of this new feature. This should make you more productive because you can use a language syntax with which you are already familiar: SQL! This tutorial will guide you through the process of updating the Z

Starting My Hadoop ToolsNiFi can interface directly with Hive, HDFS, HBase, Flume and Phoenix. And I can also trigger Spark and Flink through Kafka and Site-To-Site. Sometimes I need to run some Pig scripts. Apache Pig is very stable and has a lot of functions and tools that make for some smart processing. You can easily augment and add this piece to a larger pipeline or part of the process.Pig SetupI like to use Ambari to install the HDP 2.5 clients on my NiFi box to have access to all the tools I may need.Then I can just do:yum install pigPig to Apache NiFi 1.0.0ExecuteProcessWe call a shell script that wraps the Pig script.Output of script is stored to HDFS: hdfs dfs -ls /nifi-logs Shell Scriptexport JAVA_HOME=/opt/jdk1.8.0_101/ pig -x local -l /tmp/pig.log -f /opt/demo/pigscripts/test.pig You can run in different Pig modes like local, mapreduce and tez. You can also pass in parameters or the script.Pig Scriptmessages = LOAD '/opt/demo/HDF/centos7/tars/nifi/nifi-1.0.0.2.0

IntroductionThis is a continuation of Apache Storm Tuning Approach for Squirrels article that I published couple weeks ago.A topology is going to have to coexist on a Storm cluster with a variety of other topologies. Some of those topologies will burn CPU doing heavy calculations, others will consume large amounts of network bandwidth.No one sane can claim to provide a silver bullet for how to set up your cluster for the best performance. However, I’d like to share a few recipes and guidelines for dealing with issues as they arise. I listed below 6 categories of contention, describing common problems and strategies to alleviate or eliminate resource contention. I hope that this article at least provides a good map for a deep dive onto Storm tuning. Each resource contention is probably worth it an article.Worker Processes in a ClusterA Storm cluster is installed with a fixed number of available worker processes across all worker nodes. Each time you deploy a new topology to the cluster,

The Ranger KMS logs are present cd /usr/hdp/current/ranger-kms/ewslrwxrwxrwx 1 kms kms 19 sep 23 20:33 logs-> /var/log/ranger/kmsWe would like to change ranger KMS logs location from /var/log/ ranger/kms to /hadoop/log/ranger/kms.mkdir /hadoop/log/ranger/kmschmod 755 /hadoop/log/ranger/kmschown kms:kms /hadoop/log/ranger/kmsrm symbolic link for logs cd /usr/hdp/current/ranger-kms/ vi ranger-kmssearch for word "TOMCAT_LOG_DIR" and replace with new location.TOMCAT_LOG_DIR=/hadoop/log/ranger/kms In Ambari webUI update log directory for ranger KMS kms_log_dir property. Restart KMS

To setup an Ambari managed HDP cluster, below services’ components require database for their respective metastore databases. Ambari ServerHive OozieRanger While a lab/sandbox environment can be setup with default databases for these components, using the same is strongly not recommended for Dev/QA/UAT/Production clusters. Due diligence and planning must be done to ensure that database selection is appropriate for enterprise standard production cluster. Below are key areas to be taken into consideration while planning to select a database for Ambari and HDP components.Supported DatabasesRefer below for supported databases for Ambari and different HDP components in current state.https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-installation/content/database_requirements.html High Availability SupportIt is strongly recommended to setup High Availability for different HDP components, which can be enabled for High Availability in a production cluster.High Availability is s

Objective Cross Data Center Replication, commonly abbreviated as CDCR, is a new feature found in SolrCloud 6.x. This feature enables Solr to replicate data from one source collection to one or more target collections distributed between data centers. The current version provides an active-passive disaster recovery solution for Solr. Data updates, which include adds, updates, and deletes, are copied from the source collection to the target collection. This means the target collection should not be sent data updates outside of the CDRC functionality. Prior to SolrCloud 6.x you had to manually design a strategy for replication across data centers. This tutorial will guide you through the process of enabling CDCR between two SolrCloud clusters, each with 1 server, in a Vagrant + VirtualBox environment. NOTE: Solr 6 is being deployed as a standalone application. HDP 2.5 provides support for Solr 5.5.2 via HDPSearch which does not include CDCR functionality.Prerequisites You should have alre

Demo Extract data from images and store in HDFS. Documents with size less than 10mb stores into HBase. Document > 10mb lands into HDFS with metadata into HBase Part 1 - https://www.linkedin.com/pulse/cds-content-data-store-nosql-part-1-co-dev-neeraj-sabharwal

Step1: Please allow below ports from your OS firewall for Ambari.https://ambari.apache.org/1.2.5/installing-hadoop-using-ambari/content/reference_chap2_7.html.Step2: Please go though the required component and allow below ports from your OS firewall for HDP.https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_HDP_Reference_Guide/content/accumulo-ports.html.Step 3: In order to allow YARN jobs to run successfully, we need to add custom TCP port range to the YARN configuration.Login to Ambari UI --> Select Mapreduce2 --> Configs --> Customer mapred-site --> Add/Modify below propertyyarn.app.mapreduce.am.job.client.port-range=32000-65000 Notes: 1. 32000-65000 is the port range which will be used by Application Master to connect to Node Managers. .2. You can increase the number of ports based on job volume. .How to add exception in Centos7 firewall?Example for Step 3.#firewall-cmd --permanent --zone=public --add-port=32000-65000/tcp#firewall-cmd --reload Please comment i

HDP Sandbox comes pre-installed with SparkR and R . First let's setup the R Studio on the HDP Sandbox:   To download and install RStudio Server open a terminal window and execute the commands corresponding to get the 64-bit version wget https://download2.rstudio.org/rstudio-server-rhel-0.99.893-x86_64.rpmsudo yum install --nogpgcheck rstudio-server-rhel-0.99.893-x86_64.rpmsudo rstudio-server verify-installationsudo rstudio-server stopsudo rstudio-server start Now setup a local user on the hdp sandbox, to access the rstudio: useradd alexpasswd xxxx Next launch web-browser and point to : http://sandbox.hortonworks.com:8787/ Login using the local account created earlier above. Next lets initialize the sparkr in "yarn-client" mode Sys.setenv(SPARK_HOME="/usr/hdp/current/spark-client/").libPaths(c(file.path(Sys.getenv("SPARK_HOME"),"R","lib"),.libPaths()))library(SparkR)sc <- SparkR::sparkR.init(master = "yarn-client") Run the code in rstudio , you should get the following output:

Note: I used TinyCert to generate my own certificate authority and certificates. Probably the easiest way to generate new certificates Create your own certificate authority. Fill in your own values. Create 2 new certificates. One is your client certificate (in this case, bbukacek) and a server certificate which will be used for the NiFi keystore.Client CertificateServer Certificate Download the certificate authority, the client and server certificates and upload to your NiFi environment.Note: Easiest way is to download the client and server certificates in PKCS12 format Create a TrustStore We will use the certificate authority to create the TrustStore. Use the following syntax to create the TrustStore:> keytool -import -file <ca_file>.pem -alias cacert -keystore truststore.jks -storepass <password> Configure NiFi Now that the TrustStore is created we can configure NiFi Keystore and TrustStore.Under $NIFI_HOME/conf, open the nifi.properties.Set the web properties First an

ProblemUsing rhive with following parameters to connect to hivelibrary(RHive)Sys.setenv(HADOOP_CONF_DIR="/etc/hadoop/conf")Sys.setenv(RHIVE_HIVESERVER_VERSION="2")Sys.setenv(HIVE_HOME="/usr/hdp/current/hive-client")Sys.setenv(HADOOP_HOME="/usr/hdp/current/hadoop-client")rhive.init()> rhive.connect()2016-10-26 13:06:22,836 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1173)) - fs.default.name is deprecated. Instead, use fs.defaultFS2016-10-26 13:06:23,720 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable2016-10-26 13:06:24,631 WARN [main] shortcircuit.DomainSocketFactory (DomainSocketFactory.java:<init>(117)) - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.2016-10-26 13:06:25,237 INFO [Thread-5] jdbc.Utils (Utils.java:parseURL(309)) - Supplied authorities: 127.0.0.1:10000

Step1). Get all the properties from the below command: (Here : "http://jss1:8080" is the URL where ambari server is running)curl --user admin:admin -i -H 'X-Requested-By: ambari' -X GET http://jss1:8080/api/v1/clusters/JoyCluster/configurations?type=cluster-env\&tag=version1Step2). Save the output from above command to a file. Or just open the URL in browser and save it to a file:http://jss1:8080/api/v1/clusters/JoyCluster/configurations?type=cluster-env&tag=version1.For example here i got the following output from the above URL:{ "href" : "http://jss1:8080/api/v1/clusters/JoyCluster/configurations?type=cluster-env&tag=version1", "items" : [ { "href" : "http://jss1:8080/api/v1/clusters/JoyCluster/configurations?type=cluster-env&tag=version1", "tag" : "version1", "type" : "cluster-env", "version" : 1, "Config" : { "cluster_name" : "JoyCluster", "stack_id" : "HDP-2.4" }, "properties" : { "fetch_nonlocal_groups"

One of the great things about Apache NiFi is its ability to consume and transform many formats of data, however a particular area of complexity has been around the transformation of JSON data received (i.e. JSON-to-JSON transformation). Take the case of the GetTwitter processor that provides access to Twitter data streams (either from a filtered search, sample set, or firehose endpoints). For anyone who has worked with Twitter’s JSON schema it is very rich with detailed information for an individual tweet. There are cases where a lot of this data isn’t required for analytics and data flow managers or analysts are looking to pare it down to the necessities. There are also instances where incoming JSON data simply needs to be formatted or re-labeled for use in another system or repository, such as Hive, HBase, or MongoDB. Outside of NiFi, JSON-to-JSON transformation has been simplified in the Jolt Java library which offers a declarative approach to defining JSON output. Jolt provi