Community Articles

Ensure yarn container size is optimized for spark. No more than 1-2 containers per CPU core.Minimize the use of <operation>ByKey (other than reduceByKey, which is recommended). These functions have a lot of overhead shuffling data via the internal hashtables. While they are useful, they should be utilized sparingly to get the best performance.Don't always rely on the default rdd partitioning. It will generally always split things into 100 pieces unless otherwise defined. Redefining that default in a way that makes sense for dataset sizes and number of containers will get better performance. If there isn't a fixed number that works across the board, manually repartitioning early in jobs can help to speed things up.Use the G1 garbage collector instead of the parallel garbage collector. It takes a little fine tuning with memory, but it performs better.Use dataframes where possible. As progressive versions of spark roll out, there is a lot of performance optimization of dataframes op

How to use Ambari View for HAWQ Query Monitoring? This beta version of Ambari HAWQ View provides live query monitoring directly in Ambari UI, displaying a list of queries running in a HAWQ cluster. This allows admins to see the database activity on the UI in a user-friendly interface. The view front-end is based on EmberJS framework. The Java backend extends the framework provided by the ambari-views project. The front-end polls the REST API every 5 seconds, querying the pg_stat_activity view and rendering reformatted data on the UI.How to Deploy: Deployment simply requires copying the hawq-view jar to the ambari-server host and restarting ambari-server. ssh into ambari_server_host $ cd /var/lib/ambari-server/resources/views/$ curl -O https://s3-us-west-2.amazonaws.com/pivotal-ambari/stable/hawq-view/hawq-view-1.0.0.0-1.jar$ ambari-server restartCreating an Instance of HAWQ View: a) The HAWQ View instance connects to the HAWQ Master through JDBC. Ensure that HAWQ pg_hba.conf is upd

Livy: Livy is an open source REST interface for interacting with Spark. Authorized users can launch a Spark session and submit code. Two different users can access their own private data and session, and they can collaborate on a notebook. Only the Livy server can submit a job securely to a Spark session.Steps to follow to configure livy interpreter to work with secure HDP cluster:Setup proxy for livy interpreter in core-site.xmlGo to Ambari->HDFS->config->customer-core-site and add below properties: hadoop.proxyuser.livy.groups=* hadoop.proxyuser.livy.hosts=*2. Configure livy interpreter in Zeppelin and add below configurations:livy.superusers=zeppelin-spark Note - The value for livy.superusers should be your zeppelin principal. That would be zeppelin-{$Cluster_name} For example, in this case you can find it by running below command: klist -kt /etc/security/keytabs/zeppelin.server.kerberos.keytabKeytab name: FILE:/etc/security/keytabs/zeppelin.server.kerberos.keytabKVNO Times

Apache Ranger uses an embedded Tomcat server to provide the Web UI functionality for administration of Ranger. A previous HCC article provided details on maintenance of the log files that are managed by the log4j configuration, including xa_portal.log, ranger_admin_perf.log, xa_portal_sql.log. We're going to focus on maintenance of the access_log* logs that get automatically generated by Tomcat, but which are not managed by this log4j configuration. With embedded Tomcat, the configuration is contained within the code for the AccessLogValve (as you can see, it uses an hourly rotation pattern unless overridden by ranger.accesslog.dateformat). We'll use the logrotate application in CentOS/RHEL to manage these access_log* logs as the number of files can grow large without rotation and removal in place. You can check to see how many of these files you have on your Ranger Admin node by running (there would be one access_log* file per hour for each day during which the service has ran con

This is the first of a series of short articles on using Apache Spark with Hortonworks HDP for beginners. If you’re reading this, you don’t need need me to define what Spark is, there are numerous references on the web that can speak about that its API, being data structure centric and in my opinion one of the most important Open Source projects. The intent of this article is let you know what helped me get started using Spark on the Hortonworks Data Platform. This is not a tutorial.I’m assuming you have access to a an HDP 2.5.3 cluster or to the Hortonworks Sandbox for HDP 2.5.3 or above. Also, I’m going to assume that you are familiar with SQL, Apache Hive, and using the Linux/Unix Bourne Shell. The problem I was having, which pushed me to use Spark was that using Hive for data processing was limiting. There were things I was used to using in my past life as an Oracle DBA that were not available. Hive is a fantastic product, but Hive SQL didn’t give me all the bells and whist

Just some clarifications to the instructions provided in this article. Hopefully, it will save you time to get you up and running faster.1) Nifi 401:Unauthorized ErrorIf you see a similar error message in the Nifi console:ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.standard.PostHTTP PostHTTP[id=834bb9f9-a15d-42bd-8d7a-3f00c810d729] Failed to Post StandardFlowFileRecord[uuid=bc74e1c5-12e7-4da2-93b4-a3dc624218ac,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1471370511531-1, container=default, section=1], offset=11780, length=407],offset=0,name=12406098021784,size=407] to http://gcm-http.googleapis.com/gcm/send: response code was 401:Unauthorizedand when you click "Notify Customer" in the Analyst's console and no event is updated on the Android emulator in Android Studio.Resolution 401:Unauthorized ErrorOpen open the UpdateAttribute processor and add a new property called Authorization and set its value to key=your google browser key2) Wrong Google API n

# Ruby Scripting in NiFiToday I was asked about an issue that I didn't know how to solve using NiFi. On the surface it sounded simple; just map an attribute to another value. The attribute to map was 'port' and based on the port number just add an attribute to more easily identify the system downstream. E.g. for port 10003; syslog:local, for 10004; syslog:db1, etc. After a bit of digging I found a few different options to solve this.## Many UpdateAttribute ProcessorsThe first is to create a new UpdateAttribute processor for each incoming stream. This labels (places an attribute) on all files that come in from that listener. It looked like this:This looks a bit confusing and tedious but is very precise and arguably easier to read, especially when we label the processors. It also has the added advantage of not having to change the port in more than one place. If for example, the local logs com

bug (HIVE-15355) is being worked on by Hive engineering team at HortonworksYou can use the following workaround :1- use "SORT BY 0" at the end of the query which will force to use one reducer , please use this only if you have a small query.2- try to use set hive.mv.files.thread=0; before running the query.If you have any question regarding the above, please let me know.

Introduction Using NiFi, data can be exposed in such a way that a receiver can pull from it by adding an Output Port to the root process group. For Storm, we will use this same mechanism - we will use the Site-to-Site protocol to pull data from NiFi's Output Ports. In this tutorial we learn to capture NiFi app log from the Sandbox and parse it using Java regex and ingest it to Phoenix via Storm or Directly using NiFi PutSql Processor. Prerequisites 1) Assuming you already have latest version of NiFi-1.x/HDF-2.x downloaded as zip file (HDF and HDP cannot be managed by Ambari on same nodes as of now) on to your HW Sandbox Version 2.5, else execute below after ssh connectivity to sandbox is established: # cd /opt/ # wget http://public-repo-1.hortonworks.com.s3.amazonaws.com/HDF/centos6/2.x/updates/2.0.1.0/HDF-2.0.1.0-centos6-tars-tarball.tar.gz # tar -xvf HDF-2.0.1.0-12.tar.gz 2) Storm, Zeppelin are Installed on your VM and started. 3) Hbase is Installed with phoeix Query Server. 4) Mak

Prerequisites: HDP 2.5 on Hortonworks Sandbox up and running.Apache NiFi installed and up and running: To quickly install Apache NiFi on the HDP 2.5 Sandbox by Ambari wizard, check Section 2 on the NiFi tutorial.Assumes a minimum of knowledge on how to access Ambari HDFS View, Hive View and Zeppelin UI on the sandbox.This demo has only been tested and developed for the HDP 2.5 Sandbox OS, which is CentOS 6.8. The NiFi flow may need to be tweaked depending on the target OS where the messages are generated. Enable rsyslog on your Sandbox:In this tutorial, we will enable rsyslog to forward syslog messages to localhost on port 7780 as following:- Edit /etc/rsyslog.conf and add the following line:*.* @127.0.0.1:7780- Restart syslog for the changes to take effect:[root@sandbox ~]# service rsyslog restartShutting down system logger: [ OK ]Starting system logger: [ OK ]- You can optionally read from the port 7780 to make s

If you are getting below error while refreshing client configs via Ambari UI then here are the steps to fix it! 1. Remove KERBEROS_CLIENT for problematic hosts using below curl call curl -H "X-Requested-By:ambari"-u admin:admin -X DELETE "http://<AMBARI-SERVER>:8080/api/v1/clusters/<cluster-name>/hosts/<host-name>/host_components/KERBEROS_CLIENT" 2. Ensure from Ambari UI that KERBEROS_CLIENT got removed. 3. From Ambari UI --> goto the problematic host --> click on Add button --> select Kerberos Client 4. Installation should complete without any issue.

Metron provides a Kibana-based UI that is designed to be a single pane of glass to utilize the big data approach (having all data available to you at the same time) to filter through the irrelevant and display the just the information, alerts, and context an analyst/investigator is looking for all on the same pane. The Metron UI has several advantages over conventional SIEM tools, including flexibility, and having the needle as well as all the context for the needle presented together on the same screen, requiring no jumping around from console to console to gather the information. Let us first describe the parts of the UI panel. Specifically there are 4 sections: Pinned Queries and Filters, Ingest Histogram Panels, Detailed Message Tables, and finally the PCAP panel. We will go through all the parts in detail in this article and then we will how how to tie everything together and create a single pane of glass interface to filter through data and create context for the search.The f

Incremental Updates: A Strategy for Appending Records in HiveAs part of the Stinger.next initiative, we expect to see Hive supporting full CRUD operations (Insert, Select, Update, Delete). While some transaction support exists for Hive today, it is still evolving with regard to performance and scalability. In the meantime, customers may still need to work with the traditional options for Hive table integration: OVERWRITE or APPEND. The OVERWRITE option requires moving the complete record set from source to Hadoop. While this approach may work for smaller data sets, it may be prohibitive with regard to scale.The APPEND option can limit data movement to only new or updated records. As true Inserts and Updates are not yet available in Hive, we need to consider a process of preventing duplicate records as Updates are appended to the cumulative record set.In this article, we will look at a strategy for appending Updates and Inserts from delimited and RDBMS sources to existing Hive tabl

IntroductionApache Zeppelin is a web-based notebook that enables interactive data analytics while Apache Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs. Pig-latin is a very powerful languages for data flow processing. One drawback of pig community complains about is that pig-latin is not a standard language like sql so very few BI tools integrate with it. So it is pretty to hard to visualize the result from pig. Now the good news is that Pig is integrated in zeppelin 0.7 where you can write pig latin and visualize the result.Use pig interpreterPig interpreter is supported from zeppelin 0.7.0, so first you need to install zeppelin, you can refer this link for how to install and start zeppelin.Zeppelin supports 2 kinds of pig interpreters for now.%pig (default interpreter)%pig.query%pig is like the pig grunt shell. Anything you can run in pig grunt shell can be run in %pig.script interpreter, it is used for run

You can run the attack library for OSX or Linux from an edge node or from outside the cluster.I ran against mine from my OSX laptop against my cluster that I had network access to.You should try to scan from inside your network, from an edge node and from a remote site on the Internet.You will need Python 2.7 or Python 3.x installed first.git clone git@github.com:CERT-W/hadoop-attack-library.git pip install requests lxmlYou may need root or sudo access to install on your machine.One of the scanners hits the WebHDFS link that you may have seen a warning about.python hdfsbrowser.py timscluster Beginning to test services accessibility using default ports ...Testing service WebHDFS[+] Service WebHDFS is availableTesting service HttpFS[-] Exception during requesting the service[+] Sucessfully retrieved 1 servicesdrwxrwxrwx hdfs:hdfs 2017-01-15T05:50:27+0000 /drwxrwxrwx yarn:hadoop 2017-01-11T19:25:26+0000 app-logs /app-logsdrwxrwxrwx hdfs:hdfs 2016-12-21T23:12:40+0000 apps /app

ProblemWhen trying to start the secondary hive metastore service, in this example, zookeeper is unable to create the znode with the appropriate permissions. This can be seen mostly on the non-Ambari managed clusters.17/01/19 18:25:17 ERROR metastore.HiveMetaStore: Metastore Thrift Server threw an exception...org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hivedelegationMETASTORE/keys        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166)        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236)        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:473)        at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:92)        at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6031)        at org.apache.hadoop.h

We just updated Hortonworks Data Cloud for AWS to Technical Preview #1.12. The release is packed with goodies such as: Support for deploying compute nodes with spot pricing.Support for executing node recipes - custom scripts that can be run pre- or post- cluster deployment for customizing the cluster and installing additional software.Support for HDP 2.6 (Technical Preview) that can launch two new cluster configurations for Spark 2.1 and Druid. To create an HDP 2.6 cluster, launch the cloud controller and when creating a cluster choose HDP Version: HDP 2.6 (Technical Preview) and then choose one of the available cluster types:For more details, refer to the Release Notes: http://hortonworks.github.io/hdp-aws/releasenotes/.To get started with HDCloud for AWS, visit http://hortonworks.github.io/hdp-aws/.To get started with Spark 2.1, see Vinay's blog at http://hortonworks.com/blog/try-apache-spark-2-1-zeppelin-hortonworks-data-cloud/. Have fun!

Hive Compare Tables and Databases before and after Upgrades.Run below script to collect all databases.hive -e "show databases" | sed "s/ *$//g" | sed "s/^ *//g" | sort 1>databaselist Run below script to collect all tables in database.for dbname in `cat databaselist` ; do hive -e "use ${dbname} ; show tables ;" | sed "s/ *$//g" | sed "s/^ *//g" | sort 1> tablelist; done ; Execute Scripts above before upgrade and once after upgrade .Simply do vimdiff afterwards to identify the difference if any. Happy Migrating or Upgrading.

During the upgrade : if you are using postgres DB, you will face a script failure on the following statement Alter table users add constraint "uniq_user_0" Solution : psql -U ambari_user -d NN_HA_ambari alter table ambari.user drop constraint "uniq_user_0"   After the Upgrade : Spark 1-you have to recompile and rebuild your jars using the new dependencies of Spark 1.6.2 Your pom.xml needs to have the new versions ( new jars of HDP 2.5.3 ) 2-Update any custom jars and check all versions used in your Pom.xml i.e <spark.version>1.6.2</spark.version><hbase.version>1.2.1</hbase.version>   3- Remove the argument spark.yarn.jar if you are using it when submitting the job. --conf spark.yarn.jar=hdfs://NN_HA/user/oozie/share/lib/lib_20160515013357/spark/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar. Error noticed when leaving the arg above : java.lang.IllegalArgumentException: Invalid ContainerId: container_e97_1484824056011_0038_02_000001 or jav

In order to secure access to the Zeppelin UI, we will want to enable TLS (as well as authentication) to ensure confidentiality of communication and to assure the identity of the Zeppelin server. Zeppelin uses Jetty as the underlying HTTP server, so we'll refer to Jetty documentation.In this how-to we'll use a self-signed certificate. In Production environments, you will likely obtain a CA-issued certificate or a trusted root certificate from your PKI team specific to your environment.Since self-signed certificates won't be trusted by your browser by default, we'll show how to trust this certificate on OS X 10.11.6 with Chrome version 55.0.2883.95 (other OS/browser combinations are out of the scope of this article).To generate the self-signed certificate, we'll use the openssl and keytool utilities as follows (see this Jetty doc for reference):openssl genrsa -des3 -out zeppelin.keyopenssl req -new -x509 -key zeppelin.key -out zeppelin.crtkeytool -keystore keystore -import -alias zeppeli

There are quite a few different reasons (already posted) in which i've seen this error obtained. Nonetheless I ran into this issue with a client recently in which none of the posted articles accounted for. The customer was attempting to load the Ambari UI and received the "unable to connect to Ambari server. Confirm ambari server is running and you can reach ambari server from this machine" error. We checked Iptables which were turned on, but when disabled the issue persisted.We checked Ambari Server permspace but the amount specified was sufficient.We checked SElinux and it was disabled.We were finally able to load into the Ambari Server after checking "tripwire":ps -ef |grep tripwireOnce tripwire was disabled, we were successfully able to login. More information regarding tripwire can be found here:https://en.wikipedia.org/wiki/Open_Source_Tripwire

Hadoop can be deployed on a variety of scales. The requirements at each of these will be different. Hadoop has a large number of tunable parameters that can be used to influence its operation. Furthermore, there are a number of other technologies which can be deployed with Hadoop for additional capabilities. Performance Monitoring In HadoopMultiple tools exist to monitor large clusters for performance and troubleshooting. This section briefly highlights two such tools.Ganglia is a performance monitoring framework for distributed systems. Ganglia provides a distributed service which collects metrics on individual machines and forwards them to an aggregator which can report back to an administrator on the global state of a cluster.Ganglia is designed to be integrated into other applications to collect statistics about their operation. Hadoop includes a performance monitoring framework which can use Ganglia as its backend. Instructions are available on the Hadoop wiki as to how to enable

This article describe important configuration steps to consider for securing the Hortonworks Data Platform (HDP) Hadoop clusters that are created from Azure marketplace images of HDP. The current HDP offering on Azure utilizes a template that locks down all ports and protocols by default, except for port 22, for security reasons. This article explains the methods to secure and limit access to the service endpoints (for example those of Ambari) to known IP addresses from the Internet from this default configuration.Here are the steps to create an Azure IaaS cluster:Login to your Azure portal and bring up the Marketplace listings:2. Find the Hortonworks listing.3. Fill in the information to create a cluster.4. Fill in the details for the security information. We recommend using SSH keys instead of passwords for direct access to the nodes, and a strong password for Ambari. 5. Accept the terms and launch the cluster.6. Wait for the cluster deployment to complete.7. If you need to access th

DESCRIPTION:CREATE TABLE EXAMPLE_REPORT1 > ( > key string, > claim_type_code string, > yearservice string, > monthservice string > ) > STORED BY "org.apache.hadoop.hive.hbase.HBaseStorageHandler" > WITH SERDEPROPERTIES("hbase.columns.mapping" = ":key,claim_type_code:claim_type_code, > yearservice:yearservice,monthservice:monthservice") > TBLPROPERTIES("hbase.table.name"="EXAMPLE_REPORT1”);Exception received was as follows:FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208)at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:326) at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:301) at org.apache.hadoop.hbase.client.ClientScanner.initializeScanner

HDB 2.1.1Reference: http://hdb.docs.pivotal.io/211http://hdb.docs.pivotal.io/211/hdb/releasenotes/HAWQ211ReleaseNotes.htmlhttp://hdb.docs.pivotal.io/211/hdb/install/install-ambari.htmlDownload HDB from Hortonworks at http://hortonworks.com/downloads/ or directly from Pivotal at https://network.pivotal.io/products/pivotal-hdb (You need to create a pivotal account)What to look out for If you use only only 1 Master Node, you cannot have a Hawq Master and StandbyIf I install Hawq Master on Same node with Ambari need to change PostGres Port from 5432 on InstallPrep Ensure that httpd is installedyum install httpdsudo service httpd statussudo service httpd startGet and Install repo Log onto Pivotal and download hdb-2.1.1.0-7.tar/* On Ambari Node */1. mkdir /staging2. chmod a+rx /staging3. scp -i <<your key>> -o 'StrictHostKeyChecking=no' hdb-2.1.1.0-7.tar root@<<ambarinode>>:~/staging4. tar -zxvf hdb-2.1.1.0-7.tarcd /staging/hdb-2.1.1.0./setup_repo.sh/* You should s