Resources

Permissions are available under existing notebook -> padlock buttonThere are three different permissions: Owners – this is the owner(s) of the notebook. This permissions allows owners to give further - read/write - permission to that notebookReaders – these users can only read scripts created in the section of that notebook. They can neither execute nor edit the scripts - see belowWriters – these users can edit the scripts in notebook. With that permissions the users can also read the notebook. Hence, there is no need to add them to be Reader as wellReader users:When they try to execute the scriptWhen they edit the script - the screen comes up after few seconds. But the content create is not saved

In Zeppelin LDAP Authentication with OpenLDAP and How to Set Up OpenLDAP we've shown how to use LDAP Authentication with Zeppelin. In this article, we'll harden that configuration by ensuring that Zeppelin and OpenLDAP communicate over LDAPS.LDAPS is a secure protocol that uses TLS to assure authenticity, confidentiality, and integrity of communications. This prevents man-in-the-middle attacks that sniff traffic to discover LDAP credentials communicated in plaintext, which could compromise the security of the cluster.The first step is to modify the configuration of the OpenLDAP server, as root, to expose LDAPS connectivity, we'll need to modify /etc/openldap/ldap.conf. Please recall that we created /etc/openldap/certs/myldap.field.hortonworks.com.cert in the How to Set Up OpenLDAP article#TLS_CACERTDIR /etc/openldap/certsTLS_CACERT /etc/openldap/certs/myldap.field.hortonworks.com.certURI ldaps://myldap.field.hortonworks.com ldap://myldap.field.hortonworks.comBASE dc=field,dc=hortonwor

Auto-recovery in Ambari is a useful way of getting cluster components restarted automatically in the event that a component fails (without the need for human intervention).Ambari 2.4.0 introduced dynamic auto-recovery, which allows auto-start properties to be configured without needing an ambari-agent / ambari-server restart. Currently, the simplest way to manage the auto-recovery features within Ambari is via the REST API (documented within this article), although on-going work in the community will bring the feature to the UI: https://issues.apache.org/jira/browse/AMBARI-2330Check Auto-Recovery SettingsTo check if auto-recovery is enabled for all components, run the following command on the Ambari server node:curl -u admin:<password> -i -H 'X-Requested-By: ambari' -X GET http://localhost:8080/api/v1/clusters/<cluster_name>/components?fields=ServiceComponentInfo/component_name,ServiceComponentInfo/service_name,ServiceComponentInfo/category,ServiceComponentInfo/recovery_ena

Whether you're creating an Ambari cluster from scratch, taking over an existing cluster, or growing your cluster over time, it is imperative to tune Ambari and MySQL to work at a large scale of 1000-3000 Ambari Agents.Ambari Server Configs First, increase the memory used by Ambari. For large clusters, 8 GB of memory should be sufficient. If you have more than 10 concurrent users, increase it to 16 GB. Edit /var/lib/ambari-server/ambari-env.sh and change the -Xmn setting. export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms2048m -Xmx8192m Edit /etc/ambari-server/conf/ambari.properties with the following configs # The size of the Jetty connection pool used for handling incoming Ambari Agent requests.# 10 hosts => 25# 50 hosts => 35# 100 hosts => 75# 500 hosts => 100 agent.threadpool.size.max=100# Determines whether current alerts should be cached.# Enabling this can increase performance on large cluster, but can also result in lost alert data# if the cache is not flushed freque

Here is a high level overview of how to deploy Solr if approached cautiously.RequirementsInstallEdit Configuration files to work with HDFSChanging to ‘Solr’ userAdding NodesStarting SolrCloudCreating a collectionVerify & Enjoy#Requirements for this guideInstallation on a Linux server.Solr 5.5.0 requires Java 1.7 and higher.HDP 2.3 or 2.4Apache ZooKeeper — HDP 2.3 or 2.4 and Solr both use Apache ZooKeeper to manage services for the cluster. The ZooKeeper ensemble that you are using for HDP 2.3 or 2.4 can also be used by Solr.This guide doesn’t include Kerberos requirements.#Install Lucidworks-HDPsearch packageyum install lucidworks-hdpsearchAfter installation, the HDPSearch files will be found in this directory./opt/lucidworks-hdpsearch Install only — DO NOT START SOLR SERVICE YETThe Lucidworks HDP Search package should be installed manually on each node of the cluster. Follow this guide on 1 machine first. Then go and start service on each node in cluster (See Adding nodes section

SYMPTOM: HDP upgrade failed on "Finalize". Tried setting the version manually using below command which was failing -ambari-server set-current --cluster-name=<ClusterName> --version-display-name=HDP-2.5.3.0ERROR: Please find error screenshot below - ROOT CAUSE: There was issue with the last "request_id" which was run by Express upgrade process. you can check the last request_id from ambari DB from table name "host_role_command" RESOLUTION:1. Backup the ambari DB2. Find the last "request_id" from table "host_role_command" and check the status of the request_id.3. In my case the status was "Failed"4. Modifying the status of the request_id from "FAILED" to "COMPLETED" and restart ambari server.5. Try running the command againambari-server set-current --cluster-name=<ClusterName> --version-display-name=HDP-2.5.3.0

Preparing a Raspberry PI to Run TensorFlow Image RecognitionI can easily have a Python script that polls my webcam (use official Raspberry Pi webcam) , calls TensorFlow and then sends the results to NiFi via MQTT.You need to install Python MQTT Library (https://pypi.python.org/pypi/paho-mqtt/1.1)For setting up Python, Raspberry PI with Camera, see https://dzone.com/articles/picamera-ingest-real-timeRaspberry Pi 3 B+ preparationBuy a good quality 16 GIG SD Card and from OSX, Run SD Formatter to Overwrite Format the device at FAT, download here: https://www.sdcard.org/downloads/formatter_4/. Download the BerryBoot image from here. Unzip it and then copy it to your complete SD card.For examples of RPi TensorFlow You Can Run:https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/pi_examples/You need to build tensorflow for pi, which took me over 4 hours.See:https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/makefilehttps://github.com/tensorflow/ten

Ensure yarn container size is optimized for spark. No more than 1-2 containers per CPU core.Minimize the use of <operation>ByKey (other than reduceByKey, which is recommended). These functions have a lot of overhead shuffling data via the internal hashtables. While they are useful, they should be utilized sparingly to get the best performance.Don't always rely on the default rdd partitioning. It will generally always split things into 100 pieces unless otherwise defined. Redefining that default in a way that makes sense for dataset sizes and number of containers will get better performance. If there isn't a fixed number that works across the board, manually repartitioning early in jobs can help to speed things up.Use the G1 garbage collector instead of the parallel garbage collector. It takes a little fine tuning with memory, but it performs better.Use dataframes where possible. As progressive versions of spark roll out, there is a lot of performance optimization of dataframes op

How to use Ambari View for HAWQ Query Monitoring? This beta version of Ambari HAWQ View provides live query monitoring directly in Ambari UI, displaying a list of queries running in a HAWQ cluster. This allows admins to see the database activity on the UI in a user-friendly interface. The view front-end is based on EmberJS framework. The Java backend extends the framework provided by the ambari-views project. The front-end polls the REST API every 5 seconds, querying the pg_stat_activity view and rendering reformatted data on the UI.How to Deploy: Deployment simply requires copying the hawq-view jar to the ambari-server host and restarting ambari-server. ssh into ambari_server_host $ cd /var/lib/ambari-server/resources/views/$ curl -O https://s3-us-west-2.amazonaws.com/pivotal-ambari/stable/hawq-view/hawq-view-1.0.0.0-1.jar$ ambari-server restartCreating an Instance of HAWQ View: a) The HAWQ View instance connects to the HAWQ Master through JDBC. Ensure that HAWQ pg_hba.conf is upd

Livy: Livy is an open source REST interface for interacting with Spark. Authorized users can launch a Spark session and submit code. Two different users can access their own private data and session, and they can collaborate on a notebook. Only the Livy server can submit a job securely to a Spark session.Steps to follow to configure livy interpreter to work with secure HDP cluster:Setup proxy for livy interpreter in core-site.xmlGo to Ambari->HDFS->config->customer-core-site and add below properties: hadoop.proxyuser.livy.groups=* hadoop.proxyuser.livy.hosts=*2. Configure livy interpreter in Zeppelin and add below configurations:livy.superusers=zeppelin-spark Note - The value for livy.superusers should be your zeppelin principal. That would be zeppelin-{$Cluster_name} For example, in this case you can find it by running below command: klist -kt /etc/security/keytabs/zeppelin.server.kerberos.keytabKeytab name: FILE:/etc/security/keytabs/zeppelin.server.kerberos.keytabKVNO Times

Apache Ranger uses an embedded Tomcat server to provide the Web UI functionality for administration of Ranger. A previous HCC article provided details on maintenance of the log files that are managed by the log4j configuration, including xa_portal.log, ranger_admin_perf.log, xa_portal_sql.log. We're going to focus on maintenance of the access_log* logs that get automatically generated by Tomcat, but which are not managed by this log4j configuration. With embedded Tomcat, the configuration is contained within the code for the AccessLogValve (as you can see, it uses an hourly rotation pattern unless overridden by ranger.accesslog.dateformat). We'll use the logrotate application in CentOS/RHEL to manage these access_log* logs as the number of files can grow large without rotation and removal in place. You can check to see how many of these files you have on your Ranger Admin node by running (there would be one access_log* file per hour for each day during which the service has ran con

This is the first of a series of short articles on using Apache Spark with Hortonworks HDP for beginners. If you’re reading this, you don’t need need me to define what Spark is, there are numerous references on the web that can speak about that its API, being data structure centric and in my opinion one of the most important Open Source projects. The intent of this article is let you know what helped me get started using Spark on the Hortonworks Data Platform. This is not a tutorial.I’m assuming you have access to a an HDP 2.5.3 cluster or to the Hortonworks Sandbox for HDP 2.5.3 or above. Also, I’m going to assume that you are familiar with SQL, Apache Hive, and using the Linux/Unix Bourne Shell. The problem I was having, which pushed me to use Spark was that using Hive for data processing was limiting. There were things I was used to using in my past life as an Oracle DBA that were not available. Hive is a fantastic product, but Hive SQL didn’t give me all the bells and whist

Just some clarifications to the instructions provided in this article. Hopefully, it will save you time to get you up and running faster.1) Nifi 401:Unauthorized ErrorIf you see a similar error message in the Nifi console:ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.standard.PostHTTP PostHTTP[id=834bb9f9-a15d-42bd-8d7a-3f00c810d729] Failed to Post StandardFlowFileRecord[uuid=bc74e1c5-12e7-4da2-93b4-a3dc624218ac,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1471370511531-1, container=default, section=1], offset=11780, length=407],offset=0,name=12406098021784,size=407] to http://gcm-http.googleapis.com/gcm/send: response code was 401:Unauthorizedand when you click "Notify Customer" in the Analyst's console and no event is updated on the Android emulator in Android Studio.Resolution 401:Unauthorized ErrorOpen open the UpdateAttribute processor and add a new property called Authorization and set its value to key=your google browser key2) Wrong Google API n

# Ruby Scripting in NiFiToday I was asked about an issue that I didn't know how to solve using NiFi. On the surface it sounded simple; just map an attribute to another value. The attribute to map was 'port' and based on the port number just add an attribute to more easily identify the system downstream. E.g. for port 10003; syslog:local, for 10004; syslog:db1, etc. After a bit of digging I found a few different options to solve this.## Many UpdateAttribute ProcessorsThe first is to create a new UpdateAttribute processor for each incoming stream. This labels (places an attribute) on all files that come in from that listener. It looked like this:This looks a bit confusing and tedious but is very precise and arguably easier to read, especially when we label the processors. It also has the added advantage of not having to change the port in more than one place. If for example, the local logs com

bug (HIVE-15355) is being worked on by Hive engineering team at HortonworksYou can use the following workaround :1- use "SORT BY 0" at the end of the query which will force to use one reducer , please use this only if you have a small query.2- try to use set hive.mv.files.thread=0; before running the query.If you have any question regarding the above, please let me know.

Introduction Using NiFi, data can be exposed in such a way that a receiver can pull from it by adding an Output Port to the root process group. For Storm, we will use this same mechanism - we will use the Site-to-Site protocol to pull data from NiFi's Output Ports. In this tutorial we learn to capture NiFi app log from the Sandbox and parse it using Java regex and ingest it to Phoenix via Storm or Directly using NiFi PutSql Processor. Prerequisites 1) Assuming you already have latest version of NiFi-1.x/HDF-2.x downloaded as zip file (HDF and HDP cannot be managed by Ambari on same nodes as of now) on to your HW Sandbox Version 2.5, else execute below after ssh connectivity to sandbox is established: # cd /opt/ # wget http://public-repo-1.hortonworks.com.s3.amazonaws.com/HDF/centos6/2.x/updates/2.0.1.0/HDF-2.0.1.0-centos6-tars-tarball.tar.gz # tar -xvf HDF-2.0.1.0-12.tar.gz 2) Storm, Zeppelin are Installed on your VM and started. 3) Hbase is Installed with phoeix Query Server. 4) Mak

Prerequisites: HDP 2.5 on Hortonworks Sandbox up and running.Apache NiFi installed and up and running: To quickly install Apache NiFi on the HDP 2.5 Sandbox by Ambari wizard, check Section 2 on the NiFi tutorial.Assumes a minimum of knowledge on how to access Ambari HDFS View, Hive View and Zeppelin UI on the sandbox.This demo has only been tested and developed for the HDP 2.5 Sandbox OS, which is CentOS 6.8. The NiFi flow may need to be tweaked depending on the target OS where the messages are generated. Enable rsyslog on your Sandbox:In this tutorial, we will enable rsyslog to forward syslog messages to localhost on port 7780 as following:- Edit /etc/rsyslog.conf and add the following line:*.* @127.0.0.1:7780- Restart syslog for the changes to take effect:[root@sandbox ~]# service rsyslog restartShutting down system logger: [ OK ]Starting system logger: [ OK ]- You can optionally read from the port 7780 to make s

If you are getting below error while refreshing client configs via Ambari UI then here are the steps to fix it! 1. Remove KERBEROS_CLIENT for problematic hosts using below curl call curl -H "X-Requested-By:ambari"-u admin:admin -X DELETE "http://<AMBARI-SERVER>:8080/api/v1/clusters/<cluster-name>/hosts/<host-name>/host_components/KERBEROS_CLIENT" 2. Ensure from Ambari UI that KERBEROS_CLIENT got removed. 3. From Ambari UI --> goto the problematic host --> click on Add button --> select Kerberos Client 4. Installation should complete without any issue.

Metron provides a Kibana-based UI that is designed to be a single pane of glass to utilize the big data approach (having all data available to you at the same time) to filter through the irrelevant and display the just the information, alerts, and context an analyst/investigator is looking for all on the same pane. The Metron UI has several advantages over conventional SIEM tools, including flexibility, and having the needle as well as all the context for the needle presented together on the same screen, requiring no jumping around from console to console to gather the information. Let us first describe the parts of the UI panel. Specifically there are 4 sections: Pinned Queries and Filters, Ingest Histogram Panels, Detailed Message Tables, and finally the PCAP panel. We will go through all the parts in detail in this article and then we will how how to tie everything together and create a single pane of glass interface to filter through data and create context for the search.The f

Incremental Updates: A Strategy for Appending Records in HiveAs part of the Stinger.next initiative, we expect to see Hive supporting full CRUD operations (Insert, Select, Update, Delete). While some transaction support exists for Hive today, it is still evolving with regard to performance and scalability. In the meantime, customers may still need to work with the traditional options for Hive table integration: OVERWRITE or APPEND. The OVERWRITE option requires moving the complete record set from source to Hadoop. While this approach may work for smaller data sets, it may be prohibitive with regard to scale.The APPEND option can limit data movement to only new or updated records. As true Inserts and Updates are not yet available in Hive, we need to consider a process of preventing duplicate records as Updates are appended to the cumulative record set.In this article, we will look at a strategy for appending Updates and Inserts from delimited and RDBMS sources to existing Hive tabl

IntroductionApache Zeppelin is a web-based notebook that enables interactive data analytics while Apache Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs. Pig-latin is a very powerful languages for data flow processing. One drawback of pig community complains about is that pig-latin is not a standard language like sql so very few BI tools integrate with it. So it is pretty to hard to visualize the result from pig. Now the good news is that Pig is integrated in zeppelin 0.7 where you can write pig latin and visualize the result.Use pig interpreterPig interpreter is supported from zeppelin 0.7.0, so first you need to install zeppelin, you can refer this link for how to install and start zeppelin.Zeppelin supports 2 kinds of pig interpreters for now.%pig (default interpreter)%pig.query%pig is like the pig grunt shell. Anything you can run in pig grunt shell can be run in %pig.script interpreter, it is used for run

You can run the attack library for OSX or Linux from an edge node or from outside the cluster.I ran against mine from my OSX laptop against my cluster that I had network access to.You should try to scan from inside your network, from an edge node and from a remote site on the Internet.You will need Python 2.7 or Python 3.x installed first.git clone git@github.com:CERT-W/hadoop-attack-library.git pip install requests lxmlYou may need root or sudo access to install on your machine.One of the scanners hits the WebHDFS link that you may have seen a warning about.python hdfsbrowser.py timscluster Beginning to test services accessibility using default ports ...Testing service WebHDFS[+] Service WebHDFS is availableTesting service HttpFS[-] Exception during requesting the service[+] Sucessfully retrieved 1 servicesdrwxrwxrwx hdfs:hdfs 2017-01-15T05:50:27+0000 /drwxrwxrwx yarn:hadoop 2017-01-11T19:25:26+0000 app-logs /app-logsdrwxrwxrwx hdfs:hdfs 2016-12-21T23:12:40+0000 apps /app

ProblemWhen trying to start the secondary hive metastore service, in this example, zookeeper is unable to create the znode with the appropriate permissions. This can be seen mostly on the non-Ambari managed clusters.17/01/19 18:25:17 ERROR metastore.HiveMetaStore: Metastore Thrift Server threw an exception...org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hivedelegationMETASTORE/keys        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166)        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236)        at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:473)        at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:92)        at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6031)        at org.apache.hadoop.h

We just updated Hortonworks Data Cloud for AWS to Technical Preview #1.12. The release is packed with goodies such as: Support for deploying compute nodes with spot pricing.Support for executing node recipes - custom scripts that can be run pre- or post- cluster deployment for customizing the cluster and installing additional software.Support for HDP 2.6 (Technical Preview) that can launch two new cluster configurations for Spark 2.1 and Druid. To create an HDP 2.6 cluster, launch the cloud controller and when creating a cluster choose HDP Version: HDP 2.6 (Technical Preview) and then choose one of the available cluster types:For more details, refer to the Release Notes: http://hortonworks.github.io/hdp-aws/releasenotes/.To get started with HDCloud for AWS, visit http://hortonworks.github.io/hdp-aws/.To get started with Spark 2.1, see Vinay's blog at http://hortonworks.com/blog/try-apache-spark-2-1-zeppelin-hortonworks-data-cloud/. Have fun!

Hive Compare Tables and Databases before and after Upgrades.Run below script to collect all databases.hive -e "show databases" | sed "s/ *$//g" | sed "s/^ *//g" | sort 1>databaselist Run below script to collect all tables in database.for dbname in `cat databaselist` ; do hive -e "use ${dbname} ; show tables ;" | sed "s/ *$//g" | sed "s/^ *//g" | sort 1> tablelist; done ; Execute Scripts above before upgrade and once after upgrade .Simply do vimdiff afterwards to identify the difference if any. Happy Migrating or Upgrading.