Community Articles

SYMPTOM: Installing Ranger Admin with Ambari 2.2.1, I got the following exception during the installation Ranger Admin installation failed - xa_audit_db_postgres.sql DB schema import failed!ERROR: ### 2016-05-10 10:25:47,952 - Execute['python /usr/hdp/current/ranger-admin/db_setup.py'] {'logoutput': True, 'environment': {'RANGER_ADMIN_HOME': '/usr/hdp/current/ranger-admin', 'JAVA_HOME': '/usr/lib/jvm/jre-1.7.0-openjdk.x86_64'}, 'user': 'ranger'} 2016-05-10 10:25:48,081 [I] DB FLAVOR :POSTGRES 2016-05-10 10:25:48,081 [I] --------- Verifying Ranger DB connection --------- 2016-05-10 10:25:48,081 [I] Checking connection 2016-05-10 10:25:48,332 [I] connection success 2016-05-10 10:25:48,332 [I] --------- Verifying Ranger DB tables --------- 2016-05-10 10:25:48,332 [I] Verifying table x_portal_user in database ranger 2016-05-10 10:25:48,583 [I] Table x_portal_user already exists in database ranger 2016-05-10 10:25:48,583 [I] --------- Verifying upgrade history table --------- 2016-05-10 10:

Episode 94 – Roaring newshttps://roaringelephant.org/2018/06/26/episode-94-roaring-news/I this weeks edition of Roaring Big Data News, Dave talks about modernizing Hadoop and a billion java errors. Jhon has an article on improving your learning data sets. We finish with a discussion about the newly released HDP 2.6.5 with an emphasis on the deprecation notices and Yarn Containers. Play in new window | Download (Duration: 37:40 — 26.1MB)Dave Modernizing Hadoop: Reaching the plateau of productivity https://www.zdnet.com/article/modernizing-hadoop-reaching-the-plateau-of-productivity/1 billion Java errors, here’s what causes 97% of them https://blog.takipi.com/we-crunched-1-billion-java-logged-errors-heres-what-causes-97-of-them/https://blog.takipi.com/the-top-10-exceptions-types-in-production-java-applications-based-on-1b-events/Jhon Why you need to improve your training data, and how to do it https://petewarden.com/2018/05/28/why-you-need-to-improve-your-training-data-and-how-t

See Part 1: https://community.hortonworks.com/articles/101679/iot-ingesting-gps-data-from-raspberry-pi-zero-wire.html Augment Your Streaming Ingested Data So with Hortonworks Data Flow : Flow Processing I can augment any stream of data in motion. My GPS data stream from my RPIWZ is returning latitude and longitude. That is valuable data that can be input to a lot of different APIs. One that seemed relevant to me was the current weather conditions where the device was. You could also do some predictive analytics to figure where you might be and check the current weather conditions or a weather forecast depending on how long it will take you to get there. A GPS is giving us latitude and longitude that can be used as input to many kinds of APIs. Once use case that works for a moving device is checking the current weather conditions where you are. Weather Underground has a REST API call that will convert Lat/Long to City/State (there are other APIs that will do this as well,

Short Description: This article includes manual steps to be performed, if a cluster is setup with Hdfs, Ranger and Ranger KMS services and Ranger service is syncing users from LDAP/AD and Hadoop group mapping is configured using SSSD. Steps provided will help Ranger KMS service to resolve groups with spaces or no name as same as in Hadoop.Article:To resolve groups with spaces or no name, Ranger KMS process should have access to hadoop native library path for using JniBasedUnixGroupsMapping class.Steps for Ambari 2.6.x version: Create a sh file ranger-kms-env-javaopts.sh in Ranger KMS conf directory which will specify -Djava.library.path.vim /usr/hdp/current/ranger-kms/conf/ranger-kms-env-javaopts.sh Add below content and save ranger-kms-env-javaopts.sh file.export JAVA_OPTS=" ${JAVA_OPTS} -Djava.library.path=${JAVA_LIBRARY_PATH}:/usr/hdp/current/hadoop-client/lib/native " Update the ownership of ranger-kms-env-javaopts.sh with the user which is used to start Ranger KMS process.

Episode 93 – Apache Kylin: Extreme OLAP Engine for Big Datahttps://roaringelephant.org/2018/06/19/episode-93-apache-kylin-olap-cubes-in-hadoop/In this episode Apache PMC member Dong Li joins us to explains how Apache Kylin can deploy Analytical OLAP cubes in your Big Data environment. http://kylin.apache.org/ Play in new window | Download (Duration: 46:14 — 32.0MB)Dong LiTechnical Partner & Senior Architect of Kyligence (linkedin)PMC Member of Apache Kylinhttp://en.kyligence.io/Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

Was trying to dig up some TSP benchmark info for nifi listenhttp, which allows for providing a rest proxy, with no luck, so i tried to create one myself. This is a very rough effort, i can improve it by capturing how many client instances i have running , so we can see the TPS drop and rise as the clients drop and rise. At some point i ran out of servers and resources to run more clients, you will the chart drops off at the end , before going back up. This is where i noticed some of my clients had crashed and i restarted them.Anyway, getting to the matter. PurposeThe benchmark only meassure how much load can a ListenHTTP processor handle , when subjected to real world traffic.SetupThe nifi cluster is setup on an m4.4xlarge ( 16 cores CPU, 32 GB RAM), The node is also hosting the kafka broker and zookeeper. HDF version is 3.1.1The NiFi is a simple Listenhttp processor forwarding to updateattribute. updateattribute burns the flowfile. The idea was to only measure Listenhttp performance

The purpose of this document is how to leverage “R” to predict HDFS growth assuming we have access to the latest fsimage of a given cluster. This way we can forecast how much capacity would need to be added to the cluster ahead of time. In case of on-prem clusters, ordering H/W can be a lengthy process and for some organizations, it can take months till they can actually add capacity to their clusters. Giving teams managing Hadoop clusters an insight on HDFS growth rate can help with placing H/W orders ahead of timeA prereq. for this script, you would need access to a recent fsimage file, R and a machine with hadoop binaries installed on it to run OIV.Example: hdfs oiv -i fsimage_xxxxx -p Delimited > fsimage.outLoading required librarieslibrary(dplyr)library(anytime)library(lubridate)library(prophet)library(rmarkdown) --> only needed if you are using RStudio and planning on exporting the script to a PDF or Word formatData Filtering & manipulationThe only three columns we need

Launching Cloudbreak on GCPBefore launching Cloudbreak on GCP, you must meet the following prerequisites.Meet the PrerequisitesBefore launching Cloudbreak on GCP, you must meet the following prerequisites. GCP AccountService AccountSSH Key PairRegion and zoneGCP accountIn order to launch Cloudbreak on GCP, you must log in to your GCP account. If you don't have an account, you can create one at https://console.cloud.google.com. Once you log in to your GCP account, you must either create a project or use an existing project.To create a new project, provide name, choose organization or leave it as No Organization.Now Select this newly created Project.On the main dashboard page, you will find the Project ID. You will need this to define your credential in Cloudbbreak in a later stepGCP - APIs DashboardGo to the Service Accounts screen by (1) clicking the menu in the top left, (2) hovering over APIs and Services, and (3) Clicking on Dashboard.GCP - APIs & Services DashboardVerify that t

According to default Oozie log4j configuration in Ambari - Log get's rotated by every hour and retention is set to 30 days..log4j.appender.oozie=org.apache.log4j.rolling.RollingFileAppenderlog4j.appender.oozie.RollingPolicy=org.apache.oozie.util.OozieRollingPolicylog4j.appender.oozie.File=${oozie.log.dir}/oozie.loglog4j.appender.oozie.Append=truelog4j.appender.oozie.layout=org.apache.log4j.PatternLayoutlog4j.appender.oozie.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - SERVER[${oozie.instance.id}] %m%n# The FileNamePattern must end with "-%d{yyyy-MM-dd-HH}.gz" or "-%d{yyyy-MM-dd-HH}" and also start with the# value of log4j.appender.oozie.Filelog4j.appender.oozie.RollingPolicy.FileNamePattern=${log4j.appender.oozie.File}-%d{yyyy-MM-dd-HH}# The MaxHistory controls how many log files will be retained (720 hours / 24 hours per day = 30 days); -1 to disablelog4j.appender.oozie.RollingPolicy.MaxHistory={{oozie_log_maxhistory}}.If you want to configure DRFA to roll the log file daily, pl

Requirement: HDP 2.6.1 Fix version is knox: 0.12.0, according to KNOX-841 Method I: Using "solr" as data service {/usr/hdp/current/knox-server/data/services/solr}Add below in Ambari UI > KNOX > Configs > Advanced topology for Solr UI. default.xml <service> <role>SOLR</role> <url>http://<solr-hostname>:8983</url> </service> Make sure service def is in Upper Case. Restart KNOX Use below URL for access Solr UI using KNOX https://<knox-hostname>:8443/gateway/default/solrMethod II: Using "solrapi" as data service {/usr/hdp/current/knox-server/data/services/solrapi} for Banana UI KNOX-528Add below in Ambari UI > KNOX > Configs > Advanced topology for Solr UI.default.xml <service> <role>SOLRAPI</role> <url>http://<solr-hostname>:8983/solr</url> </service> Login into sorl server:Please make

Row and Columnar Storage For Hive Customers often ask about columnar storage formats for Hive tables and when to use them. It depends on your uses cases. If your data access patterns mostly involve selecting a few columns to perform aggregations, then using columnar storage will save disk space, reduce I/O when fetching data, and improve query execution time. In this article I tested the columnar ORC file format for Hive to quantify how it outperforms row based Text files when used for queries that aggregate and group on a few columns from a table with many columns. ORC is a columnar storage format used in Hadoop for Hive tables. It is an efficient file format for storing data in which records contain many columns. An example is Clickstream (web) data to analyze website activity and performance. Queries typically retrieve a subset of columns for each row.SUMMARY OF TEST RESULTS: Comparing ORC vs Text files for storing 22 million clickstream page view records in Hive tables

1.Prerequisites Enable HDFS location access for the HIve table Enable HDFS Policy in Ranger Restrict POSIX access in HDFS File system Grant Hive table access in Ranger 2. Enable HDFS location access for the user Login to Ranger and select HDFS Policy Make sure to mention the Hive table location (In below example “Resource Path” points to default Hive Warehouse location) Validate the HDFS location access is restricted to ensure Ranger policy is working as expected 3. Create table policy in Hive Access if not exists Select the Hive Policy Add a new policy if not exists Grant table access to required users and validate the access by querying the table 4. Enable table policy in Hive Column Masking Select “Masking” under Hive policy Select “Add New Policy” Provide the required information in the Policy Details. Select the Masking option as per the requirement Below example restricts access for “Sales1” user on column “ip” by masking it using Hash option Before applying Masking

1. Prerequisites Install Ranger Install Knox Test in lower environments Inform stakeholders or plan for short outage as few services requires a restart Identify the IP addresses and users to allow access 2. Enable Knox-Ranger Plugin in Ambari Login to Ambari > select Ranger > configs > Ranger Plugin Enable Knox Plugin by clicking the “off” button, and restart the required services Ambari suggests 3. Create a Knox Policy for Hive After enabling the Knox Plugin for Ranger in Ambari, Knox policy should be automatically displayed in Ranger. Select the Knox Policy Add a new policy, if not exists Grant groups/users access to required IP address 4. Connections Test Below are the IPs I considered for testing 172.25.39.156 172.25.40.41 Test Cases: Allow connections through any IPs for the group “sales” Allow only connection access through IP range for group “sales” Case 1: Allow connections through Knox → Hive for any IPs for the group “sales” Test connection from both the IP

ObjectiveUse Oozie's Hive 2 Action to create a workflow which will connect to Hive Serve 2 in a Kerberized environment. Execute a Hive query script which will sum the number of crimes in the crimes database table for a particular year - passed in as a parameter queryYear in thejob.properties file. Write the results to a new Hive table - crimenumbersProcedureLog into the edge server containing the Oozie client. Change users to the oozie user.[root@jyoung-hdp234-1 ~]# su - oozieAuthenticate to the KDC using the oozie service account kerberos keytab[oozie@jyoung-hdp234-1 ~]$ kinit -kt /etc/security/keytabs/oozie.service.keytab oozie/jyoung-hdp234-1.openstacklocal@EXAMPLE.COMDownload the City of Chicago crime data in CSV form.[oozie@jyoung-hdp234-1 ~]$ mkdir -p /tmp/crime[oozie@jyoung-hdp234-1 ~]$ cd /tmp/crime[oozie@jyoung-hdp234-1 crime]$ curl -o crime -L https://data.cityofchicago.org/api/views/ijzp-q8t2/rows.csv?accessType=DOWNLOADPut the crime csv into HDFS into the /tmp/crime folder.

If you looking to upload template using NIFI REST API, below command curl command upload the template as multipart form postcurl -XPOST -H "Authorization: Bearer {{ token }}" https://localhost:9091/nifi-api/process-groups/10c263d0-0160-1000-0000-00006a157654/templates/upload -k -v -F template=@NifiTemplate.xml

Sqoop is used for importing data from multiple sources onto HDFS. One of the most common use can is to use Hive imports sqoop import --connect jdbc:mysql://localhost:3306/sqoop --username root -P --split-by id --columns id,name --table customer --target-dir /user/cloudera/ingest/raw/customers --fields-terminated-by "," --hive-import --create-hive-table --hive-table sqoop_workspace.customersIf you want to specify a specific Queue for sqoop job -Dmapred.job.queuename=queueName needs to be immediately added after the import keyword. sqoop import -Dmapred.job.queuename=queueName --connect jdbc:mysql://localhost:3306/sqoop --username root -P --split-by id --columns id,name --table customer --target-dir /user/cloudera/ingest/raw/customers --fields-terminated-by "," --hive-import --create-hive-table --hive-table sqoop_workspace.customers
This will launch the sqoop job in the specific queu but the hive job will be launched in the default queue. To launch the hive job is specific queue

Perform below steps on the servers chosen for Datanodes having different OS version 1. Prerequisites Minimum Software Requirements: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_support-matrices/content/ch_matrices-ambari.html Check and set the maximum Open File descriptors on new host https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_support-matrices/content/ch_matrices-ambari.html Check DNS and NSCD https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-installation/content/check_dns.html Disable THP Disable SELinux Enable NTP Swappiness Install same Python version Install same JDK Version   2. Manage Stack Version Go to Ambari Versions page:

If a user has partial access to a Hive table access via ranger policies, he/she won’t be able to run the describe statement against this table. As per screenshot below, user “yarn” has access to all columns of customer table except column “c_last_name”.If user “yarn” tries to run “describe customer;” a Permission Denied message will be displayed.To overcome this inconvenience, we can add “xasecure.hive.describetable.showcolumns.authorization.option=show-all” property in “Custom ranger-hive-security” section in Hive configs via ambari and restart Hive serviceOnce restart is done, try running the describe statement again. We should be able to see the table description.

This is a guide to walk a user through enabling high availability for Oozie in an Ambari-managed cluster in which Kerberos has been enabled. It is assume that Ambari 2.4.0 (or above) is installed and Kerberos has been previously set up. However, if Kerberos has not yet been set up, the Kerberos-related items can be performed when Kerberos is enabled. Also, it is expected that an Oozie HA supported database is configured. A database such as MySQL or Oracle needs to have been configured since the Derby database is not sufficient. MotivationMost documentation properly shows the steps to enable Oozie HA; however when Kerberos is involved, some configuration properties need to be set in an alternate way. According some documentation, when HA is setup in an environment where Kerberos is enabled, the following properties must be set:oozie-site/oozie.authentication.kerberos.principal = *oozie-site/oozie.authentication.kerberos.keytab = /etc/security/keytabs/oozie.ha.keytabIf this is fol

Starting with version 2.6.5 on the 2.x line and 3.0.0 on the 3.x line, HDP has added support for running workloads against Google Cloud Storage (GCS). This article talks about how to get started using GCS for workloads.The FileSystem Abstraction Apache Hadoop includes a ‘FileSystem’ abstraction, which is used to access data on HDFS. This is an abstraction, which can be used to talk not just to HDFS, but any other storage layer which has an implementation of the FileSystem abstract class for the specific storage layer. Existing applications don’t need to change, to make use of GCS. Instead, some configuration and path changes is all that is required for existing jobs to run against Cloud Storage.Google Storage Connector The Google Storage Connector provides an implementation of the FileSystem abstraction, and is available in HDP versions mentioned above, to facilitate access to GCS. “gs” is the filesystem prefix used for Google Cloud Storage. e.g. gs://tmpSetting up Credentials

While upgrading the Ambari server from 2.6.2 to Ambari 2.7.0.0-897 ,the ambari-server upgrade command fails with dependency unavailability.[root@c2140-node4 yum.repos.d]# ambari-server upgrade Using python /usr/bin/python Upgrading ambari-serverTraceback (most recent call last): File "/usr/sbin/ambari-server.py", line 35, in <module> from ambari_commons.os_utils import remove_file File "/usr/lib/ambari-server/lib/ambari_commons/os_utils.py", line 39, in <module> from ambari_commons.os_linux import os_change_owner, os_getpass, os_is_root, os_run_os_command, \ File "/usr/lib/ambari-server/lib/ambari_commons/os_linux.py", line 25, in <module> from ambari_commons import subprocess32 File "/usr/lib/ambari-server/lib/ambari_commons/subprocess32.py", line 146, in <module> import importlibImportError: No module named importlibSolution The simple solution is to install the dependency python-importlib.noarch[root@c2140-node4 yum.repos.d]# yum install pyth

This is applicable for CentOS7/RHEL6 - for CentOS7/RHEL7 please follow https://community.hortonworks.com/articles/188269/javapython-updates-and-ambari-agent-tls-settings.htmlUpgrading to jdk1.8.0_171 version disables some of the TLSv1_1 TLSv1 protocols and algorithms. With this only option is using TLS1_2 version but CentOS6/RHEL6 uses Python 2.6 and it does not support TLS1_2. Agent-server communication would fail with below error.WARNING 2018-04-24 16:35:10,989 NetUtil.py:124 - Server at https://***.***.***.***:8440 is not reachable, sleeping for 10 seconds...INFO 2018-04-24 16:35:20,990 NetUtil.py:70 - Connecting to https://***.***.***.***:8440/caERROR 2018-04-24 16:35:20,991 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)ERROR 2018-04-24 16:35:20,991 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions. To workaround this problem you can tweak /usr/jdk64/jdk1.8.0_112/jre/lib/security/java.security file in Ambari server host to enab

Nifi build on HDF 3.1.2 and HDF 3.1.0 fail with a depency issue when trying to push data to ADLS. This is because the new version of ADLS has some dependency on hadoop 2.8 feature, which is not available in 2.7.3 which is referenced by nifi.to fix this you can build nifi again. You could eith build it against hadoop 2.8 or againt hdp 2.6.x which should have the classes that ADLS depends on.to do thatgit clone nifi repositorycd <nif-repo-home>/nifi-nar-bundles/nifi-hadoop-libraries-bundle/nifi-hadoop-libraries-nar/vi pom.xmladd a hadoop.version property to the pom.xml as shown below. if already set, no change is needed.change nifi version to match the nifi version you are running for the parent<parent> <groupId>org.apache.nifi</groupId> <artifactId>nifi-hadoop-libraries-bundle</artifactId> <version>1.5.0.3.1.2.0-7</version> </parent> <artifactId>nifi-hadoop-libraries-nar</artifactId> <pac

Note : This feature is available from HDP 3.0 (Ambari 2.7) Ambari 2.7 has a cool new feature where it is integrated with Swagger and you can try and explore all the REST APIs. Steps to use Swagger Login to Ambari Hit this url ( http://{ambari-host}:8080/api-docs) This page takes you to the API explorer where you can try different APIs. Here are some of the screenshots. You can get all the supported endpoints from http://{ambari-host}:8080/api-docs/swagger.json) . Hope this helps 🙂

This tutorial will be designed to extent my other work, and the goal is to provide the reader a frame work for a complete end-to-end solution for sentiment analysis on streaming Twitter data. The solution is going to use many different components in the HDF/HDP stack including NiFi, Solr, Spark, and Zeppelin, and it will also utilize libraries that are available in the Open Source Community. I have included the Reference Architecture, which depicts the order of events for the solution. *Note: Before starting this tutorial, the reader would be best served reading my other tutorials listed below:Twitter Sentiment using Spark Core NLP in Apache ZeppelinConnecting Solr to Spark - Apache Zeppelin Notebook*PrerequisitesEstablished Apache Solr collections called "Tweets" and "TweetSentiment"Downloaded and installed Open Scoring Completed this Tutorial: Build and Convert a Spark NLP Pipeline into PMML in Apache ZeppelinStep 1 - Download and deploy NiFi flow.The NiFi Flow can be found here: t