Community Articles

Problem:Following upgrade to 2.5 Oozie Sqoop job which previously worked now fails withoozi-W@sqoop-aa6c] Launcher exception: org/json/JSONObjectjava.lang.NoClassDefFoundError: org/json/JSONObject at org.apache.sqoop.util.SqoopJsonUtil.getJsonStringforMap(SqoopJsonUtil.java:43) at org.apache.sqoop.SqoopOptions.writeProperties(SqoopOptions.java:759) at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.createInternal(HsqldbJobStorage.java:399) at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.create(HsqldbJobStorage.java:379)Solution:Ensure that 'java-json.jar' file exists in the following locations: /usr/hdp/current/sqoop-client/lib /usr/hdp/<version>/sqoop/lib Also copy this jar into HDFS and reference it in your oozie workflow:e.g. hdfs dfs -put /usr/hdp/current/sqoop-client/lib/java-json.jar /user/oozie/oozie_scripts/lib/and change workflow.xml: <archive>/user/oozie/oozie_scripts/lib/java-json.jar#java-jso

Below are the steps for Oozie database migration from Derby to Postgresql..Step 1 - Have Postgresql server installed and ready to be configured..Step 2 - Stop Oozie service from Ambari UI..Step 3 - Install Postgresql JDBC connector.yum install postgresql-jdbc.Step 4 - On Ambari Server, run below commandambari-server setup --jdbc-db=postgres --jdbc-driver=/usr/share/java/postgresql-jdbc.jarNote - Please pass appropriate driver if /usr/share/java/postgresql-jdbc.jar does not exists. ..Step 5 - Login to Postgesql DB as postgres user and create a blank 'oozie' database and grant required permissions to the 'oozie' user.[root@ambaview ~]# su - postgres-bash-4.1$ psqlpsql (8.4.20)Type "help" for help.— postgres=# CREATE DATABASE oozie;CREATE DATABASEpostgres=#— CREATE USER oozie WITH PASSWORD 'oozie’;postgres=# CREATE USER oozie WITH PASSWORD 'oozie';CREATE ROLEpostgres=#— GRANT ALL PRIVILEGES ON DATABASE oozie TO oozie;postgres=# GRANT ALL PRIVILEGES ON DATABASE oozie TO oozie;GRANTpostgres

Affected versions:1.2.x, 1.3.0Symptoms:The SmartSense view if not displayed even if the HST-Server is up and running. Also , refer the steps that were executed during the upgrade process . It is possible this error has occurred during “hst upgrade-ambari-service” step.Un-installing old view ... Installing new view ... ('request method', 'GET') ('request data', None) ('request url', 'http://ambari-server.abcxyz.net:8083/api/v1/views/HORTONWORKS_SMARTSENSE') ('request type', 'http') ('request host', 'ambari-server.abcxyz.net:8083') ('request selector', '/api/v1/views/HORTONWORKS_SMARTSENSE') ('request header', [('Host', 'ambari-server.abcxyz.net:8083'), ('Authorization', 'Basic YWRtaW46Y24zild6MmT9ODVw'), ('User-agent', 'Python-urllib/2.7')]) ('response code', None) ('response content', None) Traceback (most recent call last): File "/usr/sbin/hst-server.py", line 1384, in <module> main(sys.argv) File "/usr/sbin/hst-server.py", line 1361, in main upgrade_ambari_plugin(options) File

Affected versions:1.2.x, 1.3.0Symptoms: 2-way ssl setup fails with an error as given below. This error message is available in hst-server.log or hst-gateway.logINFO 2016-08-25 06:16:00,690 security.py:182 - Agent key not exists, generating requestINFO 2016-08-25 06:16:00,691 security.py:242 - openssl req -new -newkey rsa:1024 -nodes -keyout "/var/lib/smartsense/hst-agent/keys/hdp1-hrdd-vm1.akrocl34v0sjr9flwro4laslcd.gx.internal.cloudapp.net.key" -subj /OU=hdp1-hrdd-vm1.akrocl34v0sjr9flwro4laslcd.gx.internal.cloudapp.net/ -out "/var/lib/smartsense/hst-agent/keys/hdp1-hrdd-vm1.akrocl34v0sjr9flwro4laslcd.gx.internal.cloudapp.net.csr" was finished with exit code: 1 - an error occurred parsing the command options.INFO 2016-08-25 06:16:00,711 security.py:188 - Agent certificate not exists, sending sign request Reason: openssl does not support SubjectName longer than 64 chars. SubjAltName should be used for longer host names.#openssl req -new -newkey rsa:1024 -nodes -keyout "./hdp1-hrdd-vm1.a

Affected versions:1.3.xSymptoms: When tried to install SmartSense then it failed on Activity Analyzer installation which is the first step of SmartSense installation.File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in _getattr_raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")resource_management.core.exceptions.Fail: Configuration parameter 'hadoop-env' was not found in configurations dictionary!Similar error occurs when trying to reinstall HST SERVER then it failed with similar message.File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in _getattr_raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")resource_management.core.exceptions.Fail: Configuration parameter 'namenode_host' was not found in configurations dictionary!Reason: SmartSense components assumed that hdfs will

Affected versions:1.3.xSymptoms: A Support bundle capture fails if case number starts with 0, like below :hst-agent.py: error: option -c: invalid integer value: '00100870'Reason: The root cause of this issue is that the case number is handled as octal based, due to the leading zero. After that is converted to decimal base.Solution:Fixed in 1.4.0 and remove the preceding zeros for 1.3.x and capture again

Affected versions: 1.2.x, 1.3.x Symptoms: hst-agent thinks it has uploaded its node file; however, hst-server doesn't find the file when it tries to use it. Below error reported in the hst-server.log file :15 Feb 2017 15:09:35,033 INFO Thread-1 BundleMonitor:286 - Uploaded archive /hadoop/1/smartsense/hst-server/tmp/a-00000000-c-00000000_devhdpvm01_0_2017-02-15_15-08-02/hdwduv0073.odevkrj.jgcz.group-a-00000000-c-00000000_devhdpvm01_0_2017-02-15_15-08-02.tgz doesn't exists. Reason: hst-server and hst agents have a common directory on the same shared mount where node bundles from each agent are uploaded and clean up before consuming by the hst-server. Solution: Fixed in 1.4.0 and HOTFIX-651 for 1.3.x Otherwise, create separate directories for each agent / server in the shared mount or replace the share mount with local filesystem.

When one stands up SolrCloud/HDPSearch, every core/replica will be writing logs to /opt/lucidworks-hdpsearch/solr/server/logs. Files like 'solr-8983-console.log' will grow very quickly for a lightly used index, at the rate of several GBs per day. The reason is it outputs all INFO and DEBUG messages to this console log. Which is great for development, but a pain for production and operations. The challenge is there are quite a few log configuration files spread around the Solr directories, and those will not necessarily yield the desired result. It gets even more complicated for a SolrCloud, where one is dealing with a cluster of nodes. There is, however, a very straightforward way to quickly address this issue: Navigate to the admin console, like http://xxx-solr-1:8983/solr Click on Logging -> Level Select a root logger (click on the level value) and modify it to be WARN The change is applied at runtime, no restart required.

In the documentation of the particular projects you can find a number of details on how these components work on their own and on which services they rely. Since the projects are open source you can of course check out the source code for more information. Therefore, this article aims to summarize, rather than explain each process in detail.In this article I am first going through some basic component descriptions to get an idea which services are in use. Then I explain the “security flow” from a user perspective (authentication –> impersonation (optional) –> authorization –> audit) and provide a short example using Knox.When reading the article keep following figure in mind:Component Descriptions and ConceptsApache RangerComponents and what they doRanger Admin Service:Provides RESTful API and a UI to manage authorization policies and service access audits based on resources, users, groups and tags.Ranger User sync:Syncs users and groups from an LDAP source (OpenLDAP or AD)Sto

Symptoms:HST Bundle capture fails during the anonymization process with an "Illegal group reference" message exception found in the hst-agent logs:ERROR shell.py:95 - Execution of command returned 1. Exception in thread "main" java.lang.IllegalArgumentException: Illegal group reference         at java.util.regex.Matcher.appendReplacement(Matcher.java:857)         at java.util.regex.Matcher.replaceAll(Matcher.java:955)         at java.lang.String.replaceAll(String.java:2223)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.handleApplyGroupPatternKeyValue(BundleAnonymizer.java:690)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyGroupPattern(BundleAnonymizer.java:673)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyPropertyRule(BundleAnonymizer.java:612)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyGroupPattern(BundleAnonymizer.java:673)         at com.hortonworks.smartsense.anonymization

Step 1: DownloadFirstly download and unzip the package (I used: nifi-toolkit-1.0.0-bin.tar.gz) Inside there are a number of folders and files but I am mostly interested in bin/tls-toolkit.sh. This can be run in either standalone or server/client mode:Standalone is for a one-off generation of certificates and keysClient/Server allows you to run the tls toolkit as a server to sign Certificate Signing Requests from clientsStep 2: Generate Keys and CertificatesInitially I ran the help:./bin/tls-toolkit.sh standalone -hThen I created three bundles for each of my servers:./bin/tls-toolkit.sh standalone -n 'localhost(3)' -C 'CN=scarroll,OU=NIFI' -O -o ../security_outputWhich generated my truststore, keystore and nifi.properties file for my three hosts and my client certificates as well. Here is the output:$ tree security_output/security_output/├── CN=scarroll_OU=NIFI.p12├── CN=scarroll_OU=NIFI.password├── localhost│ ├── keystore.jks│ ├── nifi.properties│ └── truststore.jks├── localhost_2│

Introduction - Here is a small demo how to flex MiNiFi+NiFi on a Raspberry Pi to detect Motion and send alerts to your phone via SMS, also on your SMS reply it will trigger sound ALARM [Basically to shoo away an intruder]. You can do it from where ever you have cell phone reception. - Here you can view the screen recording session that Demonstrates how it works!Prerequisite Raspberry Pi 3, a PIR motion sensor and a speaker connected to it. Details on how to connect PIR Motion Sensor to Raspberry Pi can be found in url under references. Assuming you already have latest version of HDF/NiFi and Minifi downloaded on your Mac and Pi. Else Get Latest version of MiNiFi : # wget http://apache.claz.org/nifi/minifi/0.1.0/minifi-0.1.0-bin.tar.gz Get Latest version of MiNiFi ToolKit: # wget http://apache.claz.org/nifi/minifi/0.1.0/minifi-toolkit-0.1.0-bin.tar.gz Get Latest version of NiFi: # wget http://apache.claz.org/nifi/1.2.0/nifi-1.2.0-bin.tar.gz Untar the files and start NiFi on yo

Tutorial LinkSandboxVersion: HDP 2.3.2AmbariVersion2.1.2Hadoop stack version:Hadoop2.7.1.2.3.2.0-2950-------------------------------------------Issue 1: No available when executing command yum groupinstall “Development tools“ Resolution: This occurs when you copy and pasted the command. The double quotes used are wrong. Use " instead. There run the following command with the right double quotes:yum groupinstall "Development tools" The same occurs with pip install “ipython[notebook]“Instead run pip install "ipython[notebook]"---------------------------------------Issue 2: No ~/.ipython/profile_pyspark found after executing command ipython profile create pysparkResolution: IPython was updated to 4.0.0 which uses jupyterRun jupyter notebook --generate-configThen nano /root/.jupyter/jupyter_notebook_config.py-------------------------------------------Issue 3: - profile Error when executing "~/start_ipython_notebook.sh"Resolution:Use IPYTHON_OPTS="notebook" pysparkinstead of IPYTHON_OPT

Objectives This tutorial will walk you through the process of using Cloudbreak recipes to install Anaconda on an your HDP cluster during cluster provisioning. This process can be used to automate many tasks on the cluster both pre-install and post-install. Prerequisites You should already have a Cloudbreak v1.14.0 environment running. You can follow this article to create a Cloudbreak instance using Vagrant and Virtualbox: HCC Article You should already have credentials created in Cloudbreak for deploying on AWS (or Azure). Scope This tutorial was tested in the following environment: macOS Sierra (version 10.12.4) Cloudbreak 1.14.0 TP AWS EC2 Anaconda 2.7.13 Steps Create Recipe Before you can use a recipe during a cluster deployment, you have to create the recipe. In the Cloudbreak UI, look for the "mange recipes" section. It should look similar to this: If this is your first time creating a recipe, you will have 0 recipes instead of the 2 recipes show in my interface. Now click

IntroScalable cheap storage and parallel processing are at the foundation of Hadoop. These capabilities allow Hadoop to play a key role in complementing and optimizing traditional Enterprise Data Warehouse (EDW) workloads. In addition, recent technologies like Hive LLAP (in-memory, long-running execution threads) with ACID merge and AtScale proprietary software (virtual OLAP cubes with aggregation cacheing in HDFS) now for the first time allow fast BI directly against TB- and PB-sized data on Hadoop using well-known BI tools (e.g Tableau).This article describes reference architectures for three use cases of EDW optimization: Active archiving: archiving aged EDW data on Hadoop for cheap storage and for exploratory and BI analysis.EDW offload: offloading staging data and ETL transformations to Hadoop and exporting the final data structures back to existing EDW system (for cheap storage and faster ETL on Hadoop, as well as faster BI queries on existing EDW).BI on Hadoop: Business Intel

PROBLEM : Ambari Workflow Manager view with kerberos fails to clear Oozie Service Check and errors out with proxy error in wfmanager-view.log.ERROR :11 May 2017 22:34:49,839 INFO [ambari-client-thread-26] [WORKFLOW_MANAGER 1.0.0 MyWFM] OozieDelegate:149 - Proxy request for url: [GET] http://space2.example.com:11000/oozie/v1/admin/configuration11 May 2017 22:34:49,889 ERROR [ambari-client-thread-26] [WORKFLOW_MANAGER 1.0.0 MyWFM] OozieProxyImpersonator:456 - Error in GET proxyjava.lang.RuntimeException:java.lang.NullPointerException at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1455) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441) at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:2979) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:489)ROOT CAUSE : Missing below properties in custom oozie-site.xml.oozie.service.ProxyUserService.proxyuser.<AMBARI

With the release of Apache Nifi 1.0 data flows became a multi-tenancy environment with teams of data flow "developers" working simultaneously. In these situations team members may or may not have read and write permission to all components in the flow. One pattern that has emerged to aid in separating work spaces and grouping together components for these different teams is the use of process groups (PG). The "pain point" with this type of approach is that when the Nifi canvas is loaded (or reloaded) it always shows the root group. This is frustrating for team members who want to edit a flow but only have permissions to edit components in a PG that may be nested within several other PGs. In this scenario the team member will need to load the root group, pan or search to find the PG, pan or search to find the PG, pan or.....you get the idea. It is a manual effort to find the nested PG that contains the few components a user has permissions to edit. To ease this the latest Apache Nifi 1

Pre-requisites Power BI Desktop Edition https://powerbi.microsoft.com/en-us/desktop/HDP Installation either cluster or sandbox. Can be cloud or on-premHortonworks ODBC Driver for Apache Hive (v2.1.5). Scroll down to the Add-Ons section http://hortonworks.com/downloads/#sandboxData set. I used TPC-DS which can be installed from here https://github.com/hortonworks/hive-testbenchNote: This exercise assumes a non-Kerberoized clusterStep 1 – Setup the EnvironmentHow to setup the environment is up to you. You have lots of options. I ended up using Windows Server 2016 Evaluation Edition running on VirtualBox. For HDP I’m running a 4 node cluster in the cloud. Here are a few possible scenarios: Leverage existing infrastructure. You will need hardware for HDP as well as a Windows or Mac box to run Power BI desktopRun both instances in a virtual environment using VMware or VirtualBox and install the HDP 2.5 sandbox.Run everything in the cloud with either Azure IaaS, HDInsight, or AWS.Run Power

From the logs and HDFS usage outputs it confirms the growth is related to HBASE snapshots. We can confirm this by checking if all the snapshots having same time stamp. "list_snapshots" command from hbase shell will provide an output like below. hbase> list_snapshots SYSTEM.CATALOG-ru-20160512 SYSTEM.CATALOG (Thu May 12 01:47:24 +0000 2016) SYSTEM.FUNCTION-ru-20160512 SYSTEM.FUNCTION (Thu May 12 01:47:24 +0000 2016) SYSTEM.SEQUENCE-ru-20160512 SYSTEM.SEQUENCE (Thu May 12 01:47:24 +0000 2016) SYSTEM.STATS-ru-20160512 SYSTEM.STATS (Thu May 12 01:47:32 +0000 2016) US_1-ru-20160512 US_1 (Thu May 12 01:47:32 +0000 2016) ambarismoketest-ru-20160512 ambarismoketest (Thu May 12 01:47:32 +0000 2016) dev.hadoop-ru-20160512 dev.hadoop(Thu May 12 01:47:33 +0000 2016) prod.hadoop-ru-20160512 prod.hadoop (Thu May 12 01:47:35 +0000 2016) compact.daily-ru-20160512 compact.daily (Thu May 12 01:47:43 +0000 2016) compact.hourly-ru-20160512 compact.hourly (Thu May 12 01:47:43 +0000 2016) test-ru-20160

Symptoms:Users may see an incorrect host domain name that may also include unexpected markers e.g. hadoopnode1.kjtijasdh.com , hadoopnode1.??kjtijasdh??.comDiagnosis:All data collected by HST Agents is anonymized via a process that uses a set of configurations along with anonymization rules to mask potentially sensitive data. The configurations include definitions for anonymization markers that are used to identify the start and end of a piece of data that has been anonymized via the rules. Certain anonymized content, such as host domain names, can later be deanonymized at the time of sending SmartSense recomendations so that the host domain is recognizable to the user. Typically the configured markers are unicode characters such as “¶” and “‡” which can can be incorrectly identified if the locale on the host running the HST Agent is not setup correctly which results in ?? as a default replacement.Solution:Making sure that you are using the same user that runs the HST agent on the pr

This issue is caused due to the size of "~/.beeline/history" file in user's home directory. While we run big queries, it will flood the history file and slow down beeline on start up and shutdown. To resolve this issue, please move "~/.beeline/history" file and retry beeline. In the lower versions (below 2.1.0), we can have a cron job to delete "history" file periodically. This issue has been fixed in latest Hive versions by adding "--maxHistoryRows". Please check the below JIRA for more details. https://issues.apache.org/jira/browse/HIVE-15166

The standard solutionLet's say you want to collect log messages from an edge cluster with NIFI, and push it to a central NIFI cluster via the Site To Site (S2S) protocol. This is exactly what NIFI is designed for, and results in a simple flow setup like this: A processor that tails the log filewhich sends it's flowfiles to a remote process group which is configured with the FQDN URL of the central NIFI cluster on the central NIFI cluster an INPUT port is definedand from that input port the rest of the flow is doing it's thing with the incoming flow files, like filtering, transformations and eventually sinking it into kafka, HDFS or SOLR.The NIFI S2S protocol is used for the connection between the edge NIFI cluster and the central nifi cluster.which PUSHES the flowfiles from the edge cluster to the central NIFI cluster. And now with a firewall blocking incoming connections in betweenThis standard setup however assumes the central NIFI cluster has a public FQDN and isn't behind a firewal

Error Stack is below.Package Manager failed to install packages. Error: (4, 'Interrupted system call')Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 376, in install_packages retry_count=agent_stack_retry_count File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 58, in action_upgrade self.upgrade_package(package_name, self.resource.use_repos, self.resource.skip_repos) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 56, in upgrade_package

Prior to release 1.2, Apache NiFi supports the notion of pluggable "Schema Registry”, and ConvertXToY processors including but not limited to handling format conversions between CSV, JSON and Avro. Starting from 1.2, NiFi supports a new concept - an abstraction of “record”. Any FlowFile can contain one or more record objects of a given format and schema. NiFi then offers a series of controller services which provide a ‘RecordReader' interface and will have immediate support for Avro, JSON Tree, JSON Path, Grok, and CSV as provided source formats. in the same 1.2 release, NiFi will support a series of controller services which provide a “RecordWriter" interface and will have immediate support for Avro, JSON, Text, CSV.Each of the Reader and Writer objects will support interaction with another Controller Service which will be referencing a schema registry for retrieving schemas when necessary. An Avro schema registry and an HWX schema registry will be immediately available in Apache NiFi

Below is a short Scala program and SBT build script to generate a Spark application to submit to YARN that will run on the HDP Sandbox with Spark 1.6.We consume KAFKA messages in microbatches of 2 seconds. We set a number of parameters for performance of the SparkSQL write to HDFS including enabling Tungsten, using Snappy compression, Backpressure and the KryoSerializer.We setup the Spark Context and the Spark Streaming Context, then use the KafkaUtils to prepare a Stream of generic Avro data records from a byte array. From there we then convert to a Scala Case Class that models the Twitter tweet from our source. We convert to a DataFrame, register it as a Temp Table and then save to ORC File, AVRO, Parquet and JSON.To Build this Application:sbt clean assembly Create Kafka Topiccd /usr/hdp/current/kafka-brokerbin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic meetupTo Prepare Hadoop Environment# Create HDFS Directories (also avro,