Resources

It is often the case that we need to install Hortonworks in environments with strict requirements. One such requirement may be that all http traffic must go through a dedicated proxy server.When installing Hortoworks HDP using Ambari, you can find instructions for configuring Ambari to use the proxy on the docs.hortonworks.com website. For example, here is the page for configuring Ambari 2.2http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_ambari_reference_guide/content/ch_setting_up_an_internet_proxy_server_for_ambari.htmlNotice that the instructions mention that you must also configure yum to use the proxy. It’s important to note that the above instructions for yum will configure all repositories to use the proxy, and you may not want this behavior.So while it is great to set the proxy at yum’s global level, you should review any existing repository configurations to determine if they should not use the proxy. If any repositories should not use the proxy, then you can update

ENVIRONMENTThis problem has been replicated and fixed on HDP 2.5.xPROBLEMI have followed up the https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_zeppelin-component-guide/content/zepp-with-spark.html to add additional jar for livy - copy jar file into /usr/hdp/<version>/livy/repl-jars folder. Restarted livy service afterwards. When running: %livy import de.xxx.statistik.program.model.LineModel I am getting: <console>:25: error: not found: value de import de.xxx.statistik.program.model.LineModel SOLUTIONAdded the following into Ambari -> MapReduce2 -> mapreduce.application.classpath - :$PWD/*Before Change$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/ha

If you haven't already see the 1st tutorial I made which guides you through the setup of rapidminer to read from hive. We will pick up where it leaves off. Read-from-hive-using-rapidmineradd a "set role" operator next to the your "retrieve from hive operator" that is located in the "Radoop Nest".This allows you to select the column you wish to use in the model.In this case I set the name field to the category column in my dataset. You can obtain this dataset here: dataThis is just for illustrative purposes so if you have data that has labels already feel free to use in place of this. Now add a "split validation" operator and connect the ports.Then double click the validation operator. Add a "decision tree" operator on the left pane and add an "apply model" and "performance" operator and connect them all. For performance select accuracy or whatever you wish to check. If you are using the sample data provided in this tutorial you will see some errors. Click on the error icons on each ope

First you need to have Rapidminer downloaded and installed on your machine. https://my.rapidminer.com/nexus/account/index.html#downloadsOnce installed open Rapidminer and look at the list of operators. There is a link at the bottom left to "Get More Operators"Click the link then search for "Radoop"Select both packages and click install. After Rapidminer restarts you will see in the extensions folder the new operators we downloaded. Now we need to configure the connection. In the toolbar select "connections" then "Manage Radoop Connections" Select "+ New Connection".If you have your Hadoop config files available you can use those to set the properties otherwise select "manual" Select the hadoop version you have. In my case "Hortonworks 2.x" and supply the master url... If you have multiple masters select the check box and provide the details. Click "OK"Now click ">> Quick Test" If successful you are all set to read from Hive. Drag an "Radoop Nest" operator onto the canvas. Select

Ambari user sync will fail to map or import users when trying to pull from groups with 1,500+ members.What we see when we use ldapsearch to query a large group is: <snip> member;range=0-1499: CN=Elgine Metzger,OU=users,OU=test,DC=j4ck3l,DC=net member;range=0-1499: CN=Friedolf Welter,OU=users,OU=test,DC=j4ck3l,DC=net </snip> This is seems to come from the LDAP Policy value: MaxValRange "MaxValueRange controls the number of values that are returned on a single attribute on a single object. Default"1500 Hard Limit: 5000" -- http://ldapwiki.com/wiki/MaxValRange To fix this: Go to the domain controller that we’re connecting to for the sync Find the file ntdsutil.exe (most likely under c:\windows\system32 or c:\winnt\system32 Run the ntdsutil.exe Type “ldap policies” and enter Type "connections" and enter Type "Connect to server [YourDCName]" and enter Type "q" and enter Type "Show Values" to see the current settings Type “Set MaxValRange to 2500” and enter Type “Commit Change

We tried 3 scenarios and all failed.1) Creating Hive Table on HBase. (proper hive/hbase polices for user) hive> CREATE TABLE hbase_table_1(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "xyz", "hbase.mapred.output.outputtable" = "xyz"); FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'abc@ABC.COM' (action=create) at org.apache.ranger.authorization.hbase.AuthorizationSession.publishResults(AuthorizationSession.java:254) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.authorizeAccess(RangerAuthorizationCoprocessor.java:595) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.requirePermission(RangerAuth

Problem:Following upgrade to 2.5 Oozie Sqoop job which previously worked now fails withoozi-W@sqoop-aa6c] Launcher exception: org/json/JSONObjectjava.lang.NoClassDefFoundError: org/json/JSONObject at org.apache.sqoop.util.SqoopJsonUtil.getJsonStringforMap(SqoopJsonUtil.java:43) at org.apache.sqoop.SqoopOptions.writeProperties(SqoopOptions.java:759) at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.createInternal(HsqldbJobStorage.java:399) at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.create(HsqldbJobStorage.java:379)Solution:Ensure that 'java-json.jar' file exists in the following locations: /usr/hdp/current/sqoop-client/lib /usr/hdp/<version>/sqoop/lib Also copy this jar into HDFS and reference it in your oozie workflow:e.g. hdfs dfs -put /usr/hdp/current/sqoop-client/lib/java-json.jar /user/oozie/oozie_scripts/lib/and change workflow.xml: <archive>/user/oozie/oozie_scripts/lib/java-json.jar#java-jso

Below are the steps for Oozie database migration from Derby to Postgresql..Step 1 - Have Postgresql server installed and ready to be configured..Step 2 - Stop Oozie service from Ambari UI..Step 3 - Install Postgresql JDBC connector.yum install postgresql-jdbc.Step 4 - On Ambari Server, run below commandambari-server setup --jdbc-db=postgres --jdbc-driver=/usr/share/java/postgresql-jdbc.jarNote - Please pass appropriate driver if /usr/share/java/postgresql-jdbc.jar does not exists. ..Step 5 - Login to Postgesql DB as postgres user and create a blank 'oozie' database and grant required permissions to the 'oozie' user.[root@ambaview ~]# su - postgres-bash-4.1$ psqlpsql (8.4.20)Type "help" for help.— postgres=# CREATE DATABASE oozie;CREATE DATABASEpostgres=#— CREATE USER oozie WITH PASSWORD 'oozie’;postgres=# CREATE USER oozie WITH PASSWORD 'oozie';CREATE ROLEpostgres=#— GRANT ALL PRIVILEGES ON DATABASE oozie TO oozie;postgres=# GRANT ALL PRIVILEGES ON DATABASE oozie TO oozie;GRANTpostgres

Affected versions:1.2.x, 1.3.0Symptoms:The SmartSense view if not displayed even if the HST-Server is up and running. Also , refer the steps that were executed during the upgrade process . It is possible this error has occurred during “hst upgrade-ambari-service” step.Un-installing old view ... Installing new view ... ('request method', 'GET') ('request data', None) ('request url', 'http://ambari-server.abcxyz.net:8083/api/v1/views/HORTONWORKS_SMARTSENSE') ('request type', 'http') ('request host', 'ambari-server.abcxyz.net:8083') ('request selector', '/api/v1/views/HORTONWORKS_SMARTSENSE') ('request header', [('Host', 'ambari-server.abcxyz.net:8083'), ('Authorization', 'Basic YWRtaW46Y24zild6MmT9ODVw'), ('User-agent', 'Python-urllib/2.7')]) ('response code', None) ('response content', None) Traceback (most recent call last): File "/usr/sbin/hst-server.py", line 1384, in <module> main(sys.argv) File "/usr/sbin/hst-server.py", line 1361, in main upgrade_ambari_plugin(options) File

Affected versions:1.2.x, 1.3.0Symptoms: 2-way ssl setup fails with an error as given below. This error message is available in hst-server.log or hst-gateway.logINFO 2016-08-25 06:16:00,690 security.py:182 - Agent key not exists, generating requestINFO 2016-08-25 06:16:00,691 security.py:242 - openssl req -new -newkey rsa:1024 -nodes -keyout "/var/lib/smartsense/hst-agent/keys/hdp1-hrdd-vm1.akrocl34v0sjr9flwro4laslcd.gx.internal.cloudapp.net.key" -subj /OU=hdp1-hrdd-vm1.akrocl34v0sjr9flwro4laslcd.gx.internal.cloudapp.net/ -out "/var/lib/smartsense/hst-agent/keys/hdp1-hrdd-vm1.akrocl34v0sjr9flwro4laslcd.gx.internal.cloudapp.net.csr" was finished with exit code: 1 - an error occurred parsing the command options.INFO 2016-08-25 06:16:00,711 security.py:188 - Agent certificate not exists, sending sign request Reason: openssl does not support SubjectName longer than 64 chars. SubjAltName should be used for longer host names.#openssl req -new -newkey rsa:1024 -nodes -keyout "./hdp1-hrdd-vm1.a

Affected versions:1.3.xSymptoms: When tried to install SmartSense then it failed on Activity Analyzer installation which is the first step of SmartSense installation.File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in _getattr_raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")resource_management.core.exceptions.Fail: Configuration parameter 'hadoop-env' was not found in configurations dictionary!Similar error occurs when trying to reinstall HST SERVER then it failed with similar message.File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in _getattr_raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")resource_management.core.exceptions.Fail: Configuration parameter 'namenode_host' was not found in configurations dictionary!Reason: SmartSense components assumed that hdfs will

Affected versions:1.3.xSymptoms: A Support bundle capture fails if case number starts with 0, like below :hst-agent.py: error: option -c: invalid integer value: '00100870'Reason: The root cause of this issue is that the case number is handled as octal based, due to the leading zero. After that is converted to decimal base.Solution:Fixed in 1.4.0 and remove the preceding zeros for 1.3.x and capture again

Affected versions: 1.2.x, 1.3.x Symptoms: hst-agent thinks it has uploaded its node file; however, hst-server doesn't find the file when it tries to use it. Below error reported in the hst-server.log file :15 Feb 2017 15:09:35,033 INFO Thread-1 BundleMonitor:286 - Uploaded archive /hadoop/1/smartsense/hst-server/tmp/a-00000000-c-00000000_devhdpvm01_0_2017-02-15_15-08-02/hdwduv0073.odevkrj.jgcz.group-a-00000000-c-00000000_devhdpvm01_0_2017-02-15_15-08-02.tgz doesn't exists. Reason: hst-server and hst agents have a common directory on the same shared mount where node bundles from each agent are uploaded and clean up before consuming by the hst-server. Solution: Fixed in 1.4.0 and HOTFIX-651 for 1.3.x Otherwise, create separate directories for each agent / server in the shared mount or replace the share mount with local filesystem.

When one stands up SolrCloud/HDPSearch, every core/replica will be writing logs to /opt/lucidworks-hdpsearch/solr/server/logs. Files like 'solr-8983-console.log' will grow very quickly for a lightly used index, at the rate of several GBs per day. The reason is it outputs all INFO and DEBUG messages to this console log. Which is great for development, but a pain for production and operations. The challenge is there are quite a few log configuration files spread around the Solr directories, and those will not necessarily yield the desired result. It gets even more complicated for a SolrCloud, where one is dealing with a cluster of nodes. There is, however, a very straightforward way to quickly address this issue: Navigate to the admin console, like http://xxx-solr-1:8983/solr Click on Logging -> Level Select a root logger (click on the level value) and modify it to be WARN The change is applied at runtime, no restart required.

In the documentation of the particular projects you can find a number of details on how these components work on their own and on which services they rely. Since the projects are open source you can of course check out the source code for more information. Therefore, this article aims to summarize, rather than explain each process in detail.In this article I am first going through some basic component descriptions to get an idea which services are in use. Then I explain the “security flow” from a user perspective (authentication –> impersonation (optional) –> authorization –> audit) and provide a short example using Knox.When reading the article keep following figure in mind:Component Descriptions and ConceptsApache RangerComponents and what they doRanger Admin Service:Provides RESTful API and a UI to manage authorization policies and service access audits based on resources, users, groups and tags.Ranger User sync:Syncs users and groups from an LDAP source (OpenLDAP or AD)Sto

Symptoms:HST Bundle capture fails during the anonymization process with an "Illegal group reference" message exception found in the hst-agent logs:ERROR shell.py:95 - Execution of command returned 1. Exception in thread "main" java.lang.IllegalArgumentException: Illegal group reference         at java.util.regex.Matcher.appendReplacement(Matcher.java:857)         at java.util.regex.Matcher.replaceAll(Matcher.java:955)         at java.lang.String.replaceAll(String.java:2223)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.handleApplyGroupPatternKeyValue(BundleAnonymizer.java:690)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyGroupPattern(BundleAnonymizer.java:673)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyPropertyRule(BundleAnonymizer.java:612)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyGroupPattern(BundleAnonymizer.java:673)         at com.hortonworks.smartsense.anonymization

Step 1: DownloadFirstly download and unzip the package (I used: nifi-toolkit-1.0.0-bin.tar.gz) Inside there are a number of folders and files but I am mostly interested in bin/tls-toolkit.sh. This can be run in either standalone or server/client mode:Standalone is for a one-off generation of certificates and keysClient/Server allows you to run the tls toolkit as a server to sign Certificate Signing Requests from clientsStep 2: Generate Keys and CertificatesInitially I ran the help:./bin/tls-toolkit.sh standalone -hThen I created three bundles for each of my servers:./bin/tls-toolkit.sh standalone -n 'localhost(3)' -C 'CN=scarroll,OU=NIFI' -O -o ../security_outputWhich generated my truststore, keystore and nifi.properties file for my three hosts and my client certificates as well. Here is the output:$ tree security_output/security_output/├── CN=scarroll_OU=NIFI.p12├── CN=scarroll_OU=NIFI.password├── localhost│ ├── keystore.jks│ ├── nifi.properties│ └── truststore.jks├── localhost_2│

Introduction - Here is a small demo how to flex MiNiFi+NiFi on a Raspberry Pi to detect Motion and send alerts to your phone via SMS, also on your SMS reply it will trigger sound ALARM [Basically to shoo away an intruder]. You can do it from where ever you have cell phone reception. - Here you can view the screen recording session that Demonstrates how it works!Prerequisite Raspberry Pi 3, a PIR motion sensor and a speaker connected to it. Details on how to connect PIR Motion Sensor to Raspberry Pi can be found in url under references. Assuming you already have latest version of HDF/NiFi and Minifi downloaded on your Mac and Pi. Else Get Latest version of MiNiFi : # wget http://apache.claz.org/nifi/minifi/0.1.0/minifi-0.1.0-bin.tar.gz Get Latest version of MiNiFi ToolKit: # wget http://apache.claz.org/nifi/minifi/0.1.0/minifi-toolkit-0.1.0-bin.tar.gz Get Latest version of NiFi: # wget http://apache.claz.org/nifi/1.2.0/nifi-1.2.0-bin.tar.gz Untar the files and start NiFi on yo

Tutorial LinkSandboxVersion: HDP 2.3.2AmbariVersion2.1.2Hadoop stack version:Hadoop2.7.1.2.3.2.0-2950-------------------------------------------Issue 1: No available when executing command yum groupinstall “Development tools“ Resolution: This occurs when you copy and pasted the command. The double quotes used are wrong. Use " instead. There run the following command with the right double quotes:yum groupinstall "Development tools" The same occurs with pip install “ipython[notebook]“Instead run pip install "ipython[notebook]"---------------------------------------Issue 2: No ~/.ipython/profile_pyspark found after executing command ipython profile create pysparkResolution: IPython was updated to 4.0.0 which uses jupyterRun jupyter notebook --generate-configThen nano /root/.jupyter/jupyter_notebook_config.py-------------------------------------------Issue 3: - profile Error when executing "~/start_ipython_notebook.sh"Resolution:Use IPYTHON_OPTS="notebook" pysparkinstead of IPYTHON_OPT

Objectives This tutorial will walk you through the process of using Cloudbreak recipes to install Anaconda on an your HDP cluster during cluster provisioning. This process can be used to automate many tasks on the cluster both pre-install and post-install. Prerequisites You should already have a Cloudbreak v1.14.0 environment running. You can follow this article to create a Cloudbreak instance using Vagrant and Virtualbox: HCC Article You should already have credentials created in Cloudbreak for deploying on AWS (or Azure). Scope This tutorial was tested in the following environment: macOS Sierra (version 10.12.4) Cloudbreak 1.14.0 TP AWS EC2 Anaconda 2.7.13 Steps Create Recipe Before you can use a recipe during a cluster deployment, you have to create the recipe. In the Cloudbreak UI, look for the "mange recipes" section. It should look similar to this: If this is your first time creating a recipe, you will have 0 recipes instead of the 2 recipes show in my interface. Now click

IntroScalable cheap storage and parallel processing are at the foundation of Hadoop. These capabilities allow Hadoop to play a key role in complementing and optimizing traditional Enterprise Data Warehouse (EDW) workloads. In addition, recent technologies like Hive LLAP (in-memory, long-running execution threads) with ACID merge and AtScale proprietary software (virtual OLAP cubes with aggregation cacheing in HDFS) now for the first time allow fast BI directly against TB- and PB-sized data on Hadoop using well-known BI tools (e.g Tableau).This article describes reference architectures for three use cases of EDW optimization: Active archiving: archiving aged EDW data on Hadoop for cheap storage and for exploratory and BI analysis.EDW offload: offloading staging data and ETL transformations to Hadoop and exporting the final data structures back to existing EDW system (for cheap storage and faster ETL on Hadoop, as well as faster BI queries on existing EDW).BI on Hadoop: Business Intel

PROBLEM : Ambari Workflow Manager view with kerberos fails to clear Oozie Service Check and errors out with proxy error in wfmanager-view.log.ERROR :11 May 2017 22:34:49,839 INFO [ambari-client-thread-26] [WORKFLOW_MANAGER 1.0.0 MyWFM] OozieDelegate:149 - Proxy request for url: [GET] http://space2.example.com:11000/oozie/v1/admin/configuration11 May 2017 22:34:49,889 ERROR [ambari-client-thread-26] [WORKFLOW_MANAGER 1.0.0 MyWFM] OozieProxyImpersonator:456 - Error in GET proxyjava.lang.RuntimeException:java.lang.NullPointerException at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1455) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441) at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:2979) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:489)ROOT CAUSE : Missing below properties in custom oozie-site.xml.oozie.service.ProxyUserService.proxyuser.<AMBARI

With the release of Apache Nifi 1.0 data flows became a multi-tenancy environment with teams of data flow "developers" working simultaneously. In these situations team members may or may not have read and write permission to all components in the flow. One pattern that has emerged to aid in separating work spaces and grouping together components for these different teams is the use of process groups (PG). The "pain point" with this type of approach is that when the Nifi canvas is loaded (or reloaded) it always shows the root group. This is frustrating for team members who want to edit a flow but only have permissions to edit components in a PG that may be nested within several other PGs. In this scenario the team member will need to load the root group, pan or search to find the PG, pan or search to find the PG, pan or.....you get the idea. It is a manual effort to find the nested PG that contains the few components a user has permissions to edit. To ease this the latest Apache Nifi 1

Pre-requisites Power BI Desktop Edition https://powerbi.microsoft.com/en-us/desktop/HDP Installation either cluster or sandbox. Can be cloud or on-premHortonworks ODBC Driver for Apache Hive (v2.1.5). Scroll down to the Add-Ons section http://hortonworks.com/downloads/#sandboxData set. I used TPC-DS which can be installed from here https://github.com/hortonworks/hive-testbenchNote: This exercise assumes a non-Kerberoized clusterStep 1 – Setup the EnvironmentHow to setup the environment is up to you. You have lots of options. I ended up using Windows Server 2016 Evaluation Edition running on VirtualBox. For HDP I’m running a 4 node cluster in the cloud. Here are a few possible scenarios: Leverage existing infrastructure. You will need hardware for HDP as well as a Windows or Mac box to run Power BI desktopRun both instances in a virtual environment using VMware or VirtualBox and install the HDP 2.5 sandbox.Run everything in the cloud with either Azure IaaS, HDInsight, or AWS.Run Power

From the logs and HDFS usage outputs it confirms the growth is related to HBASE snapshots. We can confirm this by checking if all the snapshots having same time stamp. "list_snapshots" command from hbase shell will provide an output like below. hbase> list_snapshots SYSTEM.CATALOG-ru-20160512 SYSTEM.CATALOG (Thu May 12 01:47:24 +0000 2016) SYSTEM.FUNCTION-ru-20160512 SYSTEM.FUNCTION (Thu May 12 01:47:24 +0000 2016) SYSTEM.SEQUENCE-ru-20160512 SYSTEM.SEQUENCE (Thu May 12 01:47:24 +0000 2016) SYSTEM.STATS-ru-20160512 SYSTEM.STATS (Thu May 12 01:47:32 +0000 2016) US_1-ru-20160512 US_1 (Thu May 12 01:47:32 +0000 2016) ambarismoketest-ru-20160512 ambarismoketest (Thu May 12 01:47:32 +0000 2016) dev.hadoop-ru-20160512 dev.hadoop(Thu May 12 01:47:33 +0000 2016) prod.hadoop-ru-20160512 prod.hadoop (Thu May 12 01:47:35 +0000 2016) compact.daily-ru-20160512 compact.daily (Thu May 12 01:47:43 +0000 2016) compact.hourly-ru-20160512 compact.hourly (Thu May 12 01:47:43 +0000 2016) test-ru-20160