Resources

IoTWorking with IoT data is a many layered process, not unlike a parfait. Scratch that, an Onion. In fact, an Onion Omega2, which is a great device that I just got yesterday does IoT really easily. This is so much easier to setup that RPI or other platforms. It also has a ton of pluggable modules that stack on top of this small chip. It's pretty small powered, but it's under $10.The device is extremely well documented at their site.Needed to run real tools, I added a USB stick and used that for storage and for extra SWAP space.opkg updateopkg install kmod-usb-storage-extras e2fsprogs kmod-fs-ext4umount /tmp/mounts/USB-A1mkfs.ext4 /dev/sda1mkdir /mnt/sda1mount /dev/sda1 /mnt/sda1mount /dev/sda1 /mnt/ ; tar -C /overlay -cvf - . | tar -C /mnt/ -xf - ; umount /mnt/opkg updateopkg install block-mountopkg updateopkg install swap-utils block-mountdd if=/dev/zero of=/tmp/mounts/USB-A1/swap.page bs=1M count=256mkswap /tmp/mounts/USB-A1/swap.pageswapon /tmp/mounts/USB-A1/swap.pagefreeb

SYMPTOM: When you create an Oozie workflow which contains a SSH function, it can fail with an error of "Not able to perform operation [ssh -o PasswordAuthentication=no -o KbdInteractiveDevices=no -o StrictHostKeyChecking=no -o ConnectTimeout=20 root@localhost mkdir -p oozie-oozi/0000009-131023115150605-oozie-oozi-W/Ssh--ssh/ ] | ErrorStream: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password)."   ROOT CAUSE: Passwordless SSH must be setup correctly from oozie@server1 to $user@server2For example:  RESOLUTION: If an Oozie workflow contains an ssh command from server 1 to server 2 as root, then the passwordless SSH must be setup as the following: oozie@server1 > root@server2   This article created by Hortonworks Support (Article: 000001722) on 2015-04-01 11:05 OS: Linux Type: Configuration,Executing_Jobs

Depends on whether we are talking about under HDFS or at the OS level, but the answer to both is Yes...For accounts under HDFS, the home directory is determined by the value of dfs.user.home.dir.prefix. see: https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xmlFor OS users you can pre-create your service users before install, then set ignore_groupsusers_create=true in cluster-env.xml, and then launch the installer. The service users and groups are here: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_command-line-installation/content/create-system-users-and-groups.html

This tutorial will demonstrate how to execute PySpark jobs on an HDP cluster and pass in parameter values using the Livy REST interface. I will also demonstrate how to interact with Livy via Apache Zeppelin and use forms in Zeppelin to pass in parameter values. Livy is an open source REST interface for interacting with Spark. Livy requires at least Spark 1.6. By default, Livy is built against Apache Spark 1.6.2 with HDP 2.5.3 If on HDP 2.4.2, then Livy 0.2.0 might be part of HDP 2.4.2 repos. If not, then it can be cloned from: https://github.com/cloudera/livy Livy is built against Apache Spark 1.6.2 with HDP 2.5.3. The following was tested with HDP 2.5.3 and Spark 1.6.2. Business Use Case: A PySpark script that resides on the Livy server aggregates order statistics from a Hive table for a date range passed in as parameters to the Python script. The Start and End dates are passed in as parameter values to the Python script. The job runs on the cluster via YARN and produces the followi

it is not completely obvious but you can certainly run Python scripts within Oozie workflows using the Shell action.Here's a sample job.properties file, nothing special about it.nameNode=hdfs://sandbox.hortonworks.com:8020jobTracker=sandbox.hortonworks.com:8050queueName=defaultexamplesRoot=oozieoozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/pythonHere's a sample workflow that will look for a script called script.py inside scripts folder<workflow-app xmlns="uri:oozie:workflow:0.4" name="python-wf"> <start to="python-node"/> <action name="python-node"> <shell xmlns="uri:oozie:shell-action:0.2"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </p

When upgrading to HDP 2.5 or later with Falcon, you might encounter the following error when starting the ActiveMQ server:
ERROR - [main:] ~ Failed to start ActiveMQ JMS Message Broker. Reason: java.lang.NegativeArraySizeException (BrokerService:528)You might also encounter this error if downgrading Falcon from HDP 2.5 to an earlier release. If you encounter this error, delete the ActiveMQ history and then restart Falcon. If you want to retain the history, be sure to back up the ActiveMQ history prior to deleting it.
 cd <ACTIVEMQ_DATA_DIR>rm -rf ./localhostcd /usr/hdp/current/falcon-server
su -l <FALCON_USER>


./bin/falcon-stop
./bin/falcon-start

Original ArticleCan I authorize access to Kafka over a non-secure channel via Ranger?Yes. you can control access by ip-address.Can I authorize access to Kafka over non-secure channel by user/user-groups?No, one can’t use user/group based access to authorize Kafka access over a non-secure channel. This is because it isn't possible to assert client’s identity over the non-secure channel.Why do we have to specify public user group on all policies items created for authorizing Kafka access over non-secure channel? Kafka can’t assert the identity of client user over a non-secure channel. Thus, Kafka treats all users for such access as an anonymous user (a special user literally named ANONYMOUS).Ranger's public user group is a means to model all users which, of course, includes this anonymous user (ANONYMOUS).What are the specific things to watch out for when setting up authorization for accessing Kafka over non-secure channel? Make sure that all broker-ips have Kafka admin access to all t

ERRORS:from HDFS log:016-10-30 17:44:04,226 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(903)) - Request to collection ranger_audits failed due to (401) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://hwx.com:8886/solr/ranger_audits_shard1_replica1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/> <title>Error 401 Authentication required</title> </head> <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /solr/ranger_audits_shard1_replica1/update. Reason: <pre> Authentication required</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/> From Hive log:016-10-30 17:50:46,189 WARN [org.apache.ranger.audit.queue.AuditBatchQueue1]: provider.BaseAuditHandler (BaseAuditHandler.java:logFailedEven

Hi, I am a software engineer in my last semester, and my graduation project is using impala.I have developed a handy tool for communicating with impala server through your java application, I know it is a simple one, but I want people to test it, try it, give me advices about it to enhance it, as it may become more handy for people building java apps on top of impala server.  Thanks in advance ...

PROBLEM:Hive queries were getting stuck/hang on both mr and tez engines while selecting from a table containing few csv files. While the query was working fine for few csv files, for others it just hangs. Nothing in the logs also. I was using Hive 1.2.1 on HDP 2.5.3.0 After some investigation, I found out that those files have some empty values '' in the fields where an rpad function was getting used. You can easily reproduce the issue by firing:select rpad('',1,''); You will see that the query just hangs.The reason is it goes to an infinite loop. More details here: HIVE-15792 RESOLUTION: nvl will not work in this case. That is select nvl('','D'); --will return '' I resolved using a query like this:SELECT rpad(CASE WHEN LENGTH(nvl(COLUMN_NAME,null)) > 0 THEN COLUMN_NAME ELSE null END, 1, ''); In this case the query will return null for both null and empty string values occurring in COLUMN_NAME. Hope this helps. Thanks, Rajdeep

Original postUse case: Access control on table customer, exclude column SSN.User Hive has access to see only name column. Access on column SSN is restricted.Column level security can be controlled in couple of clicks by Ranger UI.

Need to upload and share some of your application logs with a the support team or a costumer and tried of doing bunch of ssh, scp ...? https://transfer.sh/ is your tool !for instance assume you want to upload hello.txt.curl --upload-file ./hello.txt https://transfer.sh/hello.txtYou will get a unique shareable link once the upload is done.https://transfer.sh/66nb8/hello.txtMore examples can be found here

I found it tricky to make Kafka to work with SSL in a kerberized cluster. In this article I share ambari settings I used and console (producer/consumer) sample commands:1- Install Ambari and deploy a cluster with Kafka2- Kerberize cluster using Ambari (it can be AD Wizard, MIT Kerberos or Manual Keytabs)3- Create keystore and truststore with proper certificates, more details here: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_security/content/ch_wire-ssl-certs.html4- Change Kafka Settings on Ambari as below:Kafka Brokers - Listerners: listeners=SASL_PLAINTEXT://localhost:6667, SASL_SSL://localhost:6668Custom kafka-broker - security.inter.broker.protocol=SASL_PLAINTEXTCustom kafka-broker - ssl.client.auth=noneCustom kafka-broker - ssl.key.password=YOUR-PASSWORDCustom kafka-broker - ssl.keystore.location=/path-to-your-keystoreCustom kafka-broker - ssl.keystore.password=YOUR-PASSWORDCustom kafka-broker - ssl.truststore.location=/path-to-your-truststoreCustom kafka-broker - ss

Hive view is a web GUI that lets you easily interact with HIVE, which comes pre-deployed in the Ambari managed cluster. Ambari 2.5.0 comes with the new version of Hive view (2.0.0). The hive view 2.0 supports a lot of new features, one of them is Table Manager. Before Ambari 2.5.0, managing and creating table had to done through query tab, by writing queries. Table manager in Hive 2.0.0 provides a GUI wizard to create a table definition, or create a table, alter a table and load data by uploading the file of format CSV, JSON or XML.This article is an introduction to Table manager in Hive View 2.0Creating a new Hive View 2.0 Instance Ambari comes with auto created instances of Hive view. If you wish to create a new instance of hive view, follow the steps:Click Manage Ambari to open the Ambari Administration user interface.Click Views, expand the Hive View, and click Create Instance.On the Create Instance page, select the Version as 2.0.0. Table Manager Wizard To create a table using Ta

GOALS Configure Ranger + Ranger KMS Create an encryption Zone OS used is CentOS/Redhat 6.6 At the end, should be able to create an encryption zone and validate using Hive NOTE: This article is in a walkthrough mode wherein snapshots were taken from relevant screens to create a step-by-step guide on installing Ranger and Ranger KMS along with creation and validation of encrypted zones in Hadoop. https://www.scribd.com/presentation/341887712/HDFS-Encryption-Zone-Hive-Orig

IntroductionGetting Apache Metron (Incubating) installed on an Ubuntu cluster has not yet been something that has gotten a lot of attention. Building on the work of Michael Miklavcic in setting up Elasticsearch and Kibana on Ubuntu, we can proceed to tackle how to install Metron on a working HDP Ubuntu cluster with Elasticsearch and Kibana. For maximum transparency, we will do this manually.This article assumes the following to be installed and running: HDP 2.5.0+ with HDFSHBaseStormZookeeperKafkaElasticsearch 2.4Kibana 4.5.3We also assume that the access node that we are on has Oracle Java 8 on the classpath.Install MetronPreliminariesThe following steps should be done on an access node. This node should have the following installed:The Hadoop clientThe Storm clientI will assume that the user executing the following sectionHas suitable permissions to write to /apps in HDFSHas suitable permissions to start Storm topologies on the clusterHas sudo accessIn order to build Metron, we will

OverviewRaspberry Pi is an interesting little machine. Even more so with the release of the Raspberry Pi 3. While it may not technically be the most powerful device in this form-factor, it definitely wins on overall community, breadth of support and stability. Naturally, there is sizable interest in running something like NiFi on this little edge device. And that's exactly what we have been doing for the past 8 months (as of May 2016). Today I'd like to share a set of best practices for running NiFi on the Pi that came out of this exercise.In general, recommendations fall into 2 categories:Raspberry Pi configuration – OS, system-level settings for running an always-on NiFi instance.NiFi configuration – tweaks and changes within NiFi itselfLet’s start with some prep work.Raspberry Pi ConfigurationI’m assuming the Raspberry Pi is fully under one’s control, including physical access to SD card and USB slots, together with a root account.SD CardIf you bought a starter kit, it probably came

IntroductionWhat we’re doing: Committing to <2 hours setup time Deploying a matched pair of 3 node HDF clusters into two different zones on AWS EC2 Configuring some basic performance optimisations in EC2/NiFi Setting up site-to-site to work with EC2's public/private FQDNsWhat you will need:AWS account with access / credits to deploy 3x EC2 machines in two different regionsApprox 80Gb SSD disk per machine, for 480GB total3x Elastic IPs per region(Preferable) iTerm or similar ssh client with broadcast capabilityYou should use two different AWS regions for your clusters, I’ll refer to them as regionA and regionBCaveats:We’re going to use a script to setup OS pre-requisites and deploy HDF on the server nodes to save time, as this article is specific to AWS EC2 setup and not generic HDF deploymentWe’re not setting up HDF Security, though we will restrict access to our nodes to specific IPsThis is not best practice for all use cases, particularly on security; you are advised to take what

Environment- HDP 2.5.3- Kerberos disabledProblemI have a problem to use hiveContext with zeppelin. For example this code does not works:%pysparkfrom pyspark.sql import HiveContext sqlContext = HiveContext(sc)sample07 = sqlContext.table("default.sample_07")sample07.show()Here is the error displayed:You must build Spark with Hive. Export 'SPARK_HIVE=true' and run build/sbt assembly Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext. : java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Permission denied at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522) at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:204) at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238) at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:225) at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scal

ProblemWhen trying to import larger notebooks into Zeppelin, the import does nothing. Checking the /var/log/zeppelin/ logs, it is apparent that this is a jetty websocket text size issue.WARN [2017-03-13 14:22:30,427] ({qtp110945054-15} Parser.java[notifyWebSocketException]:235) - org.eclipse.jetty.websocket.api.MessageTooLargeException: Text message size [2408484] exceeds maximum size [1024000] at org.eclipse.jetty.websocket.api.WebSocketPolicy.assertValidTextMessageSize(WebSocketPolicy.java:140) at org.eclipse.jetty.websocket.common.Parser.assertSanePayloadLength(Parser.java:127) at org.eclipse.jetty.websocket.common.Parser.parseFrame(Parser.java:482) at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:254) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480) at org.eclipse.jetty.io.AbstractConn

ProblemHere is my user I run %livy.sparkr with%sh whoami admin Permissions set to directory and file:%shls -lrt /home/admintotal 12-rwxr-xr-x 1 admin admin 3950 Mar 9 13:12 common.rdrwxrwxr-x 2 admin admin 4096 Mar 10 15:25 analysisdrwxrwxr-x 3 admin admin 4096 Mar 13 12:49 motorAnalysis%shls -lrt /home/admin/motorAnalysistotal 8-rwxrwxr-x 1 admin admin 3950 Mar 10 16:49 common.rdrwxrwxr-x 5 admin admin 4096 Mar 10 16:49 dataRunning %livy.sparkr I am getting%livy.sparkr ...source("/home/admin/motorAnalysis/common.r") null device 1 Error in file(filename, "r", encoding = encoding) : cannot open the connection Calls: source -> file In addition: Warning message: In file(filename, "r", encoding = encoding) : cannot open file '/home/admin/motorAnalysis/common.r': No such file or directory SolutionRestart livy interpreter from Zeppelin UI fixes the problem

Q: Is there any way to change the user my hive jobs run as under LLAP? They always seem to run as 'hive' user.A: Hive LLAP does not currently support hive.server2.enable.doAs=true. All sessions will run under the hive account.

ProblemWhen I run the below code on Pig View (HDP 2.5.3 and Ambari 2.4.2) in Kerberos enabledREGISTER '/usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-pig-adapter.jar';a = LOAD 'test123.foo' USING org.apache.hive.hcatalog.pig.HCatLoader();dump a;I get the following errors- PIG log...07 Mar 2017 12:22:32,971 ERROR [ambari-client-thread-28] [PIG 1.0.0 Pig] JobService:232 - Exception occurred :java.io.FileNotFoundException: File /user/<userID>/pig/jobs/pighivetest_07-03-2017-12-22-15/stderr not found....Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File /user/<userID>/pig/jobs/pighivetest_07-03-2017-12-22-15/stderr not found....- Application log...2017-03-07 12:22:45,223 [main] ERROR org.apache.thrift.transport.TSaslTransport - SASL negotiation failurejavax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.se

IntroductionNiFi Site to Site (S2S) is a communication protocol used to exchange data between NiFi instances or clusters. This protocol is useful for use case where we have geographically distributed clusters that need to communicate. Examples include:IoT: collect data from edge node (MiNiFi) and send them to NiFi for aggregation/storage/analysisConnected cars : collect data locally by city or country with a local HDF cluster, and send it back to a global HDF cluster in core Data CenterReplication : synchronization between two HDP clusters (on prem/cloud or Principal/DR)S2S provides several benefits such as scalability, security, load balancing and high availability. More information can be found hereContextNiFi can be secured by enabling SSL and requiring users/nodes to authenticate with certificates. However, in some scenarios, customers have secured and unsecured NiFi clusters that should communicate. The objective of this tutorial is to show two approaches to achieve this. Discussi

There are two great additions you can make to your current Hive. The first is HPL/SQL that brings stored procedure programming to the Hadoop world. The second is Hive Mall which brings advanced functions and machine learning to your Hive queries.HPL/SQL HPL/SQL is included in Hive 2.0 and will be included in Hive 2.1 on HDP 2.6. You can manually download and install it now. It is Hybrid Procedural SQL on Hadoop. For developers coming from Oracle and SQL Server, these procedures will feel very familiar and will allow you to port a lot of your existing PL/SQL and TSQL code over to Hive.This gives you another interface to Hive and Hadoop, it will be included in future Hadoop and be tied into the very fast Hive LLAP 2.1.HPL/SQL https://community.hortonworks.com/content/idea/43847/hplsql-make-sql-on-hadoop-more-dynamic.htmlhttp://www.hplsql.org/connections http://www.hplsql.org/cli http://www.hplsql.org/download http://www.hplsql.org/start To Run A Stored Procedurecd hplsql-0.3.17 ./hpl