Resources

This is a cool HAT that attaches without soldering to a Raspberry Pi 3. The main use is with Google's Android Things which is an IoT OS based on Android that you can run on Raspberry Pi and other devices.You can also use this HAT with regular Raspian and Python.This HAT is cool because it has input and output. As part of our Python script, we ingest data from it's sensor, but we also allow for input to it's A, B and C buttons clearly marked on the top. By selecting one of the buttons you can display the current temperature in Celsius, Fahrenheit or the current altitude. The altitude is some code I found commented out in the Pimoroni Rainbow HAT interface Python script. It determines altitude based on a starting value and pressure readings. Looks kind of cool and is relatively accurate. For point of reference, Princeton New Jersey is pretty close to sea level.My script is heavily copied and customized from my previous IoT Python scripts and from the RainbowHAT examples.This h

I am always ingesting social data, IoT and mobile streams into my HDFS cluster. Unfortunately my cluster is a ephemeral cloud based Hortonworks HDP 2.6 Hadoop cluster, so I don't have a permanent store for my data. I have my processes run for a few weeks and then they are destroyed. I wanted to quick way to save all my ORC files. Enter NiFi. Backup First we list from some top level directories in HDFS to capture all the files and sub-directories we want to backup. Each processor maintains a timestamp to know what files it processed already, as new files are added they will be assimilated into the flow. For massive data migration, we can run this on many nodes and use the Distributed Cache service to maintain the state. Restore The flow is very simple to restore, read from the local file system and write to HDFS. For HDFS, I use /${path} as the directory so each file is written to the correct sub-directory for it's file group. Easy it's like

So I might be late in posting this article, but I want to publicize the changes in HDP 2.6.0 documentation and blog postings for Hive, Hive Views, and Ambari that were released in early April 2017. After all, I don't think all of you out there are upgraded to HDP 2.6.0 yet.To complement the G.A. release of Hive low-latency analytical processing (LLAP), the "Hive Performance Tuning Guide" has been completely rewritten to focus on how to optimize your environment and queries for interactive processing. While the guide is tailored to run a Hive data warehouse on the Tez framework, the new edition still has relevant tips and links to information for users who do batch processing on a MapReduce framework. A couple of LLAP blogs to consider reading are "Fast Analytics in the Cloud with Hive LLAP" and "Top 5 Performance Boosters with Apache Hive LLAP".Important debugging and optimization tools to get the most out of your Hive queries and your architecture are documented in Hive View 2.0 and T

Incremental Backup of Data from HDP to Azure using Falcon for Disaster Recovery and Burst capacity Introduction Apache Falcon simplifies the configuration of data motion with: replication; lifecycle management; lineage and traceability. This provides data governance consistency across Hadoop components. Scenario In this tutorial we will walk through a scenario where email data gets processed on multiple HDP 2.2 clusters around the country then gets backed up hourly on a cloud hosted cluster . In our example: This cluster is hosted on Windows Azure. Data arrives from all the West Coast production servers. The input data feeds are often late for up to 4 hrs. The goal is to clean the raw data to remove sensitive information like credit card numbers and make it available to our marketing data science team for customer churn analysis. To simulate this scenario, we have a pig script grabbing the freely available Enron emails from the internet and feeding it into the pipeline.

Define and Process Data Pipelines in Hadoop With Apache Falcon Introduction Apache Falcon is a framework to simplify data pipeline processing and management on Hadoop clusters. It makes it much simpler to onboard new workflows/pipelines, with support for late data handling and retry policies. It allows you to easily define relationships between various data and processing elements and integrate with metastore/catalog such as Apache Hive/HCatalog. Finally it also lets you capture lineage information for feeds and processes. In this tutorial we are going to walkthrough the process of: Defining the feeds and processes Defining and executing a data pipeline to ingest, process and persist data continuously Prerequisites Download Hortonworks Sandbox Complete the Learning the Ropes of the Hortonworks Sandbox tutorial, you will need it for logging into ambari as an administrator user. Complete the Creating Falcon Cluster tutorial to start the falcon service, prepare HDFS dir

Mirroring Datasets Between Hadoop Clusters with Apache Falcon Introduction Apache Falcon is a framework to simplify data pipeline processing and management on Hadoop clusters. It provides data management services such as retention, replications across clusters, archival etc. It makes it much simpler to onboard new workflows/pipelines, with support for late data handling and retry policies. It allows you to easily define relationship between various data and processing elements and integrate with metastore/catalog such as Hive/HCatalog. Finally it also lets you capture lineage information for feeds and processes. In this tutorial we are going walk the process of mirroring the datasets between Hadoop clusters. Prerequisites Download Hortonworks Sandbox 2.5 Complete the Learning the Ropes of the Hortonworks Sandbox tutorial, you will need it for logging into Ambari as an administrator user. Complete the Leveraging Apache Falcon with Your Hadoop Clusters tutorial to start the

Generally you will get permission denied error as below when you run oozie shell action without setting environment variable to username who is submitting the oozie workflow.Permission denied: user=yarn, access=WRITE oozie shell actionBelow are my sample configsworkflow.xml<workflow-app xmlns="uri:oozie:workflow:0.3" name="shell-wf"> <start to="shell-node"/> <action name="shell-node"> <shell xmlns="uri:oozie:shell-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <exec>test.sh</exec> <file>/user/root/test.sh</file> </shell> <ok to="end"/> <e

You can execute any shell script using ExecuteProcess Processor in Nifi. For example I have a very simple shell script below which writes "Hello world" to a file.

Synopsis:In addition to authentication and access control, data encryption adds a robust layer of security, by making data unreadable in transit over the network or at rest on a disk. Encryption helps protect sensitive data, in the case of an external breach or unauthorized access by privileged users. The automation of this task is expected to save close to 4-6 hours of manual intervention per occurrence.It can be used as a disaster recovery custom solution.Github link for the code:https://github.com/RajdeepBiswas/EncryptedDataTransfer Script (common code) location:cluster1: Under root@cluster1 /root/scripts/dataCopy/hdfs_data_move.sh cluster2: Under root@cluster2 /root/scripts/dataCopy/hdfs_data_move.sh Usage: Scenario1: For copying encrypted hdfs folder from cluster2 to cluster1 Example folder name: /tmp/zone_encr_test encrypted with key “testKey123” In cluster2: sudo su root cd /root/scripts/dataCopy/ ./hdfs_data_move.sh export keys After above execution finishes:In cluster1: sudo s

DataTransfer:Generic HDFS data and Hive Database transfer automation between any environment(Production/QA/Development) utilizing Amazon S3 storageGithub link for the code:https://github.com/RajdeepBiswas/DataTransfer Synopsis: Exporting and importing data between different layers of environment like production, QA and development is a recurring task.Due to security considerations, this environments cannot talk to each other. Hence we are using Amazon S3 storage as an intermediate storage point for transferring data seamlessly across environments. The automation of this task is expected to save close to 4 hours of manual intervention per occurrence. The code can be re-used for disaster recovery automation.Code location: Place your scripts here: Script: /root/scripts/dataCopy/datamove.shConfiguration File: /root/scripts/dataCopy/conf/conf_datamove_devs3.conf Note: The name of the configuration files can be different for different S3 locations. This can be passed to the script. But it ne

This failure to start HiveServer Interactive / Hive LLAP is due to a known problem in Ambari 2.5 where certain keytab files are not generated if you enable LLAP after your cluster is Kerberized. The Ambari Kerberos wizard generates keytabs for all services that are present as of the time the Kerberos Wizard is run. If HiveServer Interactive is not enabled when the wizard runs, certain essential keytabs will not be present when you try to enable HiveServer Interactive / LLAP, nor are they generated at that time.There are two options for resolving this problem:Regenerate keytabs using the Ambari Kerberos wizard, refer to the Ambari documentation for this process.On all cluster nodes, copy the hive.service.keytab to hive.llap.zk.sm.keytab. If your keytabs are stored in the default location, cp /etc/security/keytabs/hive.service.keytab /etc/security/keytabs/hive.llap.zk.sm.keytab

PROBLEM:Click Alerts and then Actions > Manage Alert Group -> Custom Alert Group.Then, click the + sign on the right side and pick any alert definition and press OK. Click Save and you will see the 500(server error) Server Error on the alert group screen:And in the ambari-server.log there is error:WARN [qtp-ambari-client-510524] ServletHandler:563 - /api/v1/clusters/<cluster-name>/alert_groups/155java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429) at java.util.HashMap$KeyIterator.next(HashMap.java:1453) at org.eclipse.persistence.indirection.IndirectSet$1.next(IndirectSet.java:471) at org.apache.ambari.server.orm.entities.AlertGroupEntity.setAlertTargets(AlertGroupEntity.java:313) at org.apache.ambari.server.controller.internal.AlertGroupResourceProvider.updateAlertGroups(AlertGroupResourceProvider.java:344) at org.apache.ambari.server.controller.internal.AlertGroupResourceProvider.access$100(AlertGroupResourceProvider.jav

Here is a simple example--Running a Select count inside Beeline for a table by name patb0: jdbc:hive2://node1:10000> select * from patb;| patb.ida | patb.name | patb.month || 5 | Ujjain | 6 |1 row selected (0.388 seconds)Now we go ahead and set a variable.0: jdbc:hive2://node1:10000> set hivevar:tablename=patb;No rows affected (0.005 seconds)0: jdbc:hive2://node1:10000> select * from ${hivevar:tablename};| patb.ida | patb.name | patb.month || 5 | Ujjain | 6 |1 row selected (0.14 seconds)Resolution: Please set the variable as set hivevar:tablename=patb;

Consider the example:Total input paths = 7 Input size for job = 510K1) While are using a custom InputFormat which extends ‘org.apache.hadoop.mapred.FileInputFormat’ and having ‘isSplitable’ as false.Expected : 7 splits [As FileInputFormat doesn't split file smaller than blockSize (128 MB) so there should be one split per file] Actual: 4 splits 2) Default value for 'hive.input.format' is CombineHiveInputFormat. After setting ‘set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;’, there are 7 splits as expected. From above two points, it looks hive uses ‘CombineHiveInputFormat’ on top of the custom InputFormat to determine number of splits. How splits were calculated:For deciding the number of mappers when using CombineInputFormat, data locality plays a role. Now to find where those files belong we can get it from command: hadoop fsck /<file-path> -files -blocks -locations 1. On. a.a.a.a/user/user1/hive/split/file1_0000[/default-rack/a.a.a.a:1019, /default-rack/

PROBLEM:After moving the Zookeeper servers and setting correctly in the yarn configs, Resource managers come up but are in standby state.Even after removing the znode - rmstore, none of the nodes transition to active.ROOT CAUSE:Zookeeper data is stored in znode yarn-leader-election. This is used for RM leader election. This znode has stale data about zookeeper leader.RESOLUTION:1. Login into zkcli2. rmr /yarn-leader-election3. Restart Resource managers.

Customer has a Hive Query with a Group By Clause and is stuck on the last Reducer. This looks like a Reducer Skew in the Group By Clause.You may please use the below said configuration in the Hive CLI at the session level.Inside the Hive CLI.set hive.groupby.skewindata=true;<Run the Query>

SYMPTOM: FAILED: SemanticException [Error 10122]: Bucketized tables do not support INSERT INTO: Table: XXXXROOT CAUSE: Customer has a ACID enabled Cluster. Customer created a Table for which the TBLPROPERTIES ('transactional'='true') was not set. Customer's table has 50 Billion records in it.WORKAROUND: For a table to be transactional, we need to set TBLPROPERTIES ('transactional'='true') We can convert a table to Transactional Table using the Alter Table Command, assuming the table satisfies acid requirements --Bucketed,--Using ORC--Not sorted.RESOLUTION: Logically alter table to set transactional=true is supported.

PROBLEM: When the hostname is mixed like: 172.26.93.148 GRAFANA-hdp253-s1.openstacklocal GRAFANA-hdp253-s1 172.26.93.149 GRAFANA-hdp253-s2.openstacklocal GRAFANA-hdp253-s2 172.26.93.150 GRAFANA-hdp253-s3.openstacklocal GRAFANA-hdp253-s3 Ambari creates the datasource with a lower case hostname:Once you enter the hostname with mixed case, Grafana works:RESOLUTION Grafana 2.6.0 backend uses Go 1.5. Go's DNS lookup had a bug where the look up is case sensitive: https://github.com/golang/go/issues/12806 We can manual workaround by using the original casing in the Grafana Data Source URL as mentioned in the description. Another workaround is to update the /etc/hosts file to have all lowercase patterns as well. This bug was fixed in Go 1.6, so we will no longer have this issue once we upgrade Grafana to a later version (say 4.1.x) in a future version of Ambari.

PROBLEM:To run Hive in Silent Mode. HOW TO hive -S -e "Show databases" For example: hive -S -e "show databases;" SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/spark/lib/spark-assembly-1.4.1.2.3.2.0-2950-hadoop2.7.1.2.3.2.0-2950.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] WARNING: Use "yarn jar" to launch YARN applications. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/spark/lib/spark-assembly-1.4.1.2.3.2.0-2950-hadoop2.7.1.2.3.2.0-2950.jar!/org/slf4

Please log on to the backend database: mysql> use hive; Database changed: mysql> show tables; The table of importance to us in this scenario is HIVE_LOCKS. (MySQL is sensitive to Upper and Lower Caps) mysql> describe HIVE_LOCKS;+---------------------+--------------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |+---------------------+--------------+------+-----+---------+-------+| HL_LOCK_EXT_ID | bigint(20) | NO | PRI | NULL | || HL_LOCK_INT_ID | bigint(20) | NO | PRI | NULL | || HL_TXNID | bigint(20) | YES | MUL | NULL || HL_DB | varchar(128) | NO | | NULL | || HL_TABLE | varchar(128) | YES | | NULL | || HL_PARTITION | varchar(767) | YES | | NULL | || HL_LOCK_STATE | char(1) | NO | | NULL | || HL_LOCK_TYPE | char(1) | NO | | NULL | || HL_LAST_HEARTBEAT | bigint(20) | NO | | NULL | || HL_ACQUIRED_AT | bigint(20) | YES | | NULL | || HL_USER | varchar(128) | NO | | NULL | || HL_HOST | varchar(128) | NO | | NULL | || HL_HEARTBEAT_COUNT | int(11) |

PROBLEM: For example, while running from shell, we usually append the single quotes around the date type: hive> select * from students where datestamp='2014-09-23'; OKfred flintstone351.282014-09-23 Time taken: 0.761 seconds, Fetched: 1 row(s) But in case of Hue, internally it doesn't append these single quotes and throws error on browse data. STEPS TO REPRODUCE: 1. Create table: CREATE TABLE students(name varchar(64), age int, gpa decimal(3,2)) PARTITIONED BY ( datestamp date);2. Insert: INSERT INTO TABLE students PARTITION (datestamp = '2014-09-23') VALUES ('fred flintstone', 35, 1.28);3. Login into hue -> Go to HCatalog -> Tables -> select 'students' -> Click on Browse data 4. This will generate error:RESOLUTION: There is an internal bug reported. Please reach out to support.

While designing your flow, one will think of handling failures as well. Some may ignore (terminate) it while some may want to analyze their failed flow files. You can view/download any flowfile in the connection via NiFi UI. However if there are multiple flowfiles then this becomes little tedious since you have to do it for each flowfile. . In that case, I used below flow to put all failed flow files to a temp location on my nifi node and then access it from there for further analysis. In this I am generating random 2B text file and passing it to FTP server. If FTP server is unreachable it will put all flow files to failure/reject queue. You can list all files in this queue. I have viewed one such file in image below. Then all these files are passed to PutFile processor which is configured to write all the files it receives to /tmp/test location on the nifi node. Please note: There could be multiple ways to do this. This is the approach used by me to quickly download all failed flow

I didn't want to clear the ATS and lose all the job history. I wanted to fix the corruption and preserve the ATS leveldb entries. 2017-06-15 19:17:43,871 INFO service.AbstractService (AbstractService.java:noteFailure(272)) - Service org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer failed in state INITED; cause: org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: 43 missing files; e.g.: /data/hadoop/ats/leveldb/leveldb-timeline-store/domain-ldb/000015.sst <snipped>2017-06-15 19:17:43,871 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(211)) - Stopping ApplicationHistoryServer metrics system... 2017-06-15 19:17:43,873 INFO impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted. 2017-06-15 19:17:43,875 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(217)) - ApplicationHistoryServer metrics system stopped. 2017-06-15

PROBLEM: We see below message on initial runs:2017-02-28 19:37:48,681 INFO [main] impl.TimelineClientImpl: Timeline service address: http://<timeline-server-hostname>:8188/ws/v1/timeline/ 2017-02-28 19:37:48,823 INFO [main] client.AHSProxy: Connecting to Application History server at <history-server-hostname>/<history-server-ip>:10200 2017-02-28 19:37:49,016 WARN [main] ipc.Client: Failed to connect to server: <resource-manager-A>/<resource-manager-A-ip>:8032: retries get failed due to exceeded maximum allowed retries number: 0 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)ROOT CAUSE: In Yarn configs -> custom yarn-site.xml, you might have configuration for rm1 as resource-manager-

This is a known issue https://issues.apache.org/jira/browse/NIFI-3800 which is fixed in Apache NiFi 1.2.0 (HDF 2.1.4 and later) Workaround: You can downgrade your Java version to openjdk-1.8.0_121 if you are using something higher than that.