Community Articles

Mirroring Datasets Between Hadoop Clusters with Apache Falcon Introduction Apache Falcon is a framework to simplify data pipeline processing and management on Hadoop clusters. It provides data management services such as retention, replications across clusters, archival etc. It makes it much simpler to onboard new workflows/pipelines, with support for late data handling and retry policies. It allows you to easily define relationship between various data and processing elements and integrate with metastore/catalog such as Hive/HCatalog. Finally it also lets you capture lineage information for feeds and processes. In this tutorial we are going walk the process of mirroring the datasets between Hadoop clusters. Prerequisites Download Hortonworks Sandbox 2.5 Complete the Learning the Ropes of the Hortonworks Sandbox tutorial, you will need it for logging into Ambari as an administrator user. Complete the Leveraging Apache Falcon with Your Hadoop Clusters tutorial to start the

Generally you will get permission denied error as below when you run oozie shell action without setting environment variable to username who is submitting the oozie workflow.Permission denied: user=yarn, access=WRITE oozie shell actionBelow are my sample configsworkflow.xml<workflow-app xmlns="uri:oozie:workflow:0.3" name="shell-wf"> <start to="shell-node"/> <action name="shell-node"> <shell xmlns="uri:oozie:shell-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <exec>test.sh</exec> <file>/user/root/test.sh</file> </shell> <ok to="end"/> <e

You can execute any shell script using ExecuteProcess Processor in Nifi. For example I have a very simple shell script below which writes "Hello world" to a file.

Synopsis:In addition to authentication and access control, data encryption adds a robust layer of security, by making data unreadable in transit over the network or at rest on a disk. Encryption helps protect sensitive data, in the case of an external breach or unauthorized access by privileged users. The automation of this task is expected to save close to 4-6 hours of manual intervention per occurrence.It can be used as a disaster recovery custom solution.Github link for the code:https://github.com/RajdeepBiswas/EncryptedDataTransfer Script (common code) location:cluster1: Under root@cluster1 /root/scripts/dataCopy/hdfs_data_move.sh cluster2: Under root@cluster2 /root/scripts/dataCopy/hdfs_data_move.sh Usage: Scenario1: For copying encrypted hdfs folder from cluster2 to cluster1 Example folder name: /tmp/zone_encr_test encrypted with key “testKey123” In cluster2: sudo su root cd /root/scripts/dataCopy/ ./hdfs_data_move.sh export keys After above execution finishes:In cluster1: sudo s

DataTransfer:Generic HDFS data and Hive Database transfer automation between any environment(Production/QA/Development) utilizing Amazon S3 storageGithub link for the code:https://github.com/RajdeepBiswas/DataTransfer Synopsis: Exporting and importing data between different layers of environment like production, QA and development is a recurring task.Due to security considerations, this environments cannot talk to each other. Hence we are using Amazon S3 storage as an intermediate storage point for transferring data seamlessly across environments. The automation of this task is expected to save close to 4 hours of manual intervention per occurrence. The code can be re-used for disaster recovery automation.Code location: Place your scripts here: Script: /root/scripts/dataCopy/datamove.shConfiguration File: /root/scripts/dataCopy/conf/conf_datamove_devs3.conf Note: The name of the configuration files can be different for different S3 locations. This can be passed to the script. But it ne

This failure to start HiveServer Interactive / Hive LLAP is due to a known problem in Ambari 2.5 where certain keytab files are not generated if you enable LLAP after your cluster is Kerberized. The Ambari Kerberos wizard generates keytabs for all services that are present as of the time the Kerberos Wizard is run. If HiveServer Interactive is not enabled when the wizard runs, certain essential keytabs will not be present when you try to enable HiveServer Interactive / LLAP, nor are they generated at that time.There are two options for resolving this problem:Regenerate keytabs using the Ambari Kerberos wizard, refer to the Ambari documentation for this process.On all cluster nodes, copy the hive.service.keytab to hive.llap.zk.sm.keytab. If your keytabs are stored in the default location, cp /etc/security/keytabs/hive.service.keytab /etc/security/keytabs/hive.llap.zk.sm.keytab

PROBLEM:Click Alerts and then Actions > Manage Alert Group -> Custom Alert Group.Then, click the + sign on the right side and pick any alert definition and press OK. Click Save and you will see the 500(server error) Server Error on the alert group screen:And in the ambari-server.log there is error:WARN [qtp-ambari-client-510524] ServletHandler:563 - /api/v1/clusters/<cluster-name>/alert_groups/155java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429) at java.util.HashMap$KeyIterator.next(HashMap.java:1453) at org.eclipse.persistence.indirection.IndirectSet$1.next(IndirectSet.java:471) at org.apache.ambari.server.orm.entities.AlertGroupEntity.setAlertTargets(AlertGroupEntity.java:313) at org.apache.ambari.server.controller.internal.AlertGroupResourceProvider.updateAlertGroups(AlertGroupResourceProvider.java:344) at org.apache.ambari.server.controller.internal.AlertGroupResourceProvider.access$100(AlertGroupResourceProvider.jav

Here is a simple example--Running a Select count inside Beeline for a table by name patb0: jdbc:hive2://node1:10000> select * from patb;| patb.ida | patb.name | patb.month || 5 | Ujjain | 6 |1 row selected (0.388 seconds)Now we go ahead and set a variable.0: jdbc:hive2://node1:10000> set hivevar:tablename=patb;No rows affected (0.005 seconds)0: jdbc:hive2://node1:10000> select * from ${hivevar:tablename};| patb.ida | patb.name | patb.month || 5 | Ujjain | 6 |1 row selected (0.14 seconds)Resolution: Please set the variable as set hivevar:tablename=patb;

Consider the example:Total input paths = 7 Input size for job = 510K1) While are using a custom InputFormat which extends ‘org.apache.hadoop.mapred.FileInputFormat’ and having ‘isSplitable’ as false.Expected : 7 splits [As FileInputFormat doesn't split file smaller than blockSize (128 MB) so there should be one split per file] Actual: 4 splits 2) Default value for 'hive.input.format' is CombineHiveInputFormat. After setting ‘set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;’, there are 7 splits as expected. From above two points, it looks hive uses ‘CombineHiveInputFormat’ on top of the custom InputFormat to determine number of splits. How splits were calculated:For deciding the number of mappers when using CombineInputFormat, data locality plays a role. Now to find where those files belong we can get it from command: hadoop fsck /<file-path> -files -blocks -locations 1. On. a.a.a.a/user/user1/hive/split/file1_0000[/default-rack/a.a.a.a:1019, /default-rack/

PROBLEM:After moving the Zookeeper servers and setting correctly in the yarn configs, Resource managers come up but are in standby state.Even after removing the znode - rmstore, none of the nodes transition to active.ROOT CAUSE:Zookeeper data is stored in znode yarn-leader-election. This is used for RM leader election. This znode has stale data about zookeeper leader.RESOLUTION:1. Login into zkcli2. rmr /yarn-leader-election3. Restart Resource managers.

Customer has a Hive Query with a Group By Clause and is stuck on the last Reducer. This looks like a Reducer Skew in the Group By Clause.You may please use the below said configuration in the Hive CLI at the session level.Inside the Hive CLI.set hive.groupby.skewindata=true;<Run the Query>

SYMPTOM: FAILED: SemanticException [Error 10122]: Bucketized tables do not support INSERT INTO: Table: XXXXROOT CAUSE: Customer has a ACID enabled Cluster. Customer created a Table for which the TBLPROPERTIES ('transactional'='true') was not set. Customer's table has 50 Billion records in it.WORKAROUND: For a table to be transactional, we need to set TBLPROPERTIES ('transactional'='true') We can convert a table to Transactional Table using the Alter Table Command, assuming the table satisfies acid requirements --Bucketed,--Using ORC--Not sorted.RESOLUTION: Logically alter table to set transactional=true is supported.

PROBLEM: When the hostname is mixed like: 172.26.93.148 GRAFANA-hdp253-s1.openstacklocal GRAFANA-hdp253-s1 172.26.93.149 GRAFANA-hdp253-s2.openstacklocal GRAFANA-hdp253-s2 172.26.93.150 GRAFANA-hdp253-s3.openstacklocal GRAFANA-hdp253-s3 Ambari creates the datasource with a lower case hostname:Once you enter the hostname with mixed case, Grafana works:RESOLUTION Grafana 2.6.0 backend uses Go 1.5. Go's DNS lookup had a bug where the look up is case sensitive: https://github.com/golang/go/issues/12806 We can manual workaround by using the original casing in the Grafana Data Source URL as mentioned in the description. Another workaround is to update the /etc/hosts file to have all lowercase patterns as well. This bug was fixed in Go 1.6, so we will no longer have this issue once we upgrade Grafana to a later version (say 4.1.x) in a future version of Ambari.

PROBLEM:To run Hive in Silent Mode. HOW TO hive -S -e "Show databases" For example: hive -S -e "show databases;" SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/spark/lib/spark-assembly-1.4.1.2.3.2.0-2950-hadoop2.7.1.2.3.2.0-2950.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] WARNING: Use "yarn jar" to launch YARN applications. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/spark/lib/spark-assembly-1.4.1.2.3.2.0-2950-hadoop2.7.1.2.3.2.0-2950.jar!/org/slf4

Please log on to the backend database: mysql> use hive; Database changed: mysql> show tables; The table of importance to us in this scenario is HIVE_LOCKS. (MySQL is sensitive to Upper and Lower Caps) mysql> describe HIVE_LOCKS;+---------------------+--------------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |+---------------------+--------------+------+-----+---------+-------+| HL_LOCK_EXT_ID | bigint(20) | NO | PRI | NULL | || HL_LOCK_INT_ID | bigint(20) | NO | PRI | NULL | || HL_TXNID | bigint(20) | YES | MUL | NULL || HL_DB | varchar(128) | NO | | NULL | || HL_TABLE | varchar(128) | YES | | NULL | || HL_PARTITION | varchar(767) | YES | | NULL | || HL_LOCK_STATE | char(1) | NO | | NULL | || HL_LOCK_TYPE | char(1) | NO | | NULL | || HL_LAST_HEARTBEAT | bigint(20) | NO | | NULL | || HL_ACQUIRED_AT | bigint(20) | YES | | NULL | || HL_USER | varchar(128) | NO | | NULL | || HL_HOST | varchar(128) | NO | | NULL | || HL_HEARTBEAT_COUNT | int(11) |

PROBLEM: For example, while running from shell, we usually append the single quotes around the date type: hive> select * from students where datestamp='2014-09-23'; OKfred flintstone351.282014-09-23 Time taken: 0.761 seconds, Fetched: 1 row(s) But in case of Hue, internally it doesn't append these single quotes and throws error on browse data. STEPS TO REPRODUCE: 1. Create table: CREATE TABLE students(name varchar(64), age int, gpa decimal(3,2)) PARTITIONED BY ( datestamp date);2. Insert: INSERT INTO TABLE students PARTITION (datestamp = '2014-09-23') VALUES ('fred flintstone', 35, 1.28);3. Login into hue -> Go to HCatalog -> Tables -> select 'students' -> Click on Browse data 4. This will generate error:RESOLUTION: There is an internal bug reported. Please reach out to support.

While designing your flow, one will think of handling failures as well. Some may ignore (terminate) it while some may want to analyze their failed flow files. You can view/download any flowfile in the connection via NiFi UI. However if there are multiple flowfiles then this becomes little tedious since you have to do it for each flowfile. . In that case, I used below flow to put all failed flow files to a temp location on my nifi node and then access it from there for further analysis. In this I am generating random 2B text file and passing it to FTP server. If FTP server is unreachable it will put all flow files to failure/reject queue. You can list all files in this queue. I have viewed one such file in image below. Then all these files are passed to PutFile processor which is configured to write all the files it receives to /tmp/test location on the nifi node. Please note: There could be multiple ways to do this. This is the approach used by me to quickly download all failed flow

I didn't want to clear the ATS and lose all the job history. I wanted to fix the corruption and preserve the ATS leveldb entries. 2017-06-15 19:17:43,871 INFO service.AbstractService (AbstractService.java:noteFailure(272)) - Service org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer failed in state INITED; cause: org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: 43 missing files; e.g.: /data/hadoop/ats/leveldb/leveldb-timeline-store/domain-ldb/000015.sst <snipped>2017-06-15 19:17:43,871 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(211)) - Stopping ApplicationHistoryServer metrics system... 2017-06-15 19:17:43,873 INFO impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted. 2017-06-15 19:17:43,875 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(217)) - ApplicationHistoryServer metrics system stopped. 2017-06-15

PROBLEM: We see below message on initial runs:2017-02-28 19:37:48,681 INFO [main] impl.TimelineClientImpl: Timeline service address: http://<timeline-server-hostname>:8188/ws/v1/timeline/ 2017-02-28 19:37:48,823 INFO [main] client.AHSProxy: Connecting to Application History server at <history-server-hostname>/<history-server-ip>:10200 2017-02-28 19:37:49,016 WARN [main] ipc.Client: Failed to connect to server: <resource-manager-A>/<resource-manager-A-ip>:8032: retries get failed due to exceeded maximum allowed retries number: 0 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)ROOT CAUSE: In Yarn configs -> custom yarn-site.xml, you might have configuration for rm1 as resource-manager-

This is a known issue https://issues.apache.org/jira/browse/NIFI-3800 which is fixed in Apache NiFi 1.2.0 (HDF 2.1.4 and later) Workaround: You can downgrade your Java version to openjdk-1.8.0_121 if you are using something higher than that.

SYMPTOM :Hive query with group by clause stuck in reducer phase for a very long time having large amount of dataROOT CAUSE:This happens in the case when GROUPBY clause is not optimized. By default Hive puts the data with the same group-by keys to the same reducer. If the distinct value of the group-by columns has data skew, one reducer may get most of the shuffled data and will be stuck for a very long time on this reducer. WORKAROUND:In this case increasing the tez container memory will not help. We can avoid data skewness using the following properties before running the query,>set hive.tez.auto.reducer.parallelism=true >set hive.groupby.skewindata=true ; >set hive.optimize.skewjoin=true;

This is due to a setting in yarn-site.xml:yarn.resourcemanager.fs.state-store.retry-policy-specHere is an example: yarn.resourcemanager.fs.state-store.retry-policy-spec: 2000,500It is specified in pairs of sleep-time and number-of-retriesThis will mean that Hive Client will try connecting to nn1 for 500 times at an interval of 2000 Milliseconds.Contingent on the fact that it could not connect to nn1 for 500 times at an interval of 2000 Milliseconds, it will then try connecting to nn2.Resolution/Workaround: Lower the value of yarn.resourcemanager.fs.state-store.retry-policy-spec to 1000,10.

ISSUE : Hive Metastore stops responding until a restart happens.WORKAROUND : Please collect the jstack output 5 times at an interval of 30 seconds when hivemetastore is hung.jstack -F <hivemetastorepid> > jstack.out Then analyze the jstack output and look for the keyword "locked".If you see something similar to below then there are locked threads with respect to filesystem cache."pool-5-thread-180" #12021 prio=5 os_prio=0 tid=0x00007f63de37e000 nid=0x3900 runnable [0x00007f6391fc3000] java.lang.Thread.State: RUNNABLEat org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2886)- locked <0x00000005c0032ca8> (a org.apache.hadoop.fs.FileSystem$Cache)at org.apache.hadoop.fs.FileSystem.closeAllForUGI(FileSystem.java:473)at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:131)Please add the following to hive-site.xml and restart Hive services. These properties will disable the hdfs filesystem cache.fs.hdfs.impl.disable.cache=true

# SANDBOX must have only Host-only network, quorum is sandbox.hortonworks.com /etc/hosts is 192.168.56.101 sandbox.hortonworks.com# create an hbase table from hiveCREATE TABLE IF NOT EXISTS hbase_hive_table(key string, value string)STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf:json")TBLPROPERTIES ("hbase.table.name" = "hbase_hive_table");# in hbase shell access the tablehbase(main):001:0> describe 'hbase_hive_table'Table hbase_hive_table is ENABLEDhbase_hive_table COLUMN FAMILIES DESCRIPTION{NAME => 'cf', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}1 row(s) in 0.2860 seconds# insert into HBase table through HiveINSERT OVERWRITE TABLE HBASE_HIVE_TABLE SELECT CODE, DESCRIPT

SYMPTOM Select statement fails for view with different orderingFAILING QUERIES: select id, dept, emp, fname from testview order by id, dept; select id, emp, dept, fname from testview order by id, dept;select emp, dept, id, fnamefrom testview order by id, dept;SUCCESSFUL QUERIES:select emp, fname, id, dept from testview order by id, dept; select emp, citystate, fname, dept from testview order by id, dept;select emp, fname, dept, id from testview order by id, dept;EXCEPTION:Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating VALUE._col1 at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:86) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:343) ... 17 moreCaused by: java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method) at org.apache.hadoop.io.Text.set(Text.java:225) at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryHiveVarchar.init(LazyBinaryHiveVarcha