Resources

SYMPTOM :Hive query with group by clause stuck in reducer phase for a very long time having large amount of dataROOT CAUSE:This happens in the case when GROUPBY clause is not optimized. By default Hive puts the data with the same group-by keys to the same reducer. If the distinct value of the group-by columns has data skew, one reducer may get most of the shuffled data and will be stuck for a very long time on this reducer. WORKAROUND:In this case increasing the tez container memory will not help. We can avoid data skewness using the following properties before running the query,>set hive.tez.auto.reducer.parallelism=true >set hive.groupby.skewindata=true ; >set hive.optimize.skewjoin=true;

This is due to a setting in yarn-site.xml:yarn.resourcemanager.fs.state-store.retry-policy-specHere is an example: yarn.resourcemanager.fs.state-store.retry-policy-spec: 2000,500It is specified in pairs of sleep-time and number-of-retriesThis will mean that Hive Client will try connecting to nn1 for 500 times at an interval of 2000 Milliseconds.Contingent on the fact that it could not connect to nn1 for 500 times at an interval of 2000 Milliseconds, it will then try connecting to nn2.Resolution/Workaround: Lower the value of yarn.resourcemanager.fs.state-store.retry-policy-spec to 1000,10.

ISSUE : Hive Metastore stops responding until a restart happens.WORKAROUND : Please collect the jstack output 5 times at an interval of 30 seconds when hivemetastore is hung.jstack -F <hivemetastorepid> > jstack.out Then analyze the jstack output and look for the keyword "locked".If you see something similar to below then there are locked threads with respect to filesystem cache."pool-5-thread-180" #12021 prio=5 os_prio=0 tid=0x00007f63de37e000 nid=0x3900 runnable [0x00007f6391fc3000] java.lang.Thread.State: RUNNABLEat org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2886)- locked <0x00000005c0032ca8> (a org.apache.hadoop.fs.FileSystem$Cache)at org.apache.hadoop.fs.FileSystem.closeAllForUGI(FileSystem.java:473)at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:131)Please add the following to hive-site.xml and restart Hive services. These properties will disable the hdfs filesystem cache.fs.hdfs.impl.disable.cache=true

# SANDBOX must have only Host-only network, quorum is sandbox.hortonworks.com /etc/hosts is 192.168.56.101 sandbox.hortonworks.com# create an hbase table from hiveCREATE TABLE IF NOT EXISTS hbase_hive_table(key string, value string)STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf:json")TBLPROPERTIES ("hbase.table.name" = "hbase_hive_table");# in hbase shell access the tablehbase(main):001:0> describe 'hbase_hive_table'Table hbase_hive_table is ENABLEDhbase_hive_table COLUMN FAMILIES DESCRIPTION{NAME => 'cf', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}1 row(s) in 0.2860 seconds# insert into HBase table through HiveINSERT OVERWRITE TABLE HBASE_HIVE_TABLE SELECT CODE, DESCRIPT

SYMPTOM Select statement fails for view with different orderingFAILING QUERIES: select id, dept, emp, fname from testview order by id, dept; select id, emp, dept, fname from testview order by id, dept;select emp, dept, id, fnamefrom testview order by id, dept;SUCCESSFUL QUERIES:select emp, fname, id, dept from testview order by id, dept; select emp, citystate, fname, dept from testview order by id, dept;select emp, fname, dept, id from testview order by id, dept;EXCEPTION:Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating VALUE._col1 at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:86) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:343) ... 17 moreCaused by: java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method) at org.apache.hadoop.io.Text.set(Text.java:225) at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryHiveVarchar.init(LazyBinaryHiveVarcha

PROBLEM DESCRIPTION:Flume agent configured without any sources fails to start in Ambari. However, a message in the service status log indicates that the flume agent has started successfully. The following sample configuration file works in flume node if ran manually with flume-ng command. However, the same configuration fails with Ambari.# Flume agent configagent1.sinks = HdfsSink1agent1.channels = channel1agent1.channels.channel1.type=org.apache.flume.channel.kafka.KafkaChannelagent1.channels.channel1.brokerList=node11.openstacklocal:6667agent1.channels.channel1.kafka.topic=testagent1.channels.channel1.zookeeperConnect=node11.openstacklocal:2181agent1.channels.channel1.capacity=10000agent1.channels.channel1.transactionCapacity=1000agent1.channels.channel1.parseAsFlumeEvent=falseagent1.channels.channel1.kafka.consumer.group.id=test.hdfs-cagent1.sinks.HdfsSink1.channel=channel1agent1.sinks.HdfsSink1.hdfs.appendTimeout=10000agent1.sinks.HdfsSink1.hdfs.batchSize=1000agent1.sinks.HdfsSink

PROBLEM DESCRIPTION:Flume agent configured without any sources fails to start in Ambari. However, a message in the service status log indicates that the flume agent has started successfully. The following sample configuration file works in flume node if ran manually with flume-ng command. However, the same configuration fails with Ambari.# Flume agent configagent1.sinks = HdfsSink1agent1.channels = channel1agent1.channels.channel1.type=org.apache.flume.channel.kafka.KafkaChannelagent1.channels.channel1.brokerList=node11.openstacklocal:6667agent1.channels.channel1.kafka.topic=testagent1.channels.channel1.zookeeperConnect=node11.openstacklocal:2181agent1.channels.channel1.capacity=10000agent1.channels.channel1.transactionCapacity=1000agent1.channels.channel1.parseAsFlumeEvent=falseagent1.channels.channel1.kafka.consumer.group.id=test.hdfs-cagent1.sinks.HdfsSink1.channel=channel1agent1.sinks.HdfsSink1.hdfs.appendTimeout=10000agent1.sinks.HdfsSink1.hdfs.batchSize=1000agent1.sinks.HdfsSink

While this article provides a mechanism through which we could setup Spark with HiveContext, there are some limitation that when using Spark with HiveContext. For e.x Hive support writing query result to HDFS using the "INSERT OVERWRITE DIRECTORY" i.e INSERT OVERWRITE DIRECTORY 'hdfs://cl1/tmp/query'SELECT * FROM REGIONAbove command will result is writing the result of above query to HDFS. However if the same query is passed to Spark with HiveContext, this will fail since "INSERT OVERWRITE DIRECTORY" is not a supported feature when using Spark. This is tracked via this jira. If the same needs to be achieved via spark -- it could achieved by using the Spark CSV library ( required in case of Spark1 ).Below is the code snippet on how to achieve the same.DataFrame df = hiveContext.sql("SELECT * FROM REGION"); df.write() .format("com.databricks.spark.csv") .option("delimiter", "\u0001") .save("hdfs://cl1/tmp/query");Above command will save the result in HDFS under dir /tm

By default, Cloudbreak uses a built-in PostgreSQL database to persist data. For production environments, we suggest that you use an external database, an RDS served by your cloud provider. If you choose to use the default database, you should know that Cloudbreak deployer includes features for dumping and restoring built-in databases. Cloudbreak deployer uses Docker for the underlying infrastructure and uses Docker volume for storing data. There are two separate volumes:a volume called `common` for storing live dataa volume called `cbreak_dump` for database dumps You can override default live data volume any time by extending your `Profile` with the following variable: export COMMON_DB_VOL="my-live-data-volume" To create database dumps, execute the following commands: cbd db dump common cbdbcbd db dump common uaadbcbd db dump common periscopedb The dump command has an optional third parameter, the `name` of the dump. If you give your dump a name, Cloudbreak deployer will cre

SYMPTOMCREATE EXTERNAL TABLE test(id STRING,dept STRING)row format delimitedfields terminated by ','location '/user/hdfs/testdata/';ROOT CAUSEThe files under location provided while creating table are structured in following way/user/hdfs/testdata/1/test1/user/hdfs/testdata/2/test2/user/hdfs/testdata/3/test3/user/hdfs/testdata/4/test4RESOLUTIONTo make the subdirectories accessible set the following two properties before executing the create table statementset mapred.input.dir.recursive=true;set hive.mapred.supports.subdirectories=true;

PROBLEM: For an external hive table created based on hbase, if there are any missing mappings or any other issues (Syntactical), create table statement is executed successfully. You can see the table is created. However, while trying to insert data in that table, following error is seen: 08S01: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1495472057323_3947_1_00, Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.serde2.SerDeException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyStringObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.MapObjectInspector at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:800) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838) at org.apache.hadoop.hive.ql.exec.SelectOpera

PROBLEM: Ambari alert with Operation not permitted: '/var/lib/ambari-agent/tmp/curl_krb_cache' errorAlert: ERROR 2017-04-27 20:17:10,287 alert_ha_namenode_health.py:185 - [Alert] NameNode High Availability Health on taco1.example.com fails:Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/alerts/alert_ha_namenode_health.py", line 171, in execute kinit_timer_ms = kinit_timer_ms) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 106, in curl_krb_request os.chmod(curl_krb_cache_path, 0777)OSError: [Errno 1] Operation not permitted: '/var/lib/ambari-agent/tmp/curl_krb_cache' ROOT CAUSE: Ambari running as non-root user and /var/lib/ambari-agent is owned by root.SOLUTION: Move the directory to tmp location or change the ownership of the directory per non-root user.$ mv /var/lib/ambari-agent/tmp /var/lib/ambari-agent/tmp.old OR $ chown <non-root-user>:<non-root-

PROBLEM: Unable to run insert query from Ambari Hive View with Kerberos enabled. Error: Error in /var/log/ambari-server/hive-next-view/hive-view.logat java.lang.Thread.run(Thread.java:745)Caused by: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 fromorg.apache.hadoop.hive.ql.exec.tez.TezTask at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:348) at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:251) at org.apache.ambari.view.hive2.HiveJdbcConnectionDelegate.execute(HiveJdbcConnectionDelegate.java:49) at org.apache.ambari.view.hive2.actor.StatementExecutor.runStatement(StatementExecutor.java:87) at org.apache.ambari.view.hive2.actor.StatementExecutor.handleMessage(StatementExecutor.java:70) at org.apache.ambari.view.hive2.actor.HiveActor.onReceive(HiveActor.java:38) at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)Corresponding Error in RM UI Application_xxxx

PROBLEM: Installation fails with different HDP version than desired when registering a node via Ambari.For example: When registering and installing hdp 2.4.3 and some host fails on installation and tries to install hdp 2.5.3Error:2017-05-13 20:37:05,278 - Will install packages for repository version 2.4.3.0-2272017-05-13 20:37:05,278 - Repository['HDP-2.4.3.0-227'] {'append_to_file': False, 'base_url': 'http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.4.3.0/', 'action': ['create'], 'components': ['HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-2.4.3.0-227', 'mirror_list': None}2017-05-13 20:37:05,317 - File['/etc/yum.repos.d/HDP-2.4.3.0-227.repo'] {'content': '[HDP-2.4.3.0-227]\nname=HDP-2.4.3.0-227\nbaseurl=http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.4.3.0/\n\npath=/\nenabled=1\ngpgcheck=0'

The latest and greatest version of Hortonworks HDB i.e version 2.2.0.0 addresses many of the concerns from customers/prospects who have been considering Hortonworks HDB for their production data analytics workloads.Below is the list of major improvements available in Hortonworks HDB 2.2.0.0 over its predecessor. Ranger support (in BETA mode)Lack of support for Ranger has been a deal breaker in field in past. Hortonworks HDB 2.2.0.0 is step in right direction and includs Beta support for Apache Ranger. Hortonworks HDB 2.2.0.0 now include a RESTful service that provides integration between HDB and Ranger policy management. Ranger integration enables you to use Apache Ranger to authorize user access to Hortonworks HDB resources. Using Ranger enables you to manage all of your Hadoop components’ authorization policies using the same user interface, policy store, and auditing stores. For more details on how to use Ranger for authorization in Hortonworks HDB 2.2.0.0, please refer to http://h

ISSUE : Simple select query against an ORC table without limit clause is failing with the below exception.2017-06-26 16:00:35,228 ERROR [main]: CliDriver (SessionState.java:printError(993)) - Failed with exception java.io.IOException:java.lang.RuntimeException: serious problem java.io.IOException: java.lang.RuntimeException: serious problem at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:520) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:427) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1765) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:237) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:169) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:380) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:740) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:685) at

Apache NiFi has grown tremendously over the past 2 and a half years since it was open sourced. The community is continuously thinking of, implementing, and contributing amazing new features. The newly released version 1.2.0 of NiFi is no exception. One of the most exciting features of this new release is the QueryRecord Processor and the Record Reader and Record Writer components that go along with it. If you aren't familiar with those components, there's a blog post that explains how those work.This new Processor, powered by Apache Calcite, allows users to write SQL SELECT statements to run over their data as it streams through the system. Each FlowFile in NiFi can be treated as if it were a database table named FLOWFILE. These SQL queries can be used to filter specific columns or fields from your data, rename those columns/fields, filter rows, perform calculations and aggregations on the data, route the data, or whatever else you may want to use SQL for. All from the comfy confines o

Issue:Supposedly if the table has large no of partitions in the order of 2800 or above and you wanted the API call to return all the partitions of the table in JSON format then you'll end up with the above errorAPI:curl --negotiate -u: http://mrt1.openstacklocal:50111/templeton/v1/ddl/database/<db name>/table/<table name>/partition > /tmp/partition_jsonResolution:The error message is not because of the no. of partitions in the table. JSON responses are limited to 1MB in size. Responses over this limit must be stored into HDFS using provided options instead of being directly returned. Workaround:You can use HCatalog, if your DDL command return results greater than 1MB/usr/hdp/<hdp version>/hive/bin/hcat -e "use <dbname>; show partitions <tablename>; " -D hive.ddl.output.format=json -D hive.format=json -D hive.metastore.token.signature=hcat -D proxy.user.name=<username>Reference:https://cwiki.apache.org/confluence/display/Hive/WebHCat+UsingWebHCat

ISSUE : Did OS Patch on all their hosts on the cluster and after that Datanodes are started but none of them are alive in the HDFS Dashboard and all the blocks are showing missingLinux cl-master1 2.6.32-696.3.2.el6.x86_64 #1 SMP Wed Jun 7 11:51:39 EDT 2017 x86_64 x86_64 x86_64 GNU/LinuxRESOLUTION : This is a Kernel Bug https://issues.apache.org/jira/browse/HDFS-12029 Please put the Namenode in Safe mode using the below commandHdfs dfsadmin –safemode enterThen add “-Xss2m” on in between the below lines/usr/hdp/2.6.0.3-8/hadoop-hdfs/bin/hdfs.distro on all the Datanodesexec "$JSVC" \-Xss2m \-Dproc_$COMMAND -outfile "$JSVC_OUTFILE" \ -errfile "$JSVC_ERRFILE" \ -pidfile "$HADOOP_SECURE_DN_PID" \ -nodetach \ -user "$HADOOP_SECURE_DN_USER" \ -cp "$CLASSPATH" \ $JAVA_HEAP_MAX $HADOOP_OPTS \ org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter "$@"Please restart the data nodes from Ambari and leave the safe mode o

I'd like to announce the availability of HDP 2.5.6.0. This release continues from HDP 2.5.5.0, adding about 60 Apache JIRAs and includes Knox and YARN security fixes. There are no features introduced with this maintenance release.The release details can be found in the HDP 2.5.6.0 Release Notes.Supported OS platforms are the usual CentOS 6/7, SLES 11/12, Debian 7 and Ubuntu 12/14.This will soon be available on Azure HDInsight as soon as our testing is unblocked due to some other issues.Thanks to everyone who lent a hand in this release, including the Apache community.

import refrom pyspark.sql import RowAPACHE_ACCESS_LOG_PATTERN = '^(\S+) (\S+) (\S+) \[([\w:/]+\s[+\-]\d{4})\] "(\S+) (\S+) (\S+)" (\d{3}) (\d+)'# Returns a dictionary containing the parts of the Apache Access Log.def parse_apache_log_line(logline): match = re.search(APACHE_ACCESS_LOG_PATTERN, logline) if match is None: # Optionally, you can change this to just ignore if each line of data is not critical. # For this example, we want to ensure that the format is consistent. raise Exception("Invalid logline: %s" % logline) return Row( ipAddress = match.group(1), clientIdentd = match.group(2), userId = match.group(3), dateTime = match.group(4), method = match.group(5), endpoint = match.group(6), protocol = match.group(7), responseCode = int(match.group(8)), contentSize = long(match.group(9)))log_files = "hdfs://dataset/apache_logs/"raw_log_files = sc.textFile(log_files)r

Hadoop is a marvelous ecosystem, so many improvement in the last few years and so many way to extract insight from Big Data now. There is also an other sight that Hadoop can be complex to manage especially with new release, projects, versions, features and security. This is one of the reason why it is highly recommended to do DevOps. DevOps helps to enforce testing, to ensure smooth transition between environment and so to improve overall quality of any project running on the platform.In this presentation, I tried to explain the basic concept of DevOps and how Hadoop integrate nicely with tools like Jenkins, Ansible, Git, Maven and how it is amazing to use the advantage of a Big Data platform to monitor each component, projects and quality (delivery, performance & SLA, logs).

Problem:In HDP 2.3.4 and Ambari 2.4.2, Storm nimbus was not coming up:2017-05-30 14:35:18.756 o.a.t.s.TThreadPoolServer [ERROR] Error occurred during processing of message. java.lang.RuntimeException: No nimbus leader participant host found, have you started your nimbus hosts??Following is the error from Nimbus log:2017-05-30 14:35:18.750 b.s.d.nimbus [WARN] principal: storm@EXAMPLE.COM is trying to impersonate principal: ambari-server@EXAMPLE.COM 2017-05-30 14:35:18.751 b.s.d.nimbus [WARN] impersonation attempt but nimbus.impersonation.authorizer has no authorizer configured. potential security risk, please see SECURITY.MD to learn how to configure impersonation authorizer. 2017-05-30 14:35:18.756 o.a.t.s.TThreadPoolServer [ERROR] Error occurred during processing of message. java.lang.RuntimeException: No nimbus leader participant host found, have you started your nimbus hosts?Cause:Property ''nimbus.impersonation.authorizer" was set to 'org.apache.storm.security.auth.authorizer.Imper

Problem Description:Unable to start storm nimbus from Ambari, throwing following error:raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh ln -s /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common-*.jar /usr/hdp/current/storm-nimbus/lib/ambari-metrics-storm-sink.jar' returned 1. ln: target `/usr/hdp/current/storm-nimbus/lib/ambari-metrics-storm-sink.jar' is not a directoryCause: Two legacy jars under storm lib:# ls -ld /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common* -rw-r--r-- 1 root root 1937430 May 11 15:57 /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common-2.4.1.0.22.jar -rw-r--r-- 1 root root 1933392 Nov 23 2016 /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common-2.4.2.0.136.jarSolution:As the Ambari version was 2.4.2 so, moved ambari-metrics-storm-sink-legacy-with-common-2.4.1.0.22.jar to a different location and started storm nimbus.

Following was the error in supervisor log:2017-06-09 14:40:18.348 o.a.s.d.supervisor [ERROR] Error on initialization of server mk-supervisor java.lang.RuntimeException: java.lang.ClassNotFoundException: backtype.storm.generated.LSSupervisorId at org.apache.storm.utils.LocalState.deserialize(LocalState.java:83) at org.apache.storm.utils.LocalState.get(LocalState.java:130) at org.apache.storm.local_state$ls_supervisor_id.invoke(local_state.clj:61) at org.apache.storm.daemon.supervisor$standalone_supervisor$reify__7977.prepare(supervisor.clj:1216) at org.apache.storm.daemon.supervisor$fn__7833$exec_fn__3537__auto____7834.invoke(supervisor.clj:766) at clojure.lang.AFn.applyToHelper(AFn.java:160) at clojure.lang.AFn.applyTo(AFn.java:144) at clojure.core$apply.invoke(core.clj:630) at org.apache.storm.daemon.supervisor$fn__7833$mk_supervisor__7878.doInvoke(supervisor.clj:764) at clojure.lang.RestFn.invoke(RestFn.java:436) at org.apache.storm.daemon.supervisor$_launch.invoke(supervisor.clj:120