Community Articles

PROBLEM DESCRIPTION:Flume agent configured without any sources fails to start in Ambari. However, a message in the service status log indicates that the flume agent has started successfully. The following sample configuration file works in flume node if ran manually with flume-ng command. However, the same configuration fails with Ambari.# Flume agent configagent1.sinks = HdfsSink1agent1.channels = channel1agent1.channels.channel1.type=org.apache.flume.channel.kafka.KafkaChannelagent1.channels.channel1.brokerList=node11.openstacklocal:6667agent1.channels.channel1.kafka.topic=testagent1.channels.channel1.zookeeperConnect=node11.openstacklocal:2181agent1.channels.channel1.capacity=10000agent1.channels.channel1.transactionCapacity=1000agent1.channels.channel1.parseAsFlumeEvent=falseagent1.channels.channel1.kafka.consumer.group.id=test.hdfs-cagent1.sinks.HdfsSink1.channel=channel1agent1.sinks.HdfsSink1.hdfs.appendTimeout=10000agent1.sinks.HdfsSink1.hdfs.batchSize=1000agent1.sinks.HdfsSink

PROBLEM DESCRIPTION:Flume agent configured without any sources fails to start in Ambari. However, a message in the service status log indicates that the flume agent has started successfully. The following sample configuration file works in flume node if ran manually with flume-ng command. However, the same configuration fails with Ambari.# Flume agent configagent1.sinks = HdfsSink1agent1.channels = channel1agent1.channels.channel1.type=org.apache.flume.channel.kafka.KafkaChannelagent1.channels.channel1.brokerList=node11.openstacklocal:6667agent1.channels.channel1.kafka.topic=testagent1.channels.channel1.zookeeperConnect=node11.openstacklocal:2181agent1.channels.channel1.capacity=10000agent1.channels.channel1.transactionCapacity=1000agent1.channels.channel1.parseAsFlumeEvent=falseagent1.channels.channel1.kafka.consumer.group.id=test.hdfs-cagent1.sinks.HdfsSink1.channel=channel1agent1.sinks.HdfsSink1.hdfs.appendTimeout=10000agent1.sinks.HdfsSink1.hdfs.batchSize=1000agent1.sinks.HdfsSink

While this article provides a mechanism through which we could setup Spark with HiveContext, there are some limitation that when using Spark with HiveContext. For e.x Hive support writing query result to HDFS using the "INSERT OVERWRITE DIRECTORY" i.e INSERT OVERWRITE DIRECTORY 'hdfs://cl1/tmp/query'SELECT * FROM REGIONAbove command will result is writing the result of above query to HDFS. However if the same query is passed to Spark with HiveContext, this will fail since "INSERT OVERWRITE DIRECTORY" is not a supported feature when using Spark. This is tracked via this jira. If the same needs to be achieved via spark -- it could achieved by using the Spark CSV library ( required in case of Spark1 ).Below is the code snippet on how to achieve the same.DataFrame df = hiveContext.sql("SELECT * FROM REGION"); df.write() .format("com.databricks.spark.csv") .option("delimiter", "\u0001") .save("hdfs://cl1/tmp/query");Above command will save the result in HDFS under dir /tm

By default, Cloudbreak uses a built-in PostgreSQL database to persist data. For production environments, we suggest that you use an external database, an RDS served by your cloud provider. If you choose to use the default database, you should know that Cloudbreak deployer includes features for dumping and restoring built-in databases. Cloudbreak deployer uses Docker for the underlying infrastructure and uses Docker volume for storing data. There are two separate volumes:a volume called `common` for storing live dataa volume called `cbreak_dump` for database dumps You can override default live data volume any time by extending your `Profile` with the following variable: export COMMON_DB_VOL="my-live-data-volume" To create database dumps, execute the following commands: cbd db dump common cbdbcbd db dump common uaadbcbd db dump common periscopedb The dump command has an optional third parameter, the `name` of the dump. If you give your dump a name, Cloudbreak deployer will cre

SYMPTOMCREATE EXTERNAL TABLE test(id STRING,dept STRING)row format delimitedfields terminated by ','location '/user/hdfs/testdata/';ROOT CAUSEThe files under location provided while creating table are structured in following way/user/hdfs/testdata/1/test1/user/hdfs/testdata/2/test2/user/hdfs/testdata/3/test3/user/hdfs/testdata/4/test4RESOLUTIONTo make the subdirectories accessible set the following two properties before executing the create table statementset mapred.input.dir.recursive=true;set hive.mapred.supports.subdirectories=true;

PROBLEM: For an external hive table created based on hbase, if there are any missing mappings or any other issues (Syntactical), create table statement is executed successfully. You can see the table is created. However, while trying to insert data in that table, following error is seen: 08S01: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1495472057323_3947_1_00, Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.serde2.SerDeException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyStringObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.MapObjectInspector at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:800) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838) at org.apache.hadoop.hive.ql.exec.SelectOpera

PROBLEM: Ambari alert with Operation not permitted: '/var/lib/ambari-agent/tmp/curl_krb_cache' errorAlert: ERROR 2017-04-27 20:17:10,287 alert_ha_namenode_health.py:185 - [Alert] NameNode High Availability Health on taco1.example.com fails:Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/alerts/alert_ha_namenode_health.py", line 171, in execute kinit_timer_ms = kinit_timer_ms) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 106, in curl_krb_request os.chmod(curl_krb_cache_path, 0777)OSError: [Errno 1] Operation not permitted: '/var/lib/ambari-agent/tmp/curl_krb_cache' ROOT CAUSE: Ambari running as non-root user and /var/lib/ambari-agent is owned by root.SOLUTION: Move the directory to tmp location or change the ownership of the directory per non-root user.$ mv /var/lib/ambari-agent/tmp /var/lib/ambari-agent/tmp.old OR $ chown <non-root-user>:<non-root-

PROBLEM: Unable to run insert query from Ambari Hive View with Kerberos enabled. Error: Error in /var/log/ambari-server/hive-next-view/hive-view.logat java.lang.Thread.run(Thread.java:745)Caused by: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 fromorg.apache.hadoop.hive.ql.exec.tez.TezTask at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:348) at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:251) at org.apache.ambari.view.hive2.HiveJdbcConnectionDelegate.execute(HiveJdbcConnectionDelegate.java:49) at org.apache.ambari.view.hive2.actor.StatementExecutor.runStatement(StatementExecutor.java:87) at org.apache.ambari.view.hive2.actor.StatementExecutor.handleMessage(StatementExecutor.java:70) at org.apache.ambari.view.hive2.actor.HiveActor.onReceive(HiveActor.java:38) at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)Corresponding Error in RM UI Application_xxxx

PROBLEM: Installation fails with different HDP version than desired when registering a node via Ambari.For example: When registering and installing hdp 2.4.3 and some host fails on installation and tries to install hdp 2.5.3Error:2017-05-13 20:37:05,278 - Will install packages for repository version 2.4.3.0-2272017-05-13 20:37:05,278 - Repository['HDP-2.4.3.0-227'] {'append_to_file': False, 'base_url': 'http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.4.3.0/', 'action': ['create'], 'components': ['HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-2.4.3.0-227', 'mirror_list': None}2017-05-13 20:37:05,317 - File['/etc/yum.repos.d/HDP-2.4.3.0-227.repo'] {'content': '[HDP-2.4.3.0-227]\nname=HDP-2.4.3.0-227\nbaseurl=http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.4.3.0/\n\npath=/\nenabled=1\ngpgcheck=0'

The latest and greatest version of Hortonworks HDB i.e version 2.2.0.0 addresses many of the concerns from customers/prospects who have been considering Hortonworks HDB for their production data analytics workloads.Below is the list of major improvements available in Hortonworks HDB 2.2.0.0 over its predecessor. Ranger support (in BETA mode)Lack of support for Ranger has been a deal breaker in field in past. Hortonworks HDB 2.2.0.0 is step in right direction and includs Beta support for Apache Ranger. Hortonworks HDB 2.2.0.0 now include a RESTful service that provides integration between HDB and Ranger policy management. Ranger integration enables you to use Apache Ranger to authorize user access to Hortonworks HDB resources. Using Ranger enables you to manage all of your Hadoop components’ authorization policies using the same user interface, policy store, and auditing stores. For more details on how to use Ranger for authorization in Hortonworks HDB 2.2.0.0, please refer to http://h

ISSUE : Simple select query against an ORC table without limit clause is failing with the below exception.2017-06-26 16:00:35,228 ERROR [main]: CliDriver (SessionState.java:printError(993)) - Failed with exception java.io.IOException:java.lang.RuntimeException: serious problem java.io.IOException: java.lang.RuntimeException: serious problem at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:520) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:427) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1765) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:237) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:169) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:380) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:740) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:685) at

Apache NiFi has grown tremendously over the past 2 and a half years since it was open sourced. The community is continuously thinking of, implementing, and contributing amazing new features. The newly released version 1.2.0 of NiFi is no exception. One of the most exciting features of this new release is the QueryRecord Processor and the Record Reader and Record Writer components that go along with it. If you aren't familiar with those components, there's a blog post that explains how those work.This new Processor, powered by Apache Calcite, allows users to write SQL SELECT statements to run over their data as it streams through the system. Each FlowFile in NiFi can be treated as if it were a database table named FLOWFILE. These SQL queries can be used to filter specific columns or fields from your data, rename those columns/fields, filter rows, perform calculations and aggregations on the data, route the data, or whatever else you may want to use SQL for. All from the comfy confines o

Issue:Supposedly if the table has large no of partitions in the order of 2800 or above and you wanted the API call to return all the partitions of the table in JSON format then you'll end up with the above errorAPI:curl --negotiate -u: http://mrt1.openstacklocal:50111/templeton/v1/ddl/database/<db name>/table/<table name>/partition > /tmp/partition_jsonResolution:The error message is not because of the no. of partitions in the table. JSON responses are limited to 1MB in size. Responses over this limit must be stored into HDFS using provided options instead of being directly returned. Workaround:You can use HCatalog, if your DDL command return results greater than 1MB/usr/hdp/<hdp version>/hive/bin/hcat -e "use <dbname>; show partitions <tablename>; " -D hive.ddl.output.format=json -D hive.format=json -D hive.metastore.token.signature=hcat -D proxy.user.name=<username>Reference:https://cwiki.apache.org/confluence/display/Hive/WebHCat+UsingWebHCat

ISSUE : Did OS Patch on all their hosts on the cluster and after that Datanodes are started but none of them are alive in the HDFS Dashboard and all the blocks are showing missingLinux cl-master1 2.6.32-696.3.2.el6.x86_64 #1 SMP Wed Jun 7 11:51:39 EDT 2017 x86_64 x86_64 x86_64 GNU/LinuxRESOLUTION : This is a Kernel Bug https://issues.apache.org/jira/browse/HDFS-12029 Please put the Namenode in Safe mode using the below commandHdfs dfsadmin –safemode enterThen add “-Xss2m” on in between the below lines/usr/hdp/2.6.0.3-8/hadoop-hdfs/bin/hdfs.distro on all the Datanodesexec "$JSVC" \-Xss2m \-Dproc_$COMMAND -outfile "$JSVC_OUTFILE" \ -errfile "$JSVC_ERRFILE" \ -pidfile "$HADOOP_SECURE_DN_PID" \ -nodetach \ -user "$HADOOP_SECURE_DN_USER" \ -cp "$CLASSPATH" \ $JAVA_HEAP_MAX $HADOOP_OPTS \ org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter "$@"Please restart the data nodes from Ambari and leave the safe mode o

I'd like to announce the availability of HDP 2.5.6.0. This release continues from HDP 2.5.5.0, adding about 60 Apache JIRAs and includes Knox and YARN security fixes. There are no features introduced with this maintenance release.The release details can be found in the HDP 2.5.6.0 Release Notes.Supported OS platforms are the usual CentOS 6/7, SLES 11/12, Debian 7 and Ubuntu 12/14.This will soon be available on Azure HDInsight as soon as our testing is unblocked due to some other issues.Thanks to everyone who lent a hand in this release, including the Apache community.

import refrom pyspark.sql import RowAPACHE_ACCESS_LOG_PATTERN = '^(\S+) (\S+) (\S+) \[([\w:/]+\s[+\-]\d{4})\] "(\S+) (\S+) (\S+)" (\d{3}) (\d+)'# Returns a dictionary containing the parts of the Apache Access Log.def parse_apache_log_line(logline): match = re.search(APACHE_ACCESS_LOG_PATTERN, logline) if match is None: # Optionally, you can change this to just ignore if each line of data is not critical. # For this example, we want to ensure that the format is consistent. raise Exception("Invalid logline: %s" % logline) return Row( ipAddress = match.group(1), clientIdentd = match.group(2), userId = match.group(3), dateTime = match.group(4), method = match.group(5), endpoint = match.group(6), protocol = match.group(7), responseCode = int(match.group(8)), contentSize = long(match.group(9)))log_files = "hdfs://dataset/apache_logs/"raw_log_files = sc.textFile(log_files)r

Hadoop is a marvelous ecosystem, so many improvement in the last few years and so many way to extract insight from Big Data now. There is also an other sight that Hadoop can be complex to manage especially with new release, projects, versions, features and security. This is one of the reason why it is highly recommended to do DevOps. DevOps helps to enforce testing, to ensure smooth transition between environment and so to improve overall quality of any project running on the platform.In this presentation, I tried to explain the basic concept of DevOps and how Hadoop integrate nicely with tools like Jenkins, Ansible, Git, Maven and how it is amazing to use the advantage of a Big Data platform to monitor each component, projects and quality (delivery, performance & SLA, logs).

Problem:In HDP 2.3.4 and Ambari 2.4.2, Storm nimbus was not coming up:2017-05-30 14:35:18.756 o.a.t.s.TThreadPoolServer [ERROR] Error occurred during processing of message. java.lang.RuntimeException: No nimbus leader participant host found, have you started your nimbus hosts??Following is the error from Nimbus log:2017-05-30 14:35:18.750 b.s.d.nimbus [WARN] principal: storm@EXAMPLE.COM is trying to impersonate principal: ambari-server@EXAMPLE.COM 2017-05-30 14:35:18.751 b.s.d.nimbus [WARN] impersonation attempt but nimbus.impersonation.authorizer has no authorizer configured. potential security risk, please see SECURITY.MD to learn how to configure impersonation authorizer. 2017-05-30 14:35:18.756 o.a.t.s.TThreadPoolServer [ERROR] Error occurred during processing of message. java.lang.RuntimeException: No nimbus leader participant host found, have you started your nimbus hosts?Cause:Property ''nimbus.impersonation.authorizer" was set to 'org.apache.storm.security.auth.authorizer.Imper

Problem Description:Unable to start storm nimbus from Ambari, throwing following error:raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh ln -s /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common-*.jar /usr/hdp/current/storm-nimbus/lib/ambari-metrics-storm-sink.jar' returned 1. ln: target `/usr/hdp/current/storm-nimbus/lib/ambari-metrics-storm-sink.jar' is not a directoryCause: Two legacy jars under storm lib:# ls -ld /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common* -rw-r--r-- 1 root root 1937430 May 11 15:57 /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common-2.4.1.0.22.jar -rw-r--r-- 1 root root 1933392 Nov 23 2016 /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common-2.4.2.0.136.jarSolution:As the Ambari version was 2.4.2 so, moved ambari-metrics-storm-sink-legacy-with-common-2.4.1.0.22.jar to a different location and started storm nimbus.

Following was the error in supervisor log:2017-06-09 14:40:18.348 o.a.s.d.supervisor [ERROR] Error on initialization of server mk-supervisor java.lang.RuntimeException: java.lang.ClassNotFoundException: backtype.storm.generated.LSSupervisorId at org.apache.storm.utils.LocalState.deserialize(LocalState.java:83) at org.apache.storm.utils.LocalState.get(LocalState.java:130) at org.apache.storm.local_state$ls_supervisor_id.invoke(local_state.clj:61) at org.apache.storm.daemon.supervisor$standalone_supervisor$reify__7977.prepare(supervisor.clj:1216) at org.apache.storm.daemon.supervisor$fn__7833$exec_fn__3537__auto____7834.invoke(supervisor.clj:766) at clojure.lang.AFn.applyToHelper(AFn.java:160) at clojure.lang.AFn.applyTo(AFn.java:144) at clojure.core$apply.invoke(core.clj:630) at org.apache.storm.daemon.supervisor$fn__7833$mk_supervisor__7878.doInvoke(supervisor.clj:764) at clojure.lang.RestFn.invoke(RestFn.java:436) at org.apache.storm.daemon.supervisor$_launch.invoke(supervisor.clj:120

select * from t1 where col1 = 'aaa' limit 2;If you see such scenario, make sure to verify the no. of files in each partitions of the table. It is possible that if there are too many small files in the order of KBs under each partition then most of the time is spent in just opening the file in ORC. Check if it is expected to have so many files in each partition or if it is possible to merge them before loading data.These are the parameters to consider hive.merge.tezfiles=true hive.merge.mapredfiles=truehive.merge.mapfiles=true hive.merge.orcfile.stripe.level=true In such situation, checking the no of files in the partition should be the first protocol & verifying anything else should be only secondary.

IBM Spectrum Scale 4.2.3 has been certified with the Hortonworks Data Platform (HDP) 2.6 / Ambari 2.5 on IBM Power Systems. IBM and Hortonworks collaborated on an optimized and integrated solution that was validated against a comprehensive suite of integration test cases across the full stack of HDP components and Ambari. Testing covered secure and non-secure scenarios with Accumulo, Atlas, Falcon, Flume, Hbase, HDFS, Hive, HiveServer2, Kafka, Knox, Mahout, Map Reduce, Oozie, Phoenix, Pig, Spark, Sqoop, Storm, Tez, Yarn, Zeppelin, and Zookeeper. This certification is for Spectrum Scale software and hence applies to all deployment models of Spectrum Scale, including Elastic Storage Server (ESS). Further, this certification includes a paper certification for Hortonworks Data Flow (HDF) V3.0 use with IBM Spectrum Scale. IBM’s Power platform is already certified to run HDP and offers 3x price performance compared with x86.IBM ESS (pre-integrated system powered by IBM Spectrum Scale) includ

Problem Description:On deleting few kafka topics and creating same topics again there Not Leader for this partition exception while Producing/Consuming and was reading old meta data. Getting Below error:17/03/10 10:45:35 ERROR ApplicationMaster: User class threw exception: org.apache.spark.SparkException: org.apache.spark.SparkException: Couldn't find leaders for Set([akrnohij-wng-fp9,47], [akrnohij-wng-fp4,224], [akrnohij-wng-fp11,172], [akrnohij-wng-fp1,84], [akrnohij-wng-fp10,117], [akrnohij-wng-fp10,168], [akrnohij-wng-fp4,176], [akrnohij-wng-fp1,136], [akrnohij-wng-fp9,167], [akrnohij-wng-fp2,174], [akrnohij-wng-fp11,74], [akrnohij-wng-fp3,189], [akrnohij-wng-fp11,200], [akrnohij-wng-fp11,168], [akrnohij-wng-fp4,149], [akrnohij-wng-fp7,127], [akrnohij-wng-fp6,39], [akrnohij-wng-fp10,133], [akrnohij-wng-fp9,171], [akrnohij-wng-fp5,175], [akrnohij-wng-fp7,181] Cause:The zookeeper ACL for state znode was not correct : getAcl /brokers/topics/akrnohij-wng-fp4/partitions/135/state ‘aut

SYMPTOM: While performing HDP upgrade [HDP-2.3.0 upgrade to HDP-2.5.3] there was error for which we had to start ambari server. But ambari server startup failed with below error - ERROR: Caused by: Exception [EclipseLink-116] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DescriptorException Exception Description: No conversion value provided for the value [UPGRADED] in field [host_version.state]. Mapping: org.eclipse.persistence.mappings.DirectToFieldMapping[state-->host_version.state] Descriptor: RelationalDescriptor(org.apache.ambari.server.orm.entities.HostVersionEntity --> [DatabaseTable(host_version)]) at org.eclipse.persistence.exceptions.DescriptorException.noFieldValueConversionToAttributeValueProvided(DescriptorException.java:1066) at org.eclipse.persistence.mappings.converters.ObjectTypeConverter.convertDataValueToObjectValue(ObjectTypeConverter.java:226) at org.eclipse.persistence.mappings.converters.EnumTypeConverter.conv

SYMPTOM: During upgrade from HDP 2.4.3 to HDP2.6.0 the express upgrade failed on the step "Yarn NodeMangers / Restarting NodeManger."ERROR: 2017-06-2215:02:22,791 INFO recovery.ZKRMStateStore(ZKRMStateStore.java:runWithRetries(1227)) - Exception while executing a ZKoperation. org.apache.zookeeper.KeeperException$AuthFailedException:KeeperErrorCode = AuthFailed for /rmstore atorg.apache.zookeeper.KeeperException.create(KeeperException.java:123) atorg.apache.zookeeper.KeeperException.create(KeeperException.java:51) atorg.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) ROOT CAUSE: The issue was with property Ambari-Yarn-Configs-"yarn-resourcemanager-zk-address" was set to localhost:2181 RESOLUTION: Modified address to "{zk-server-hostname}:2181" resolved the issue. After which node managers were able to start successfully.