Community Articles

SYMPTOM: user is frequently seeing following exceptions: org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept. org.apache.kafka.common.errors.RecordTooLargeException: The message is 5745799 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration. ROOT CAUSE: This is happening due to max byte settings at producer level and broker level Broker side: message.max.bytes - this is the largest size of the message that can be received by the broker from a producer Producer side: max.request.size is a limit to send the larger message. WORKAROUND: NA RESOLUTION: message.max.bytes by default is 1M (1000012 bytes) for kafka 0.10.0. if you need to publish larger messages, you will need to adjust that on the brokers and then restart them.to avoid exception at producer side you need to increase max.request.size to send the larger message. P

Failed to list function from phoenix sqlline.pyCommand run : select * from SYSTEM.FUNCTION;Error: ERROR 604 (42P00): Syntax error. Mismatched input. Expecting "NAME", got "FUNCTION" at line 1, column 22. (state=42P00,code=604)org.apache.phoenix.exception.PhoenixParserException: ERROR 604 (42P00): Syntax error. Mismatched input. Expecting "NAME", got "FUNCTION" at line 1, column 22.at org.apache.phoenix.exception.PhoenixParserException.newException(PhoenixParserException.java:33)at org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:111)at org.apache.phoenix.jdbc.PhoenixStatement$PhoenixStatementParser.parseStatement(PhoenixStatement.java:1280)at org.apache.phoenix.jdbc.PhoenixStatement.parseStatement(PhoenixStatement.java:1363)at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1434)at sqlline.Commands.execute(Commands.java:822)at sqlline.Commands.sql(Commands.java:732)at sqlline.SqlLine.dispatch(SqlLine.java:808)at sqlline.SqlLine.begin(SqlLine.java:

This article gives an 8 step procedure to recover the accidentally deleted HDFS file which is not even available in the Trash. Use these procedure with caution on production system. I strongly suggest to take supports help if you are not familiar with the internal workings.IMPORTANT :Please make sure NameNode is stopped immediatly after file deletion otherwise it's hard to recover as namenode already send out block deletion request to datanode causing physical block might get deleted by datanodeAlways take backup of "fsimage" and "edits" file before you perform this step.1).Lets create a sample file in HDFSsu - hdfshadoop fs -put /etc/passwd /tmp[hdfs@hdpnode1 current]$ hadoop fs -ls /tmp/passwd -rw-r--r-- 3 hdfs hdfs 2370 2016-04-06 12:45 /tmp/passwd2).Delete the file and also make sure that the file doesnt go to trash.hadoop fs -rmr -skipTrash /tmp/passwd3).Stop the HDFS service via Ambari.4). Go to Namenode metadata directory(dfs.namenode.name.dir).In my lab environment it is set to

OBJECTIVE:Resolve issues with some lightweight LDAP services such as the HDP Demo LDAP ProviderOVERVIEW:Some LDAP services do not properly support paging for LDAP query results. In order to support these LDAP services, paging of results needs to be disabled in the LDAP provider properties for the NiFi Registry serviceSYMPTOM:The NiFi Registry does not respond or times out, and the following error is seen repeatedly in the NiFi Registry log (nifi-registry-app.log):2018-09-01 01:01:05,189 INFO [main] o.s.l.c.AbstractRequestControlDirContextProcessor No matching response control found - looking for 'class javax.naming.ldap.PagedResultsResponseControl2018-09-01 01:01:05,274 INFO [main] o.s.l.c.AbstractRequestControlDirContextProcessor No matching response control found - looking for 'class javax.naming.ldap.PagedResultsResponseControl2018-09-01 01:01:05,359 INFO [main] o.s.l.c.AbstractRequestControlDirContextProcessor No matching response control found - looking for 'class javax.naming.lda

IoT Edge Processing with Apache NiFi and MiniFi and Multiple Deep Learning Libraries SeriesFor: https://conferences.oreilly.com/strata/strata-ny/public/schedule/detail/68140See Part 1: https://community.hortonworks.com/articles/215079/iot-edge-processing-with-deep-learning-on-hdf-32-a.htmlSee Part 2: https://community.hortonworks.com/articles/215258/iot-edge-processing-with-deep-learning-on-hdf-32-a-1.htmlSee Part 3: https://community.hortonworks.com/articles/215271/iot-edge-processing-with-deep-learning-on-hdf-32-a-2.htmlYou will notice a bit of a travel theme this article, it's because some of the images and work was done while on various holidays in August and September.Deep LearningWe are running TensorFlow 1.10, Apache MXNet 1.3, NCSDK 2.05 and Neural Compute Application Zoo (NC App Zoo).Device Type 1: Plain Raspberry Pi(Found some old Kodak slides...)Main things to do is upgrade to Python 3.6, upgrade Raspberry PI to Stretch, upgrade libraries and a few reboots.Install OpenCV (

IntroductionWhen the NiFi bootstrap starts or stops NiFi, or detects that it has died unexpectedly, it is able to notify configured recipients. Currently, the only mechanism supplied is to send an e-mail notification.Prerequisite1) Assuming you already have HDF-2.x Installed, Ambari and NiFi is up and running. If not, I would recommend "Ease of Deployment" section of this article to install it [You can also follow this article for Automated installation of HDF cluster or refer hortonworks.com for detailed steps]Configuring NiFi property files in Ambari1) To setup email notifications we have to update only two configurations file bootstrap.conf and bootstrap-notification-services.xml 2) We have to update appropriate properties in Ambari to configure it, first we have to edit Template for bootstrap.conf to update below properties. Uncomment below lines in the properties file:nifi.start.notification.services=email-notification nifi.stop.notification.services=email-notificationnifi.dead.n

Summary:The release of HDF 3.3 brings about a significant number of improvements in HDF. This article shows how you can use ambari-bootstrap project to easily generate a blueprint and deploy either HDF only clusters or combined HDP/HDF clusters in 5 easy steps. To quickly setup a single node setup, prebuilt AMIs are available for AWS as well as a script that automates these steps, so you can deploy the cluster in a few commands.Steps for each of the below option are described in this article:A. Single-node prebuilt AMIs on AWSB. Single-node fresh installsC. Multi-node fresh installsA. Single-node prebuilt AMI on AWS:Steps to launch the AMI1. Launch Amazon AWS console page in your browser by clicking here and sign in with your credentials. Once signed in, you can close this browser tab.2. Select the AMI from ‘N. California’ region by clicking one of the below options To spin up HDP 3.1/HDF 3.3, click hereTo spin up HDF 3.3 only cluster, click hereNow choose instance type: select ‘m4.2xl

Issue Description : Unable to Register new Version in Ambari-2.6.2.x versions using Redhat Satellite/spacewalk, The Save button is disabled or grayed out the Javascript logic of validating the URL fails if the Redhat satellite/spacewalk is ticked and save button remains disabled Root cause : https://issues.apache.org/jira/browse/AMBARI-22952 Solution : 1) Use javascript Workaround Go to Ambari-Server node, go to folder(for ambari-2.6.2.2 folder name will be ADMIN_VIEW\{2.6.2.2\} : cd /var/lib/ambari-server/resources/views/work/ADMIN_VIEW{2.6.2.0}/scripts Back up the current main.js cp main.js /tmp/main.js vi main.js find the line : $scope.isSaveButtonDisabled = function() { Change this functions return type from return !(enabled && $scope.validBaseUrlsExist()); to return !($scope.useRedhatSatellite || (enabled && $scope.validBaseUrlsExist())); Save it and Try again after Hard Reloading the Browser. There is an another option of adding the same via CU

This is a short how-to leveraging Zeppelin and Solr's native SQL capabilities to query the Ranger audit logs in real time. The capability to query Ranger audits has been in existence for quite a while and there are multiple articles available demonstrating how to apply a Hive external table on top of Ranger audits stored in HDFS. This article demonstrates how to leverage Zeppelin and Solr SQL to query Solr in real time without additional step of creating an external table on top of the HDFS audit.First thing you need is access to your Solr instance. I'm using the default instance packaged with Ambari infra. The Solr admin UI is available at the following address: http://{ambari-infra-ip}:8886. In the UI, you can issue arbitrary queries using standard Solr syntax. I am new to Solr and found the query syntax cumbersome. Instead, I decided to leverage Solr SQL, available as of version 6. HDP 3.0 ships with Solr 7.3.The next step is to set up Zeppelin interpreter for Solr via JDBC. Steps f

Setting up High Availability for Oozie server. Prerequisite: - To have multiple Oozie instances we need to have either MYSql/Postgres/Oracle DB instead of the Dubey Database We need to have HA Proxy /F5 Setup for Load Balancing. For Production I would suggest to get the F5 Load Balancer which should be managed by the Network Team. Steps to setup Multiple oozie servers followed by HAProxy setup. Please find the attached Steps to add Multiple Oozie Instances https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-operations/content/adding_an_oozie_server_component.html). Setting up the ZK Quorum and oozie_base_url Inorder to enable the HA feature of oozie we need to use zookeeper quorum and expose the Url of the Proxy server. Oozie-site.xml ozie.zookeeper.connection.string provide the Zookeepr quorum in my lab e.g. "hwc3206-node2.hogwarts-labs.com:2181,hwc3206-node3.hogwarts-labs.com:2181,hwc3206-node4.hogwarts-labs.com:2181" oozie.services.ext add th

Short Description:Often times there will be a need to ingest binary files to Hadoop (like PDF, JPG, PNG) where you will want to store them in HBase directly and not on HDFS itself. This article describes an example how this could be achieved. ArticleThe maximum number of files in HDFS depends on the amount of memory available for the NameNode.Each file object and each block object takes about 150 bytes of the memory. For example, if you have 10 million files and each file has 1 one block each, then you would need about 3GB of memory for the NameNode. This could pose a problem if you would like to store trillions of files, where you will run out of RAM on the NameNode trying to track all of these files. One option to resolve this issue is to store your blobs in an Object store. There are many to choose from, especially if you use your favorite cloud provider. Another alternative would be to look at Ozone: https://hortonworks.com/blog/ozone-object-store-hdfs/Below is an example how you c

During launch of HDP or HDF on azure via cloudbreak, if the following provisioning error is thrown (Check cloudbreak logs):log:55 INFO  c.m.a.m.r.Deployments checkExistence - [owner:xxxxx] [type:STACK] [id:2] [name:sparky] [flow:xxx] [tracking:] <-- 404 Not Found https://management.azure.com/subscriptions/xxxxxx/resourcegroups/spark. (104 ms, 92-byte body)/cbreak_cloudbreak_1 | 2018-09-05 14:25:22,882 [reactorDispatcher-24] launch:136 ERROR c.s.c.c.a.AzureResourceConnector - [owner:xxxxxx] [type:STACK] [id:2] [name:sparky] [flow:xxxxxx] [tracking:] Provisioning error:This means the instance type selected is not available within the region. Please change region where instance is available or change to instance type which is available within region.

Exploring Apache Flink with HDPApache Flink is an open source platform for distributed stream and batch data processing. More details on Flink and how it is being used in the industry today available here: http://flink-forward.org/?post_type=session. There are a few ways you can explore Flink on HDP 2.3:1. Compilation on HDP 2.3.2To compile Flink from source on HDP 2.3 you can use these commands:curl -o /etc/yum.repos.d/epel-apache-maven.repo https://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repoyum -y install apache-maven-3.2*git clone https://github.com/apache/flink.gitcd flinkmvn clean install -DskipTests -Dhadoop.version=2.7.1.2.3.2.0-2950 -Pvendor-reposNote that with this option I ran into a classpath bug and raised it here: https://issues.apache.org/jira/browse/FLINK-30322. Run using precompiledtarballwget http://www.gtlib.gatech.edu/pub/apache/flink/flink-0.9.1/flink-0.9.1-bin-hadoop27.tgztar xvzf flink-0.9.1-bin-hadoop27.tgzcd flink-0.9.1export HADOOP_

If you have started using Hive LLAP, you would have noticed that by default its configured to use log4j2.Default configuration makes use of advanced features from log4j2 like Rolling Over logs based on time interval and size.With time, a lot of old log files would have accumulated and typically you would compress those files manually or add additional jars and change configuration when using log4j1 to achieve the sameWith log4j2, a simple change in configuration can ensure that every time a log file is rolled over, it gets compressed for optimal use of storage space.Default configuration:To automatically compress the rolled over log files, update the highlighted line to:appender.DRFA.filePattern = ${sys:hive.log.dir}/${sys:hive.log.file}.%d{yyyy-MM-dd}-%i.gz-%i will ensure that in a rare scenario when there has been increased logging and the threshold size can be been reached more than once in the specified interval, the previously rolled over file won't get over written..gz will ensur

Test if I can publish an article. This errors out for me, and I want to know if this works now.

Overview Following my series on news author personality detection, let's dive into the streaming analytics side of things, and continue building our data science and engineering platform. This tutorial does assume that you completed part 1 of the series. After completing this part, your architecture will look like this: This tutorial will be divided in sections: Section 1: Create a custom UDF for SAM Section 2: Create a SAM application to sink to Druid Section 3: Monitor your streaming application Section 4: Create a simple superset visualization Note: Creating a custom UDF is a purely academical exercise, as is the use of SAM for sinking into Druid, considering I could have connected Kafka to Druid directly. However, one of the goal of this series is to learn as much as possible about the Hortonworks stack. Therefore, custom UDF and SAM is on the menu. Section 1: Create a custom UDF for SAM Step 1: Download the structure for your custom UDF Using the example giv

We recently had a customer who was doing stress testing on a very small cluster with just four Datanodes. Their tests included simulated crashes and sudden power-offs. They were getting recurring client errors of hdfs.DFSClient: Exception in createBlockOutputStream with various log messages including:Client:EOFException: Premature EOF: no length prefix availableIOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to tryDatanode:ReplicaAlreadyExistsException: Block <number> already exists in state FINALIZED and thus cannot be createdReplicaNotFoundException: Cannot append to a non-existent replicaFsDatasetImpl (FsDatasetImpl.java:recoverAppend(1053)) - Recover failed append to BP-<number>ReplicaNotFoundException: Cannot append to a replica with unexpeted generation stampNamenode:BlockPlacementPolicy - Failed to place enough replicas, still in need of 1 to reach 3The reason for these errors has to do with: HDFS

This Article is intended to Ambari Administrators who will be managing ambari .And for the steps mentioned below, you need access to ambari Server and ambari database.For Ambari versions below ambari-2.6.2.2 refer to comments here : https://community.hortonworks.com/questions/449/how-to-reset-ambari-admin-password.htmlThis article is for Ambari-2.7.0.0 and above.Problem Statement: you have changed the password of the admin user in UI and now you forgot the password. you want to reset the admin password.Steps1) Stop ambari serverambari-server stop2) login to your database (default password will be bigdata) psql -U ambari ambarior ( for mysql)mysql -U ambari -D ambari3) execute the command : update user_authentication set authentication_key='538916f8943ec225d97a9a86a2c6ec0818c1cd400e09e03b660fdaaec4af29ddbb6f2b1033b81b00' where user_id in (select user_id from users where user_name='admin');4) exit the database and restart ambari-serverambari-server startNow password is reset to default

Ozone is an Object store for Hadoop. It is a redundant, distributed object store built by leveraging primitives present in HDFS. Below are some key features of ozone: A Hadoop compatible file system called Ozone File system that allows programs like Hive or Spark to run against Ozone without any modifications.Ozone supports RPC and REST API for accessing the store.Built to support billions of keys in distributed environment.Ozone can run concurrently with HDFS.Like many other object stores, Ozone has a notion of volume. Only Administrators can create Volumes. Users create buckets in the volumes. To store data inside a bucket, users create keys.An ozone file system allows other Hadoop ecosystem applications like Hive and Spark to use ozone. Once a bucket is created, it is trivial to create an ozone file system.A 10-thousand foot view of Ozone OzoneManager (Om) acts as namespace manager. All ozone entities like volumes, buckets and keys are managed by Om. Om talks to an independent

IoT Edge Processing with Apache NiFi and MiniFi and Multiple Deep Learning Libraries SeriesFor: https://conferences.oreilly.com/strata/strata-ny/public/schedule/detail/68140See Part 1: https://community.hortonworks.com/articles/215079/iot-edge-processing-with-deep-learning-on-hdf-32-a.htmlSee Part 2: https://community.hortonworks.com/articles/215258/iot-edge-processing-with-deep-learning-on-hdf-32-a-1.htmlHive - SQL - IoT Data StorageIn this section, we will focus on converting JSON to AVRO to Apache ORC and storage options in Apache Hive 3. I am doing two styles of storage for one of the tables, rainbow. I am storing ORC files with an external table as well as using the Streaming API to store into an ACID table.NiFi - SQL - On Streams - CalciteSELECT *FROM FLOWFILEWHERE CAST(memory AS FLOAT) > 0 SELECT *FROM FLOWFILEWHERE CAST(tempf AS FLOAT) > 65I check the flows as they are ingested real-time and filter based on conditions such as memory or temperature. This makes for so

IoT Edge Processing with Deep Learning on HDF 3.2 and HDP 3.0 - Part 2For: https://conferences.oreilly.com/strata/strata-ny/public/schedule/detail/68140See Pre-Work: https://community.hortonworks.com/articles/203638/ingesting-multiple-iot-devices-with-apache-nifi-17.html See Part 1: https://community.hortonworks.com/articles/215079/iot-edge-processing-with-deep-learning-on-hdf-32-a.htmlStep By Step ProcessingStep 1: Install Apache NiFi (One or More Nodes or clusters) Choose: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.2.0/installing-hdf/content/install-ambari.html or docker pull hortonworks/nifi Apache NiFi Configuration for IoT https://community.hortonworks.com/articles/67756/ingesting-log-data-using-minifi-nifi.html You will need to set: nifi.remote.input.host and nifi.remote.input.socket.port in the conf/nifi.properties or Ambari settings. Step 2: Install Apache NiFi - MiniFi on Your Device(s) Download MiniFi (https://nifi.apache.org/minifi/download.html) Yo

When you apply aggregate function on a column whose data type is "void" you get the above error.How you get "void" as datatype?When you create a table from another table(CTAS) & hardcode NULL as column values in the base table, you get the datatype as "void" for those columns. Consider the following example. This is a bug https://issues.apache.org/jira/browse/HIVE-11217hive> create table test1 as select null as col1, null as col2;Query ID = hive_20161227141826_d746f7f8-f867-4376-8b58-2e03e63c9763Total jobs = 1Launching Job 1 out of 1Status: Running (Executing on YARN cluster with App id application_1481936603759_0026)-------------------------------------------------------------------------------- VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED--------------------------------------------------------------------------------Map 1 .......... SUCCEEDED 1 1 0 0 0 0------------------------------------------------

Apache Ambari is a tool for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides central management for starting, stopping, and reconfiguring Hadoop services across the entire cluster.Ambari-server Sends Commands to Ambari-agent for such operations in JSON format which is saved in /var/lib/ambari-agent/data folder. these json (command-*.json, status_command.json) Files may contain usernames and passwords which is required for few services at startupCurrently the command JSON files are stored on disk in /var/lib/ambari-agent/data. These files may be cleared out over time, but there is a need to have them removed as soon as they are no longer needed.Ambari-2.7.0 Provides a provision for deleting the command JSON files once Operation is successful to reduce the risk of leaking sensitive data stored at plaintext in the command JSON files.More Details about this can be found at the Apache JIRA: https://issues.apache.org/jira/browse/AMBARI-24228How To Enable this fe

Objective This tutorial is intended to walk you through the process of using the GetTwitter, UpdateAttribute, ReplaceText and PutElasticsearch processors in Apache NiFi to modify Twitter JSON data before sending it to Elasticsearch. This tutorial is the first part of a two part series. In this part of the series, we will create a single data flow that adds an additional field to the JSON data called twitterFilterAttribute using the ReplaceText processor. This will allow us to query Elasticsearch using a fielded query like q=twitterFilterAttribute:elasticsearch. The second part of the series will build on this example to create a process group with two GetTwitter feeds: one with elasticsearch term filter and the other with a solr term filter. Admittedly this is a contrived example. However, the concept is fundamentally useful across a variety of NiFi use cases. NOTE: The only required software components are NiFi and Elasticsearch which can be run in just about any linux environment

Why to use spark-submit if spark-shell is there ?1. Spark shell spawns executors on random nodes and hence chances of data Locality will be very less. 2. Spark-submit based on the Nodes where the data is saved spawns the excutors hence spark-submit will be more performant as compared to spark-shell.3. Spark shell is good in situations when data exploration needs to be done as it provides a interactive CLI to run your code.