Resources

To set up health_percent of LLAP, do the following:    On the Hiveserver2Interactive server nodes, edit /usr/hdp/<hdp-version>/hive/scripts/llap/yarn/package.py​Example: /usr/hdp/3.1.0.224-3/hive/scripts/llap/yarn/package.pyThe --health-percent defaults to 80. Change this to desired-number. Move the following to a temporary backup location: a. /usr/hdp/<hdp-version>/hive/scripts/llap/yarn/package.pyc b. /usr/hdp/<hdp-version>/hive/scripts/llap/yarn/package.pyo ​ Restart Hive.

Note: Cloudera does not support antivirus software of any kind.   This article contains general recommendations for excluding ZooKeeper components and directories from antivirus scans and monitoring.   The three primary locations you will want to exclude from antivirus are: Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the antivirus holds up writes. Log directories: These are write-heavy. Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the antivirus holds up writes.   Note: ZooKeeper has a user-configurable data directory. I recommend you exclude it. This directory can be found by running the following command: # grep dataDir /etc/zookeeper/conf/zoo.cfg Consider excluding the following directories and all of their subdirectories

Note: Cloudera does not support antivirus software of any kind.   This article contains general recommendations for excluding MapReduce components and directories from antivirus scans and monitoring.   The three primary locations you will want to exclude from antivirus are: Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the antivirus holds up writes. Log directories: These are write-heavy. Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the antivirus holds up writes. Note: Some directories in MapReduce are user-configurable. I recommend you exclude them. These properties can be found in Ambari > YARN > Configs > Advanced, and this one in particular should be excluded:     mapreduce.jobhistory.recovery.store.leveldb.path &nbs

Note: Cloudera does not support antivirus software of any kind.   This article contains general recommendations for excluding YARN components and directories from antivirus scans and monitoring.   The three primary locations you will want to exclude from antivirus are: Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the antivirus holds up writes. Log directories: These are write-heavy. Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the antivirus holds up writes. Note: The directories YARN uses are user-configurable. I recommend you exclude them. These properties can be found in Ambari > YARN > Configs > Advanced:   yarn.nodemanager.local-dirsyarn.nodemanager.log-dirsyarn.nodemanager.recovery.diryarn.timeline-servic

Note: Cloudera does not support antivirus software of any kind.   This article contains general recommendations for excluding HDFS components and directories from antivirus scans and monitoring.   The three primary locations you will want to exclude from antivirus are: Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the antivirus holds up writes. Log directories: These are write-heavy. Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the antivirus holds up writes. Note: The directories in HDFS are user-configurable. I recommend you exclude these, especially the data directory for the DataNode and the meta directories for the NameNode and JournalNode. These details can be found in the “hdfs-site.xml” file:       # grep -A

Note: Cloudera does not support antivirus software of any kind.   This article contains general recommendations for excluding Ambari components and directories from antivirus scans and monitoring.   The three primary locations you will want to exclude from antivirus are: Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the AV holds up writes. Log directories: These are write-heavy. Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the AV holds up writes.   Note: Ambari has a special requirement in the form of a user-configurable database. I recommend you exclude this database. However, the details of this database are set on installation; the database may be colocated with ambari-server, or on a remote host. Consult with your database adm

ExtractText NiFi Custom Processor Powered by Apache Tika Apache Tika is amazing, it is very easy to use it to analyze file and then to extract text with it. Apache Tika uses other powerful Apache projects like Apache PDFBox and Apache POI.Example Usage Feed in documents, I use my LinkProcessor which grabs links from a website and returns a JSON List.Split the resulting JSON list into individual JSON rows with SplitJSON.EvaluateJSONPath to extract just the URLs.InvokeHTTP to do a GET on that parsed URL.RouteOnAttribute to only process file types I am interested in like Microsoft Word.The new ExtractTextProcessor to extract the text of the document.Then we save the text as a file in some data store, perhaps HDFS.If you have a directory of files, you can just use GetFile to ingest them en masse.LinkProcessor (https://github.com/tspannhw/linkextractorprocessor)URL: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/index.htmlThis is an example of a URL that I want to grab all the do

In Part 1, I shared an alternative architecture for Prometheus that increases scalability and flexibility of time series metric monitoring. In part 2, I will walk through an extension of the architecture that unites metric and log data with a unified, scalable data pipeline. Part 1 Architecture Part 2 Architecture By adding log collection agents (e.g. MiNiFi) and a search cluster (e.g. Solr), the solution architecture can be extended to support log data in addition to time series metrics. This has the advantage of reducing duplicated infrastructure components for a more efficient and supportable solution.   Additional NiFi processors can be added to the flow for pre-processing (e.g. filtering, routing, scoring) the incoming data (e.g. ERROR vs INFO messages). Rulesets (e.g. Drools) from expert systems can be embedded directly into the flow, while ML models can be either directly embedded or hosted as a service that NiFi calls. Further downstream, Flink can be used to apply stat

Anyone know of someone with the following that is on the market for a senior level position in Chantilly, VA?   Active TS/SCI/Poly 14 years with Bachelors in Science; 12 Years with Masters; 9 Years with PhD. Overall experience can be considered in lieu of degree Experience with scripting languages especially Python and Bash Java Development experience, Java, JavaScript, Spring Boot Strong working Linux experience Experience with Cloud and Amazon Web Services. S3, EC2, Lambda, SNS Agile Development experience Ability to work with data modelers, data architects, software developers, DBAs to achieve project objectives

This article explains how to setup Hive Warehouse Connector (HWC), in CDP Public Cloud (tested with CDP Public Cloud Datahub runtime 7.1). Note: you must have admin privileges on your datahub cluster do this configuration. Step 1: Whitelist the path to the HWC jar In your CDP Datahub, open Cloudera Manager (CM) via the link in the management console: In CM, go to your cluster > Livy > Configuration, and search for livy-conf/livy.conf: Add the following safety valve to the configuration file:livy.file.local-dir-whitelist=/path_for_hwc/​ In our example, we are using the /tmp/ folder. Restart the Livy service via CM to propagate the configuration. Step 2: Copy the HWC jar to the whitelisted location Find the hostname of the node where Livy is installed (master3 here): Connect to the node using your user/workload password, e.g.:ssh pvidal@viz-data-engineering-master3.viz-cdp.a465-9q4k.cloudera.site​ Find the HWC jar:[pvidal@viz-data-engineering-master3 /]$ find / -name

The EFM (Edge Flow Manager) makes it super simple to write flows for MiNiFi to execute where ever it may be located (laptops, refineries, phones, OpenShift,etc).  All agents (MiNiFi) are assigned an agentClass.  Once the agent is turned on, it will phone home to EFM for run-time instructions. The run-time instructions are set at the Class level.  Meaning all agents within a class, run the same instruction (flow) set. There can be 0 to many Classes.  In this example, I will capture Windows Security Events via MiNiFi and ship them to NiFi over Site2Site   Download MiNiFi MSI and set the classname.  In this example, I set the classname to test6.  This property is set at install time (MSI) or by going directly into minifi.properties. Also, notice the setting nifi.c2.enable=true.  This informs MiNFi that run time flow instructions will be received from EFM.  Start MiNiFi. MiNiFi can be configured to send data to multi endpoint (ie Kafka, N

OPC UA is a 100% open source protocol found in many industrial networks. Much like the Apache Foundation stewards the Hadoop project the OPC Foundation https://opcfoundation.org/ is the steward of the OPC UA standard. You can access the standard at https://opcfoundation.org/developer-tools/specifications-unified-architecture for the standard. In addition to the standard itself, the OPC foundation has released implementations of clients, data servers and discovery servers for OPC UA in Java, .NET, and C. The Java repo can be found here OPC UA servers. https://github.com/OPCFoundation/UA-JavaThe following Nifi Service & Processor bundle can be used to retrieve data from OPC UA severs. https://github.com/wadesalazar/NIFI-OPCUA Information on building the bundle will be provided in a separate article To use the service, start by dropping the GetNodeIds onto the palletMost OPC servers provide a properties dialog that provides the information necessary to configure the GetNodeIds proce

Hey Everyone, In this video, I'm going to show you how to run Data Engineering workloads on Cloudera Data Hub. First, we'll deploy a Data Hub cluster with Zeppelin and Spark. Then, I'll show you an example of a PySpark job accessing data on S3. After that, we'll run another PySpark job to access data in Hive.     More Information More information on Data Hub https://www.cloudera.com/products/data-hub.html Documentation on CDP https://docs.cloudera.com/

Note: Cloudera does not support antivirus software of any kind. This article contains generic recommendations for excluding HDP components and directories from AV scans and monitoring. It is important to note that these recommendations do not apply to each service, and further, some services will have additional items to exclude which are unique to them. These details will be addressed in individual articles dedicated to the service in question. The three primary locations you will want to exclude from antivirus are:Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the AV holds up writes.Log directories: These are write-heavy.Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the AV holds up writes.Consider excluding the following directories and all of

Preface Apache HBase is an established NoSQL storage system that has been proven out in sizable production deployments. Although HBase is a key value store, there is a high demand to access data more easily like in SQL. Apache Spark includes great support for basic filtering and insertion of data via SQL though it's SparkSQL offering. The HBase project provides integration with SparkSQL through the work of the HBase Connectors subproject. This article describes the required steps of accessing HBase stored data in SQL fashion. History Several integrations for accessing HBase from Spark have occurred in the past. The first connector was developed by Cloudera Professional Services, which was called Spark on HBase. A version of that early work was contributed to the upstream HBase projects as a feature that the community considered experimental. Cloudera included a derivative of this community version (called hbase-spark) in both CDH 5 and CDH 6. Around the same time, Hortonworks also cam

Problem DescriptionIn Impala, the following error occurs: [ip-10-45-1-222.ec2.internal:21000] default> drop database test cascade;Query: drop database test cascadeERROR: ImpalaRuntimeException: Error making 'dropDatabase' RPC to Hive Metastore: CAUSED BY: TTransportException: nullAttempting the same in Hive displays the following: 0: jdbc:hive2://localhost:10000/default> drop database test cascade;Error: Error while compiling statement: FAILED: SemanticException Unable to fetch table test_wide. null (state=42000,code=40000)0: jdbc:hive2://localhost:10000/default>CauseThis issue occurs when the table in the backend has some null values for a given table or database. You need to clean up those nulls for table names before re-trying the DROP table options. SolutionTo resolve this issue, do the following to remove the Null Entry from the back end database: Note: Ensure you take a Mysql backup. This solution is NOT recommended for production use.

In the use case solved for this webinar, I am a Streaming Engineer at an airline, CloudAir. I need to find, filter and clean Twitter streams then perform sentiment analysis. Score Models in the Stream to Act     As the Streaming Engineer at CloudAIR I am responsible for ingesting data from thousands of sources, operationalizing machine learning models as part of our streams, running real-time ELT/ETL processes and building event processing systems running from devices, servers and edge nodes. For today’s use case, one of our Machine Learning engineers had given me a model that was deployed into one of our production Cloudera Machine Learning (CML) environments. I logged into Cloudera, found the model, tested it, and then extracted the information I need to add this model to our streaming ingest flow for the social media team.     I have been given permissions to access the airline-sentiment workshop in CDP Public Cloud.     I can see all the model

Hey Everyone, In this video, I'll show you how to use experiments in Cloudera AI. Experiments allow a user to run the same script with different inputs easily and compare metrics. This can be useful when performing hyperparameter optimization. In the video, I walk through an extremely simple example where we get the sum of numbers. Then, I run through an example that reflects a more real-world use case.     More Information Cloudera Machine Learning - What You Should Know https://community.cloudera.com/t5/Community-Articles/Cloudera-Machine-Learning-What-You-Should-Know/ta-p/292935 More information on Cloudera AI https://www.cloudera.com/products/machine-learning.html Documentation on CDP https://docs.cloudera.com/

Create Kafka topic /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper `hostname`:2181 --replication-factor 1 --partitions 1 --topic kafka_hive_topic Create Hive table. (update the Kafka broker hostname below) CREATE EXTERNAL TABLE kafka_hive_table (`Country Name` string , `Language` string, `_id` struct<`$oid`:string>) STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler' TBLPROPERTIES ("kafka.topic" = "kafka_hive_topic", "kafka.bootstrap.servers"="c2114-node2.labs.com:6667"); Download the sample json data. wget -O countries.json https://github.com/ozlerhakan/mongodb-json-files/blob/master/datasets/countries.json?raw=true Produce data into Kafka topic. cat countries.json | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list c2114-node2.-labs.com:6667 --topic kafka_hive_topic Describe table (to see additional Kafka specific columns) describe kafka_hive_table;+---------------+----------------------+-------------------

GOAL Migrate hive data and metadata Ensure that metadata is updated based on the target clusters Hive metastore versionI have used tpch data for demonstration purposesSteps (Performed on Old Cluster)Take a note of all the directories you need to copy from old to new cluster on HDFS [hive@xlnode-standalone datagen]$ hdfs dfs -ls /apps/hive/warehouseFound 1 itemsdrwxrwxrwx - hive hdfs 0 2017-03-01 23:31 /apps/hive/warehouse/tpch_text_2.db[hive@xlnode-standalone datagen]$ beeline -u "jdbc:hive2://localhost:10000/default" -n hive -p ''WARNING: Use "yarn jar" to launch YARN applications.Connecting to jdbc:hive2://localhost:10000/defaultConnected to: Apache Hive (version 1.2.1000.2.4.3.0-227)Driver: Hive JDBC (version 1.2.1000.2.4.3.0-227)Transaction isolation: TRANSACTION_REPEATABLE_READBeeline version 1.2.1000.2.4.3.0-227 by Apache Hive0: jdbc:hive2://localhost:10000/default> show databases;+----------------+--+| database_name |+----------------+--+| default || tpch_t

Hey Everyone, In this video, we'll show how to create and send streaming data from NiFi into Kafka. This has never been easier, thanks to the Cloudera. We'll show how you can generate mock engine data and stream it in real-time to a Data Hub cluster running Kafka. Once received, you can also see how the data is stored in Kafka using Streams Messaging Manager (SMM) as it waits to be processed by subscribers.     More information: More information on Cloudera Data Flow https://www.cloudera.com/products/cdf.html More information on Data Hub https://www.cloudera.com/products/data-hub.html Documentation on CDP https://docs.cloudera.com/

This video describes how to register an HDP cluster in DataPlane:   Open the video on YouTube here    DataPlane is a portfolio of data solutions that supports the management in discovery of data (whether at-rest or in-motion) and enable an enterprise hybrid data strategy (from data center to the cloud).   DataPlane is composed of a core platform (“DP Platform” or “Platform”) and an extensible set of apps (“DP Apps”) that are installed on the platform. Depending on the app which you plan to use, you may be required to install an agent into a cluster to support that app, as well as meet other cluster requirements.   The following are documents for reference: Configure Knox Gateway for DataPlane Configure Knox SSO for DataPlane

kafka-python is best used with newer brokers (0.10 or 0.9), but is backwards-compatible with older versions (to 0.8.0). Some features will only be enabled on newer brokers, however; for example, fully coordinated consumer groups -- i.e., dynamic partition assignment to multiple consumers in the same group -- requires use of 0.9+ kafka brokers. Supporting this feature for earlier broker releases would require writing and maintaining custom leadership election and membership / health check code (perhaps using zookeeper or consul). For older brokers, you can achieve something similar by manually assigning different partitions to each consumer instance with config management tools like chef, ansible, etc. This approach will work fine, though it does not support rebalancing on failures. See <http://kafka-python.readthedocs.org/en/master/compatibility.html> for more details.Please note that the master branch may contain unreleased features. For release documentation, please see readthe

Short Description:This article covers how to run a simple producer and consumer in kafka-python (1.4.5) on Kafka 2.0.0, within a kerberized HDP 3.1.0 cluster.ArticleRunning a producer in a kerberized HDP 3.1 Kafka 2.0.0 environment using kafka-pythonCovering pre-requisitesValidating the kerberos setup of kafka on HDP 3.1Running a producer and consumer in kafka-python 1.4.5 on a kerberized kafka clusterPre-RequisitesYou are running a kerberized HDP 3.1.0 cluster with Kafka 2.0.0 installedYou are running one of the following python versions: 2.7 , 3.4 , 3.5 , 3.6 , 3.7Note that in the example from this article, I am using SASL_PLAINTEXT. You will need to modify some parameters if you are using SASL_SSL, as well as add some ssl producer/consumer parameters will need to be specified in your kafka-python code in addition.Please also take note of the current kafka-python compatibility notes:https://kafka-python.readthedocs.io/en/master/compatibility.htmlSteps We will first need to install th

Abstract This article objective is to explain how one can Enrich Ranger Request Context with additional data which can be very useful during policy evaluation. Problem Statement As you all know Ranger Tag based policies are very powerful in implementing data governance using ABAC model, one can define data attributes at Tag level within Atlas and use them during policy condition creation using JavaScript style execution engine, by default you can only access certain context level data like user, group, action, IP address and tag attribute values etc within this javascript logic, but these are not sufficient to define complex authorization policies you see in the enterprise, authorization polices may require additional data like project/usecase user is assigned to, external data which influences policy evaluation etc. How are you going to resolve these kind of situations?Solution This is where Ranger Dynamic Context hook comes handy, Ranger framework allow you to hook your own c