Community Articles

Go here for the latest Prototype - Fraud Detection See the webinar for this Prototype here: https://www.cloudera.com/about/events/webinars/build-a-customer-churn-insights-application-with-cdp.html   The complexity of creating an end-to-end machine learning workflow is one of the biggest hurdles data science and machine learning engineers are facing. It's not a running model.fit() that is hard, but it is ingesting data, getting the data in the right format for the model training process, deploying the model in a way that is accessible to other parts of the business, and running applications that consume the model that is hard. Machine Learning is useful when it's deployed with an end-to-end workflow.     We have been working to create Applied Machine Learning Prototypes for CML that will help you build a fully working machine learning example in CML. The Prototypes will include source data, and walk through various steps: Ingest data into a useful place in CDP

Introduction How do we quickly gain insight and start working with data in a secure, governed, and scalable environment in the cloud?   This article explains how to achieve this using the Cloudera Data Warehouse platform connected with Apache Superset.   Cloudera Data Warehouse in CDP (Cloudera Data Platform) is an enterprise solution for modern analytics. It's an auto-scaling, highly concurrent, and cost-effective hybrid, a multi-cloud analytics solution that ingests data anywhere, at massive scale, from structured, unstructured, and edge sources.   Apache Superset (incubating) is a modern, enterprise-ready business intelligence web application. Pre-Requisites This exercise is performed on a Mac OS. The versions below were tested at the moment of writing this article and may change in the future:   Python         Python 3.7.5 pip 20.0.2         After Python/pip installation, install the following packages/versions in Python

JSON Batch to Single Row PhoenixI grabbed open data on Crime from Philly's Open Data (https://www.opendataphilly.org/dataset/crime-incidents), after a free sign up you get access to JSON crime data (https://data.phila.gov/resource/sspu-uyfa.json) You can grab individual dates or ranges for thousands of records. I wanted to spool each JSON record as a separate HBase row. With the flexibility of Apache NiFi 1.0.0, I can specify run times via cron or other familiar setup. This is my master flow.First I use GetHTTP to retrieve the SSL JSON messages, I split the records up and store them as RAW JSON in HDFS as well as send some of them via Email, format them for Phoenix SQL and store them in Phoenix/HBase. All with no coding and in a simple flow. For extra output, I can send them to Reimann server for monitoring.Setting up SSL for accessing HTTPS data like Philly Crime, require a little configuration and knowing what Java JRE you are using to run NiFi. You can run service nifi s

Hey Everyone, This video will show you how to create a Cloudera AI Workspace. A Workspace in CML is where all of your data science work will take place and allow you to collaborate with your team easily. Creating a basic Workspace is quick and simple if you already have an environment set up. Also, I cover the different advanced options available to help admins and users manage costs and stay secure when creating a new Workspace.     More information: Cloudera Machine Learning – What You Should Know https://community.cloudera.com/t5/Community-Articles/Cloudera-Machine-Learning-What-You-Should-Know/ta-p/292935 More information on Cloudera AI https://www.cloudera.com/products/machine-learning.html Documentation on CDP https://docs.cloudera.com/

Short Description:This sample code helps to connect to Kerberos enabled HBase cluster from Java program. Code Walkthrough:Create HBaseConfiguration and pass HBase cluster parameters. Configuration hbaseConfig = HBaseConfiguration.create();hbaseConfig.addResource("/path_to_hbase_conf/hbase-site.xml");hbaseConfig.addResource("/path_to_hbase_conf/core-site.xml");hbaseConfig.set("hadoop.security.authentication", "Kerberos"); User principal and key tab file names. Please make sure key tab files are in the respective folder. String principal = System.getProperty("kerberosPrincipal", "hbaseuser@EXAMPLE.COM");String keytab = System.getProperty("kerberosKeytab", "/path_to_keytab/hbase-client.keytab"); The essential Kerberos configuration information is the default realm and the default KDC. As with most Kerberos installations, a Kerberos configuration file krb5.conf is consulted to determine such things as the default realm and KDC. The default location i

It's very easy to integrate Apache Flume with Apache NiFi, either as a source or destination (sink).An example of using ExecuteFlumeSource and ExecuteFlumeSink in Apache NiFi. We use a Flume source to read Netcat and send to Slack. So you can send messages from Linux to a Slack chat. For the Flume sink to HDFS, we read data from a web feed. We can also connect our Flume Source to Flume Sink.Configuring a Flume SourceIf you have configured Apache Flume configuration files, this is the same thing. Data Received in Slack (Join and See Messages Live)Checking Provenance EventConfigure The ExecuteFlumeSink to Store to HDFS-rw-r--r-- 3 flume hdfs 28 2016-07-25 16:14 /flume/events/16-07-25/1610/00/events-.1469463242392To monitor what's going on in the Flume side of the process: tail -f /var/log/flume/flume-a1.log25 Jul 2016 16:14:38,854 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177) - Shutdown Metric for type: SINK, name: k1. si

To set up health_percent of LLAP, do the following:    On the Hiveserver2Interactive server nodes, edit /usr/hdp/<hdp-version>/hive/scripts/llap/yarn/package.py​Example: /usr/hdp/3.1.0.224-3/hive/scripts/llap/yarn/package.pyThe --health-percent defaults to 80. Change this to desired-number. Move the following to a temporary backup location: a. /usr/hdp/<hdp-version>/hive/scripts/llap/yarn/package.pyc b. /usr/hdp/<hdp-version>/hive/scripts/llap/yarn/package.pyo ​ Restart Hive.

Note: Cloudera does not support antivirus software of any kind.   This article contains general recommendations for excluding ZooKeeper components and directories from antivirus scans and monitoring.   The three primary locations you will want to exclude from antivirus are: Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the antivirus holds up writes. Log directories: These are write-heavy. Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the antivirus holds up writes.   Note: ZooKeeper has a user-configurable data directory. I recommend you exclude it. This directory can be found by running the following command: # grep dataDir /etc/zookeeper/conf/zoo.cfg Consider excluding the following directories and all of their subdirectories

Note: Cloudera does not support antivirus software of any kind.   This article contains general recommendations for excluding MapReduce components and directories from antivirus scans and monitoring.   The three primary locations you will want to exclude from antivirus are: Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the antivirus holds up writes. Log directories: These are write-heavy. Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the antivirus holds up writes. Note: Some directories in MapReduce are user-configurable. I recommend you exclude them. These properties can be found in Ambari > YARN > Configs > Advanced, and this one in particular should be excluded:     mapreduce.jobhistory.recovery.store.leveldb.path &nbs

Note: Cloudera does not support antivirus software of any kind.   This article contains general recommendations for excluding YARN components and directories from antivirus scans and monitoring.   The three primary locations you will want to exclude from antivirus are: Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the antivirus holds up writes. Log directories: These are write-heavy. Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the antivirus holds up writes. Note: The directories YARN uses are user-configurable. I recommend you exclude them. These properties can be found in Ambari > YARN > Configs > Advanced:   yarn.nodemanager.local-dirsyarn.nodemanager.log-dirsyarn.nodemanager.recovery.diryarn.timeline-servic

Note: Cloudera does not support antivirus software of any kind.   This article contains general recommendations for excluding HDFS components and directories from antivirus scans and monitoring.   The three primary locations you will want to exclude from antivirus are: Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the antivirus holds up writes. Log directories: These are write-heavy. Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the antivirus holds up writes. Note: The directories in HDFS are user-configurable. I recommend you exclude these, especially the data directory for the DataNode and the meta directories for the NameNode and JournalNode. These details can be found in the “hdfs-site.xml” file:       # grep -A

Note: Cloudera does not support antivirus software of any kind.   This article contains general recommendations for excluding Ambari components and directories from antivirus scans and monitoring.   The three primary locations you will want to exclude from antivirus are: Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the AV holds up writes. Log directories: These are write-heavy. Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the AV holds up writes.   Note: Ambari has a special requirement in the form of a user-configurable database. I recommend you exclude this database. However, the details of this database are set on installation; the database may be colocated with ambari-server, or on a remote host. Consult with your database adm

ExtractText NiFi Custom Processor Powered by Apache Tika Apache Tika is amazing, it is very easy to use it to analyze file and then to extract text with it. Apache Tika uses other powerful Apache projects like Apache PDFBox and Apache POI.Example Usage Feed in documents, I use my LinkProcessor which grabs links from a website and returns a JSON List.Split the resulting JSON list into individual JSON rows with SplitJSON.EvaluateJSONPath to extract just the URLs.InvokeHTTP to do a GET on that parsed URL.RouteOnAttribute to only process file types I am interested in like Microsoft Word.The new ExtractTextProcessor to extract the text of the document.Then we save the text as a file in some data store, perhaps HDFS.If you have a directory of files, you can just use GetFile to ingest them en masse.LinkProcessor (https://github.com/tspannhw/linkextractorprocessor)URL: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/index.htmlThis is an example of a URL that I want to grab all the do

In Part 1, I shared an alternative architecture for Prometheus that increases scalability and flexibility of time series metric monitoring. In part 2, I will walk through an extension of the architecture that unites metric and log data with a unified, scalable data pipeline. Part 1 Architecture Part 2 Architecture By adding log collection agents (e.g. MiNiFi) and a search cluster (e.g. Solr), the solution architecture can be extended to support log data in addition to time series metrics. This has the advantage of reducing duplicated infrastructure components for a more efficient and supportable solution.   Additional NiFi processors can be added to the flow for pre-processing (e.g. filtering, routing, scoring) the incoming data (e.g. ERROR vs INFO messages). Rulesets (e.g. Drools) from expert systems can be embedded directly into the flow, while ML models can be either directly embedded or hosted as a service that NiFi calls. Further downstream, Flink can be used to apply stat

This article explains how to setup Hive Warehouse Connector (HWC), in CDP Public Cloud (tested with CDP Public Cloud Datahub runtime 7.1). Note: you must have admin privileges on your datahub cluster do this configuration. Step 1: Whitelist the path to the HWC jar In your CDP Datahub, open Cloudera Manager (CM) via the link in the management console: In CM, go to your cluster > Livy > Configuration, and search for livy-conf/livy.conf: Add the following safety valve to the configuration file:livy.file.local-dir-whitelist=/path_for_hwc/​ In our example, we are using the /tmp/ folder. Restart the Livy service via CM to propagate the configuration. Step 2: Copy the HWC jar to the whitelisted location Find the hostname of the node where Livy is installed (master3 here): Connect to the node using your user/workload password, e.g.:ssh pvidal@viz-data-engineering-master3.viz-cdp.a465-9q4k.cloudera.site​ Find the HWC jar:[pvidal@viz-data-engineering-master3 /]$ find / -name

The EFM (Edge Flow Manager) makes it super simple to write flows for MiNiFi to execute where ever it may be located (laptops, refineries, phones, OpenShift,etc).  All agents (MiNiFi) are assigned an agentClass.  Once the agent is turned on, it will phone home to EFM for run-time instructions. The run-time instructions are set at the Class level.  Meaning all agents within a class, run the same instruction (flow) set. There can be 0 to many Classes.  In this example, I will capture Windows Security Events via MiNiFi and ship them to NiFi over Site2Site   Download MiNiFi MSI and set the classname.  In this example, I set the classname to test6.  This property is set at install time (MSI) or by going directly into minifi.properties. Also, notice the setting nifi.c2.enable=true.  This informs MiNFi that run time flow instructions will be received from EFM.  Start MiNiFi. MiNiFi can be configured to send data to multi endpoint (ie Kafka, N

OPC UA is a 100% open source protocol found in many industrial networks. Much like the Apache Foundation stewards the Hadoop project the OPC Foundation https://opcfoundation.org/ is the steward of the OPC UA standard. You can access the standard at https://opcfoundation.org/developer-tools/specifications-unified-architecture for the standard. In addition to the standard itself, the OPC foundation has released implementations of clients, data servers and discovery servers for OPC UA in Java, .NET, and C. The Java repo can be found here OPC UA servers. https://github.com/OPCFoundation/UA-JavaThe following Nifi Service & Processor bundle can be used to retrieve data from OPC UA severs. https://github.com/wadesalazar/NIFI-OPCUA Information on building the bundle will be provided in a separate article To use the service, start by dropping the GetNodeIds onto the palletMost OPC servers provide a properties dialog that provides the information necessary to configure the GetNodeIds proce

Hey Everyone, In this video, I'm going to show you how to run Data Engineering workloads on Cloudera Data Hub. First, we'll deploy a Data Hub cluster with Zeppelin and Spark. Then, I'll show you an example of a PySpark job accessing data on S3. After that, we'll run another PySpark job to access data in Hive.     More Information More information on Data Hub https://www.cloudera.com/products/data-hub.html Documentation on CDP https://docs.cloudera.com/

Note: Cloudera does not support antivirus software of any kind. This article contains generic recommendations for excluding HDP components and directories from AV scans and monitoring. It is important to note that these recommendations do not apply to each service, and further, some services will have additional items to exclude which are unique to them. These details will be addressed in individual articles dedicated to the service in question. The three primary locations you will want to exclude from antivirus are:Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the AV holds up writes.Log directories: These are write-heavy.Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the AV holds up writes.Consider excluding the following directories and all of

Preface Apache HBase is an established NoSQL storage system that has been proven out in sizable production deployments. Although HBase is a key value store, there is a high demand to access data more easily like in SQL. Apache Spark includes great support for basic filtering and insertion of data via SQL though it's SparkSQL offering. The HBase project provides integration with SparkSQL through the work of the HBase Connectors subproject. This article describes the required steps of accessing HBase stored data in SQL fashion. History Several integrations for accessing HBase from Spark have occurred in the past. The first connector was developed by Cloudera Professional Services, which was called Spark on HBase. A version of that early work was contributed to the upstream HBase projects as a feature that the community considered experimental. Cloudera included a derivative of this community version (called hbase-spark) in both CDH 5 and CDH 6. Around the same time, Hortonworks also cam

Problem DescriptionIn Impala, the following error occurs: [ip-10-45-1-222.ec2.internal:21000] default> drop database test cascade;Query: drop database test cascadeERROR: ImpalaRuntimeException: Error making 'dropDatabase' RPC to Hive Metastore: CAUSED BY: TTransportException: nullAttempting the same in Hive displays the following: 0: jdbc:hive2://localhost:10000/default> drop database test cascade;Error: Error while compiling statement: FAILED: SemanticException Unable to fetch table test_wide. null (state=42000,code=40000)0: jdbc:hive2://localhost:10000/default>CauseThis issue occurs when the table in the backend has some null values for a given table or database. You need to clean up those nulls for table names before re-trying the DROP table options. SolutionTo resolve this issue, do the following to remove the Null Entry from the back end database: Note: Ensure you take a Mysql backup. This solution is NOT recommended for production use.

In the use case solved for this webinar, I am a Streaming Engineer at an airline, CloudAir. I need to find, filter and clean Twitter streams then perform sentiment analysis. Score Models in the Stream to Act     As the Streaming Engineer at CloudAIR I am responsible for ingesting data from thousands of sources, operationalizing machine learning models as part of our streams, running real-time ELT/ETL processes and building event processing systems running from devices, servers and edge nodes. For today’s use case, one of our Machine Learning engineers had given me a model that was deployed into one of our production Cloudera Machine Learning (CML) environments. I logged into Cloudera, found the model, tested it, and then extracted the information I need to add this model to our streaming ingest flow for the social media team.     I have been given permissions to access the airline-sentiment workshop in CDP Public Cloud.     I can see all the model

Hey Everyone, In this video, I'll show you how to use experiments in Cloudera AI. Experiments allow a user to run the same script with different inputs easily and compare metrics. This can be useful when performing hyperparameter optimization. In the video, I walk through an extremely simple example where we get the sum of numbers. Then, I run through an example that reflects a more real-world use case.     More Information Cloudera Machine Learning - What You Should Know https://community.cloudera.com/t5/Community-Articles/Cloudera-Machine-Learning-What-You-Should-Know/ta-p/292935 More information on Cloudera AI https://www.cloudera.com/products/machine-learning.html Documentation on CDP https://docs.cloudera.com/

Create Kafka topic /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper `hostname`:2181 --replication-factor 1 --partitions 1 --topic kafka_hive_topic Create Hive table. (update the Kafka broker hostname below) CREATE EXTERNAL TABLE kafka_hive_table (`Country Name` string , `Language` string, `_id` struct<`$oid`:string>) STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler' TBLPROPERTIES ("kafka.topic" = "kafka_hive_topic", "kafka.bootstrap.servers"="c2114-node2.labs.com:6667"); Download the sample json data. wget -O countries.json https://github.com/ozlerhakan/mongodb-json-files/blob/master/datasets/countries.json?raw=true Produce data into Kafka topic. cat countries.json | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list c2114-node2.-labs.com:6667 --topic kafka_hive_topic Describe table (to see additional Kafka specific columns) describe kafka_hive_table;+---------------+----------------------+-------------------

GOAL Migrate hive data and metadata Ensure that metadata is updated based on the target clusters Hive metastore versionI have used tpch data for demonstration purposesSteps (Performed on Old Cluster)Take a note of all the directories you need to copy from old to new cluster on HDFS [hive@xlnode-standalone datagen]$ hdfs dfs -ls /apps/hive/warehouseFound 1 itemsdrwxrwxrwx - hive hdfs 0 2017-03-01 23:31 /apps/hive/warehouse/tpch_text_2.db[hive@xlnode-standalone datagen]$ beeline -u "jdbc:hive2://localhost:10000/default" -n hive -p ''WARNING: Use "yarn jar" to launch YARN applications.Connecting to jdbc:hive2://localhost:10000/defaultConnected to: Apache Hive (version 1.2.1000.2.4.3.0-227)Driver: Hive JDBC (version 1.2.1000.2.4.3.0-227)Transaction isolation: TRANSACTION_REPEATABLE_READBeeline version 1.2.1000.2.4.3.0-227 by Apache Hive0: jdbc:hive2://localhost:10000/default> show databases;+----------------+--+| database_name |+----------------+--+| default || tpch_t