Community Articles

We frequently get asked questions by customers about how to run, deploy and/or debug Cloudera’s Operational Database, which is based on Apache HBase & Phoenix. This post shares the most frequently asked questions and how they were addressed.   Can I run HBase on HDP 7.1.4 for my production environment?Answer: HDP 7.1.4 is only an intermediate step in the upgrade from HDP2 or HDP 3 to CDP. Once HDP 7.1.4 is deployed, you need to use the AM2CM migration tool to complete the upgrade to CDP. HDP 7 is not supported for production environments. HBase is not expected to be stable in HDP 7 but will be stable after the completion of the upgrade to CDP. How can I repair my HBase metadata?Answer: Use the hbck2 meta repair feature. For more information about this feature, see this github link. Alternatively, if there is no important data there, you can clean up the whole HDFS HBase root directory and also remove the HBase ‘znode’ from ZooKeeper using the deleteall/hbase comman

In this post, we will learn how to create a Kafka topic and produce and consume messages from a Kafka topic. After testing the basic producer and consumer example, we will test it with Spark using spark-examples.jar file.   Creating a Kafka topic:   # kafka bootstrap serverKAFKA_BROKERS="localhost:9092"# kafka topic nameTOPIC_NAME="word_count_topic"# group nameGROUP_NAME="spark-kafka-group"# creating a topic/opt/cloudera/parcels/CDH/lib/kafka/bin/kafka-topics.sh --create --topic ${TOPIC_NAME} --bootstrap-server ${KAFKA_BROKERS}# describing a topic/opt/cloudera/parcels/CDH/lib/kafka/bin/kafka-topics.sh --describe --topic ${TOPIC_NAME} --bootstrap-server ${KAFKA_BROKERS}     Producing messages to Kafka topic:   # producing kafka messages/opt/cloudera/parcels/CDH/lib/kafka/bin/kafka-console-producer.sh --topic ${TOPIC_NAME} --broker-list ${KAFKA_BROKERS}     Consuming messages from Kafka topic:   # consuming kafka messages/opt/cloudera/parcel

Intro Online shopping is on the rise as more of us stay at home and let our credit cards do the walking. Keeping pace with that trend is an unfortunate increase in credit card fraud. It’s no surprise, really. According to Forbes, online fraud has been a growing problem for the past few years. And now, as consumers and businesses adapt to the worldwide pandemic and make more credit card transactions in the card-not-present (CNP) space, the resulting uptick in online shopping and e-commerce has opened up an even bigger playground for fraudsters to try out new tricks. Fraud detection has been a major issue for financial services and institutions. But artificial intelligence has an enormous potential to reduce financial fraud. Artificial intelligence applications have a great potential to detect and prevent fraud. Therefore, we will start a series of articles talking about that and how we can use Cloudera mechanisms to implement a whole Credit Card Fraud detection solution. But first, le

If you have erasure coded some directory and perform some operations on the directory you might have observed WARN messages like belowWARN erasurecode.ErasureCodeNative: Loading ISA-L failed: Failed to load libisal.so.2 (libisal.so.2: cannot open shared object file: No such file or directory)WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicableThis WARN messages are due to ISA Library not being present on the node. Below are the steps to enable the library1) Clone the isa-l github repository.# git clone https://github.com/01org/isa-l.git2) Go to the cloned directory# cd isa-l3) Install yasm if you do not have it already# yum install -y yasm ---> centOS# apt-get install yasm ----> ubuntu4) Build the library # make -f Makefile.unx5) Copy the library files to lib directory# cp bin/libisal.so bin/libisal.so.2 /lib646) Verify that isa-l library is enabled properly# hadoop checknativeExpected output18/10/12 10

This video contains a step by step process that shows how to connect to Hive running on a secure cluster while using a JDBC Uber driver from MS Windows. Open the video on YouTube here   Prerequisites: Validate username belong to the same @DOMAIN /Realm as the one setup on cluster nodes. Install  DbVisualizer. Download Hive Uber driver for same version as HDP. Kerberos Java  Config Get Kerberos /etc/krb5.conf from cluster, scp this file from any cluster node to c:\windows. Rename krb5.conf to c:\windows\krb5.ini. Edit krb5.ini and add property: (see video)  property udp_preference_limit = 1   DbVisualizer Setup Add Hive Uber driver .jar to DBVisualizer as driver for Hive (see video). Add startup Java command line options to DBVisualizer for kerberos , under Tools (see video): -Dsun.security.krb5.debug=true-Djavax.security.auth.useSubjectCredsOnly=false-Djava.security.krb5.conf=c:\windows\krb5.in

in August 2021, we released CDP Private Cloud Base 7.1.7 LTS.This article will outline what that means.For each software release, we provide a version, General Availability (GA) date, and End of Service (EoS) date. These are all published and described here. CDP Private Cloud Base 7.1.7 LTS has a GA date of August 2021 and an EoS date of August 2025, providing a 4-year support window. We then offer two paths,Service Pack TrainThe Service Pack train is for customers who value a stable platform, service a lot of users or use-cases, and want a low risk, low costs in maintenance and operations. Rather than taking the new feature release, customers can instead take regular Cumulative Hotfix (CHF)*. These hotfixes are released pro-actively to address break-fix issues and CVEs. At certain pre-planned release date, the hotfixes will be rolled up into a Service Pack (e.g. SP1)**. * Cumulative Hotfix means each hotfix includes those that came before up to the initial release

Steps to connect from Legacy Cluster to CDP Public Cloud Hive:   Build a connection string as follows using Hive Server Endpoint from DataHub by just adding the sslTrustStore and trustStorePassword along with workload username and password for PAM authentication.jdbc:hive2://master0.repro.cloudera/default;transportMode=http;httpPath=new/cdp-proxy-api/hive;ssl=true;sslTrustStore=gateway-client-trust.jks;trustStorePassword=*****;user=chella; Get the JKS file downloaded from CDP Knox (Token Integration) and copy it to HDP/CDH host. Get the latest JDBC jar "hive-jdbc-3.1.0-SNAPSHOT-standalone.jar" from CDW.CDP Control Plane > Management Console > Data Warehouse > click the 3 dots from any of Hive VW and "Download JDBC JAR" Copy the jar "hive-jdbc-3.1.0-SNAPSHOT-standalone.jar"  to HDP/CDH host. Get into beeline from HDP/CDH host Add the jar using the following commandBeeline version 3.1.0.3.1.5.26-1 by Apache Hive 0: jdbc:hive2://c1544-node2> !addlocaldriverjar /roo

Perform the following steps to access the Hive MetaStore database :- SSH to the Data lake CM Node as 'cloudbreak' user.Run 'sudo -i' to gain root access.source activate_salt_envexport PGPASSWORD=$(salt-call pillar.get postgres:hive:password 2>/dev/null| tail -n 1 | awk '{print $1}')HIVE_DB=$(salt-call pillar.get postgres:hive:database 2>/dev/null| tail -n 1 | awk '{print $1}')HIVE_DB_USER=$(salt-call pillar.get postgres:hive:user 2>/dev/null| tail -n 1 | awk '{print $1}')psql -U ${HIVE_DB_USER} -d ${HIVE_DB}e.g. [root@xxxx-master0 ~]# source activate_salt_env(salt_3001.8) [root@xxxx-master0 ~]# export PGPASSWORD=$(salt-call pillar.get postgres:hive:password 2>/dev/null| tail -n 1 | awk '{print $1}')(salt_3001.8) [root@xxxx-master0 ~]# HIVE_DB=$(salt-call pillar.get postgres:hive:database 2>/dev/null| tail -n 1 | awk '{print $1}')(salt_3001.8) [root@xxxx-master0 ~]# HIVE_DB_USER=$(salt-call pillar.get postgres:hive:user 2>/dev/null| tail -n 1 | awk '{print $1

In NiFi the data being passed between operators is referred to as a FlowFile and can be accessed via various scripting languages in the ExecuteScript operator.In order to access the data in the FlowFile you need to understand a few requirements first.In this example we will access the json data being passed into the ExecuteScript operator via a getTwitter operator.This could be any data ingestion operator.Using the ExecuteScript (Note: Execute script will use Jython libraries which are good but limited. If you have a python script which uses other libs and produces an output you can use Execute Process instead which will execute the python script on the machine using the full python lib and the output will become your flow file. )  operator set the script type to python and in the "script body" section enter the following code:import jsonimport java.iofrom org.apache.commons.io import IOUtilsfrom java.nio.charset import StandardCharsetsfrom org.apache.nifi.processor.io import Stre

SYNOPSIS The Optimized Row Columnar (ORC) file is a columnar storage format for Hive. Specific Hive configuration settings for ORC formatted tables can improve query performance resulting in faster execution and reduced usage of computing resources. Some of these settings may already be turned on by default, whereas others require some educated guesswork. The table below compares Tez job statistics for the same Hive query that was submitted without and with certain configuration settings. Notice the performance gains with optimization. This article will explain how the performance improvements were achieved. QUERY EXECUTION Source Data: 102,602,110 Clickstream page view records across 5 days of data for multiple countries Table is partitioned by date in the format YYYY-MM-DD. There are no indexes and table is not bucketed. The HiveQL is ranking each page per user by how many times the user viewed that page for a specific date and within the United

This short example outlines how to configure Cloudera Machine Learning to integrate with the Data lake, Hive, and Iceberg tables.Apache Iceberg has a number of qualities that make it very suitable for Machine Learning. It maintains the metadata for the data layout alongside the data; this means there are no heavy network demands on the main catalog. The main catalog in this case Hive Metastore maintains a lightweight set of references to the data on the Data lake.   Iceberg also supports snapshots and time-travel so that data can be versioned and queried at a specific point in time.  Enable parallel processing in Spark   Setup user credentials and default paths to the Data lake.spark-defaults.confspark.executor.instances 2spark.executor.memory 1gspark.executor.cores 2spark.hadoop.yarn.resourcemanager.principal christopherroylesspark.yarn.access.hadoopFileSystems s3a://demo-aws-go02/ Setup a Spark session Load the appropriate Iceberg JAR Add the required SQ

Cloudera has added a StreamNative community-supported processor. Reading OpenSea NFT with Apache NiFi, Apache Pulsar, and Friends. Meetup. StreamNative   Upcoming Meetup Demo Apache Pulsar and Apache NiFi for Cloud Data Lakes Pulsar Build bin/pulsar-admin topics delete persistent://public/default/nftbin/pulsar-admin topics delete persistent://public/default/cryptobin/pulsar-admin topics create persistent://public/default/nftbin/pulsar-admin topics create persistent://public/default/cryptobin/pulsar-client consume "persistent://public/default/crypto" -s "nftreadercrypto" -n 0bin/pulsar-client consume "persistent://public/default/nft" -s "nftyreader" -n 0bin/pulsar-admin schemas get "persistent://public/default/nft" Apache NiFi Flows NiFi -> Pulsar Flow Example NFT JSON Record { "date" : "Thu, 24 Feb 2022 22:26:41 GMT", "short_description" : "", "featured" : "false", "image_thumbnail_url" : "https://lh3.googleusercontent.com/-dsYkPkvAQW6KvP0Nnbjecb4Pqo1RsLvTlKKDdSPiUR4LF

Running Livy on HDP 2.5Ingest Metrics REST API From Livy with Apache NiFi / HDFUse GetHTTP To Ingest The Status On Your Batch From LivyRunning LivyThe first step, we download Livy from github. To install on HDP 2.5, is simple. I found a node that wasn't too busy and put the project their. To run, it's simple:export SPARK_HOME=/usr/hdp/current/spark-client/ export HADOOP_CONF_DIR=/etc/hadoop/conf nohup ./livy-server &That's it, you have a basic unprotected Livy instance running. This is important, there's no security on there. You should either put Knox in front of this or enable Livy's security.I wanted to submit a Scala Spark Batch Job. So I wrote a quick one below to have something to call.Source Code for Example Spark 1.6.2 Batch Application: https://github.com/tspannhw/links https://community.hortonworks.com/repos/73816/links-spark-batch-application-example.html?shortDescriptionMaxLength=140Step 1: GetFileStore File /opt/demo/sparkrun.js with JSON to trigger Spark thr

Cloudera Machine Learning now offers Snippet to connect to Data Sources available within the CDP Environment. Administrators can configure custom Spark, Hive or Impala Virtual Warehouse data connections manually or they can use CML’s features to autodetect and configure all connections from the same CDP Environment. Data Scientists can then access the preconfigured Data Connections from their ML Projects.   The Data Connection and Snippet support simplifies getting started on ML Projects. Once a project is created, the first time the users create a session they are offered code snippets to create a connection to their selected data store. Users don’t need to look up the connection boilerplate from the documentation or copy an example code from other projects, they can easily initiate the connection via CML’s connection library and immediately start solving their business problems.       To learn more, you can read the How To article in the Cloudera Docs.

DISCLAIMER / CREDITS: The majority of the content of this community article was written by Adam Antal.My piece was correcting, adding, or revising the contents. I would like to thank Benjamin Teke, Andras Gyori, and Rudolf Reti for the reviews of this article.IntroductionThis community article is for Cloudera Data Platform Administrators and will clarify how the recent improvements in YARN logging will benefit their ability to manage and support their customers' workloads. In some circumstances YARN will generate a similar order of magnitude of data as that of the application and that can prove challenging for administrators to make use of for diagnostics. This article will illustrate how administrators can use these new features to streamline problem resolution and will be of benefit to developers that are designing YARN applications.The processes of YARN and its launched containers almost always generate application logs that can be potentially large, depending on the type of th

Zeppelin Best Practices Install & Versions Leverage Ambari to install Zeppelin and always use the latest version of Zeppelin. With HDP 2.6.2, Zeppelin 0.7.2 is available and it contains many useful stability & security fixes that will improve your experience. Zeppelin in HDP 2.5.x has many known issues that were resolved in 2.6.2Deployment Choices While you can select any node type to install Zeppelin, the best place is a gateway node. The reason gateway node makes most sense is when the cluster is firewalled off and protected from outside, users can still see the gateway node.Hardware Requirement More memory & more Cores are betterMemory: Minimum of 64 GB nodeCores: Minimum of 8 cores# of users: A given Zeppelin node can support 8-10 users. If you want more users, you can set up multiple Zeppelin instances. More details in MT section.Security: Like any software, the security depends on threat matrix and deployment choices. This section assumes a MT Zeppelin deployment. Aut

Some time ago I faced an interesting problem with a cluster failing to start after I replaced an MIT KDC with a FreeIPA KDC.   For the replacement, I installed the ipa-client package on the cluster nodes, and then, changed the KDC configuration in Cloudera Manager (CM) (changed realm and KDC details, imported Kerberos user, re-generated credentials, etc..)   The cluster refused to start. Besides that, CM's KDC Login monitor kept complaining about the KDC not being healthy. I could manually kinit successfully, though, and there seemed to be no KDC problems at first glance.   After enabling debug at different places I saw that there were socket timeouts when processes tried to connect to the KDC and that those processes were actually trying to connect to the KDC over UDP, rather than TCP. The UDP requests explained the problem, since UDP traffic was blocked between the cluster and the KDC.   What's strange, though, is that the kr

The page Connecting Kafka clients to Data Hub provisioned clusters in the Cloudera documentation explains how to configure the Kafka Console Consumer to connect to a Kafka cluster in DataHub. In this article, I will expand that explanation for a generic Java client and will show how to configure the client to use Schema Registry for the storage and retrieval of schemas.   A full version of the code in this article can be found in GitHub. Client initialization The initialization of the KafkaConsumer or KafkaProducer client is done simply by passing a Properties object to its constructor. The Properties object can be either initialized programmatically or loaded from a properties file. Example: Properties props = new Properties();props.put("bootstrap.servers", "kafka-1.example.com:9093");... initialize other properties ...KafkaConsumer<String, Transaction> consumer = new KafkaConsumer<>(props); The initialization of a KafkaProducer is identical to the above. This con

Objective This tutorial walks you through a NiFI flow that utilizes the QueryRecord processor and Record Reader/Writer controller services to convert a CVS file into JSON format and then query the data using SQL. Note: The record-oriented processors and controller services were introduced in NiFi 1.2.0. As such, the tutorial needs to be done running Version 1.2.0 or later.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.3.0QueryRecord FlowSupport Files Here is a template of the flow discussed in this tutorial: queryrecord-csvtojson.xml Here is the CSV file used in the flow: users.txt (Change the extension from .txt to .csv after downloading) Note: The CSV data originated from the site, RandomUser. This useful site provides a free API to pull down randomly generated user data. For example: https://randomuser.me/api/0.6/?results=10&format=SQL.Demo ConfigurationInput Directory Create a local input direc

A short how-to article on how to list CDP Data hub Kafka topics via command line:PrerequisitesCDP DataHub Kafka clusterWorkload username & passwordCDP PC trust storeStepsFetch CDP cert and store in a trust storeCreating TLS truststoreor simple fetch it from any node  /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks  ssh into any Data hub Kafka node using CDP workload username/passwordCopy trust store onto the nodeSet broker host   BROKER_HOST=$(hostname -f)   Create jaas.conf   KafkaClient {org.apache.kafka.common.security.plain.PlainLoginModule requiredusername="<workload username>"password="<workload password>";};   Set KAFKA_OPTS environment variable to point to location where jaas.conf is located   export KAFKA_OPTS=-Djava.security.auth.login.config=<LOCATION-OF-YOUR-JAASCONF>/jaas.conf   Create client-kerberos.properties 

This article provides the steps and fields required to configure Streams Replication Manager using external accounts between two kerberized clusters. This article assumes that Kerberos is configured properly between the clusters and we can produce and consume data correctly. Environment details: Cluster A: co-located cluster (SRM and Kafka running in this cluster) CDP 7.1.7 Cluster B: External cluster which is another CDP 7.1.7 Both clusters are using  SASL_PLAINTEXT security.protocol for their clients Clusters are configured with cross-domain realm (MIT kerberos), more details on cross-domain realm configuration here Steps to configure external accounts (feature available from CDP 7.1.7) Go to cluster A > Cloudera Manager > Administration > External accounts > Kafka Credentials tab Configure the following fields: Name Bootstrap servers Security protocol JAAS Secret [1-3] JAAS Template Kerberos Service Name SASL Mechanism Example: Name c289 Bootstrap

Spark uses the blacklist mechanism to enhance the scheduler’s ability to track failures. When a task fails on an executor, the blacklist module tracks the executor and host which has failed to execute the task. Beyond a threshold, the scheduler won't be able to schedule any more tasks on that node. If spark.blacklist.enabled is set to true, we need to always set the value of spark.blacklist.task.maxTaskAttemptsPerNode to greater than spark.task.maxFailures, else the Spark job will fail with the following error message: ERROR util.Utils: Uncaught exception in thread mainjava.lang.IllegalArgumentException: spark.blacklist.task.maxTaskAttemptsPerNode ( = 2) was >= spark.task.maxFailures ( = 2 ). Though blacklisting is enabled, with this configuration, Spark will not be robust to one bad node. Decrease spark.blacklist.task.maxTaskAttemptsPerNode, increase spark.task.maxFailures, or disable blacklisting with spark.blacklist.enabled. If you try to trigger the following job from shell of e

Running SQL on stream is super simple with SQL Stream Builder, via Flink SQL. I recorded a live demo and this article is to provide supporting artifacts to run it.   Demo Flink SSB Credit Card Fraud Demo Prerequisite Quick setup Note If you experience any challenges with running SSB, as a sanity check run the following on the cluster     flink-yarn-session -d \ -D security.kerberos.login.keytab=<keytab_filename>.keytab \ -D security.kerberos.login.principal=<workload_username>     If the above works as a sanity check, kill the yarn job.   yarn application -appStates RUNNING -listThat will display all running yarn apps. Your app will show up in this list. Then run this to kill the appyarn application -kill application_xxxx   customer_inferences table   CREATE TEMPORARY TABLE customer_inferences ( `customer_id` INT, `account_number` STRING, `center_inferred_lat` FLOAT, `center_inferred_lon` FLOAT, `max_inferred_distance` STRING,

Prerequisites  Base cluster with Hive on Tez, Zookeeper, Spark services installed  DDL files copied to the base cluster Steps Using beeline Kinit with hive user to base cluster host where Zookeeper is installed. We can get Zookeeper using CM client services.Example: Execute beeline command on Zookeeper host. Sample beeline URL: beeline -f {data.sql} -u 'jdbc:hive2://{zookeeper_server}:2181/default;" \ f"serviceDiscoveryMode=zooKeeper;ssl=true;sslTrustStore={crt.jks};trustStorePassword={password} Using Impala shell   Kinit with impala user to base cluster host where Zookeeper/impala-server is installed. We can get Zookeeper, impala server using CM client services  Execute impala-shell command on Zookeeper/Impala host. Sample impala-shell command:  impala-shell -i {impala_server}:25003 -d default -k --ssl -f {data}.sql

I often encounter the following question - "How to connect a Java application to CDP Kafka?". The good news is, it's simple.   Public docs: Connecting Kafka clients to Data Hub provisioned clusters   CDP Kafka, by default, is secure and governed. That means CDP Kafka uses TLS and CDP workload users for security and authorization. CDP Kafka supports all Apache Kafka authentication capabilities. Again, by default, CDP Kafka is automatically secured <SASL_SSL> and governed by Apache Ranger.   Prerequisites CDP Public Cloud with a Kafka DataHub instance. Connectivity requirements To connect to CDP Kafka, the following is required: CDP TrustStore CDP Workload Username/Password Kafka Broker Name JaaS File Java Example   Properties properties = new Properties(); properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "<KAFKA-BROKER>:9093,<KAFKA-BROKER>:9093,<KAFKA-BROKER>:9093"); properties.put(ProducerConfig.KEY_SERIALI