Resources

SYNOPSIS The Optimized Row Columnar (ORC) file is a columnar storage format for Hive. Specific Hive configuration settings for ORC formatted tables can improve query performance resulting in faster execution and reduced usage of computing resources. Some of these settings may already be turned on by default, whereas others require some educated guesswork. The table below compares Tez job statistics for the same Hive query that was submitted without and with certain configuration settings. Notice the performance gains with optimization. This article will explain how the performance improvements were achieved. QUERY EXECUTION Source Data: 102,602,110 Clickstream page view records across 5 days of data for multiple countries Table is partitioned by date in the format YYYY-MM-DD. There are no indexes and table is not bucketed. The HiveQL is ranking each page per user by how many times the user viewed that page for a specific date and within the United

This short example outlines how to configure Cloudera Machine Learning to integrate with the Data lake, Hive, and Iceberg tables.Apache Iceberg has a number of qualities that make it very suitable for Machine Learning. It maintains the metadata for the data layout alongside the data; this means there are no heavy network demands on the main catalog. The main catalog in this case Hive Metastore maintains a lightweight set of references to the data on the Data lake.   Iceberg also supports snapshots and time-travel so that data can be versioned and queried at a specific point in time.  Enable parallel processing in Spark   Setup user credentials and default paths to the Data lake.spark-defaults.confspark.executor.instances 2spark.executor.memory 1gspark.executor.cores 2spark.hadoop.yarn.resourcemanager.principal christopherroylesspark.yarn.access.hadoopFileSystems s3a://demo-aws-go02/ Setup a Spark session Load the appropriate Iceberg JAR Add the required SQ

Cloudera has added a StreamNative community-supported processor. Reading OpenSea NFT with Apache NiFi, Apache Pulsar, and Friends. Meetup. StreamNative   Upcoming Meetup Demo Apache Pulsar and Apache NiFi for Cloud Data Lakes Pulsar Build bin/pulsar-admin topics delete persistent://public/default/nftbin/pulsar-admin topics delete persistent://public/default/cryptobin/pulsar-admin topics create persistent://public/default/nftbin/pulsar-admin topics create persistent://public/default/cryptobin/pulsar-client consume "persistent://public/default/crypto" -s "nftreadercrypto" -n 0bin/pulsar-client consume "persistent://public/default/nft" -s "nftyreader" -n 0bin/pulsar-admin schemas get "persistent://public/default/nft" Apache NiFi Flows NiFi -> Pulsar Flow Example NFT JSON Record { "date" : "Thu, 24 Feb 2022 22:26:41 GMT", "short_description" : "", "featured" : "false", "image_thumbnail_url" : "https://lh3.googleusercontent.com/-dsYkPkvAQW6KvP0Nnbjecb4Pqo1RsLvTlKKDdSPiUR4LF

Running Livy on HDP 2.5Ingest Metrics REST API From Livy with Apache NiFi / HDFUse GetHTTP To Ingest The Status On Your Batch From LivyRunning LivyThe first step, we download Livy from github. To install on HDP 2.5, is simple. I found a node that wasn't too busy and put the project their. To run, it's simple:export SPARK_HOME=/usr/hdp/current/spark-client/ export HADOOP_CONF_DIR=/etc/hadoop/conf nohup ./livy-server &That's it, you have a basic unprotected Livy instance running. This is important, there's no security on there. You should either put Knox in front of this or enable Livy's security.I wanted to submit a Scala Spark Batch Job. So I wrote a quick one below to have something to call.Source Code for Example Spark 1.6.2 Batch Application: https://github.com/tspannhw/links https://community.hortonworks.com/repos/73816/links-spark-batch-application-example.html?shortDescriptionMaxLength=140Step 1: GetFileStore File /opt/demo/sparkrun.js with JSON to trigger Spark thr

Cloudera Machine Learning now offers Snippet to connect to Data Sources available within the CDP Environment. Administrators can configure custom Spark, Hive or Impala Virtual Warehouse data connections manually or they can use CML’s features to autodetect and configure all connections from the same CDP Environment. Data Scientists can then access the preconfigured Data Connections from their ML Projects.   The Data Connection and Snippet support simplifies getting started on ML Projects. Once a project is created, the first time the users create a session they are offered code snippets to create a connection to their selected data store. Users don’t need to look up the connection boilerplate from the documentation or copy an example code from other projects, they can easily initiate the connection via CML’s connection library and immediately start solving their business problems.       To learn more, you can read the How To article in the Cloudera Docs.

DISCLAIMER / CREDITS: The majority of the content of this community article was written by Adam Antal.My piece was correcting, adding, or revising the contents. I would like to thank Benjamin Teke, Andras Gyori, and Rudolf Reti for the reviews of this article.IntroductionThis community article is for Cloudera Data Platform Administrators and will clarify how the recent improvements in YARN logging will benefit their ability to manage and support their customers' workloads. In some circumstances YARN will generate a similar order of magnitude of data as that of the application and that can prove challenging for administrators to make use of for diagnostics. This article will illustrate how administrators can use these new features to streamline problem resolution and will be of benefit to developers that are designing YARN applications.The processes of YARN and its launched containers almost always generate application logs that can be potentially large, depending on the type of th

Zeppelin Best Practices Install & Versions Leverage Ambari to install Zeppelin and always use the latest version of Zeppelin. With HDP 2.6.2, Zeppelin 0.7.2 is available and it contains many useful stability & security fixes that will improve your experience. Zeppelin in HDP 2.5.x has many known issues that were resolved in 2.6.2Deployment Choices While you can select any node type to install Zeppelin, the best place is a gateway node. The reason gateway node makes most sense is when the cluster is firewalled off and protected from outside, users can still see the gateway node.Hardware Requirement More memory & more Cores are betterMemory: Minimum of 64 GB nodeCores: Minimum of 8 cores# of users: A given Zeppelin node can support 8-10 users. If you want more users, you can set up multiple Zeppelin instances. More details in MT section.Security: Like any software, the security depends on threat matrix and deployment choices. This section assumes a MT Zeppelin deployment. Aut

Some time ago I faced an interesting problem with a cluster failing to start after I replaced an MIT KDC with a FreeIPA KDC.   For the replacement, I installed the ipa-client package on the cluster nodes, and then, changed the KDC configuration in Cloudera Manager (CM) (changed realm and KDC details, imported Kerberos user, re-generated credentials, etc..)   The cluster refused to start. Besides that, CM's KDC Login monitor kept complaining about the KDC not being healthy. I could manually kinit successfully, though, and there seemed to be no KDC problems at first glance.   After enabling debug at different places I saw that there were socket timeouts when processes tried to connect to the KDC and that those processes were actually trying to connect to the KDC over UDP, rather than TCP. The UDP requests explained the problem, since UDP traffic was blocked between the cluster and the KDC.   What's strange, though, is that the kr

Hi, How do i change the email in my profile? 

The page Connecting Kafka clients to Data Hub provisioned clusters in the Cloudera documentation explains how to configure the Kafka Console Consumer to connect to a Kafka cluster in DataHub. In this article, I will expand that explanation for a generic Java client and will show how to configure the client to use Schema Registry for the storage and retrieval of schemas.   A full version of the code in this article can be found in GitHub. Client initialization The initialization of the KafkaConsumer or KafkaProducer client is done simply by passing a Properties object to its constructor. The Properties object can be either initialized programmatically or loaded from a properties file. Example: Properties props = new Properties();props.put("bootstrap.servers", "kafka-1.example.com:9093");... initialize other properties ...KafkaConsumer<String, Transaction> consumer = new KafkaConsumer<>(props); The initialization of a KafkaProducer is identical to the above. This con

Objective This tutorial walks you through a NiFI flow that utilizes the QueryRecord processor and Record Reader/Writer controller services to convert a CVS file into JSON format and then query the data using SQL. Note: The record-oriented processors and controller services were introduced in NiFi 1.2.0. As such, the tutorial needs to be done running Version 1.2.0 or later.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.3.0QueryRecord FlowSupport Files Here is a template of the flow discussed in this tutorial: queryrecord-csvtojson.xml Here is the CSV file used in the flow: users.txt (Change the extension from .txt to .csv after downloading) Note: The CSV data originated from the site, RandomUser. This useful site provides a free API to pull down randomly generated user data. For example: https://randomuser.me/api/0.6/?results=10&format=SQL.Demo ConfigurationInput Directory Create a local input direc

A short how-to article on how to list CDP Data hub Kafka topics via command line:PrerequisitesCDP DataHub Kafka clusterWorkload username & passwordCDP PC trust storeStepsFetch CDP cert and store in a trust storeCreating TLS truststoreor simple fetch it from any node  /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks  ssh into any Data hub Kafka node using CDP workload username/passwordCopy trust store onto the nodeSet broker host   BROKER_HOST=$(hostname -f)   Create jaas.conf   KafkaClient {org.apache.kafka.common.security.plain.PlainLoginModule requiredusername="<workload username>"password="<workload password>";};   Set KAFKA_OPTS environment variable to point to location where jaas.conf is located   export KAFKA_OPTS=-Djava.security.auth.login.config=<LOCATION-OF-YOUR-JAASCONF>/jaas.conf   Create client-kerberos.properties 

This article provides the steps and fields required to configure Streams Replication Manager using external accounts between two kerberized clusters. This article assumes that Kerberos is configured properly between the clusters and we can produce and consume data correctly. Environment details: Cluster A: co-located cluster (SRM and Kafka running in this cluster) CDP 7.1.7 Cluster B: External cluster which is another CDP 7.1.7 Both clusters are using  SASL_PLAINTEXT security.protocol for their clients Clusters are configured with cross-domain realm (MIT kerberos), more details on cross-domain realm configuration here Steps to configure external accounts (feature available from CDP 7.1.7) Go to cluster A > Cloudera Manager > Administration > External accounts > Kafka Credentials tab Configure the following fields: Name Bootstrap servers Security protocol JAAS Secret [1-3] JAAS Template Kerberos Service Name SASL Mechanism Example: Name c289 Bootstrap

Spark uses the blacklist mechanism to enhance the scheduler’s ability to track failures. When a task fails on an executor, the blacklist module tracks the executor and host which has failed to execute the task. Beyond a threshold, the scheduler won't be able to schedule any more tasks on that node. If spark.blacklist.enabled is set to true, we need to always set the value of spark.blacklist.task.maxTaskAttemptsPerNode to greater than spark.task.maxFailures, else the Spark job will fail with the following error message: ERROR util.Utils: Uncaught exception in thread mainjava.lang.IllegalArgumentException: spark.blacklist.task.maxTaskAttemptsPerNode ( = 2) was >= spark.task.maxFailures ( = 2 ). Though blacklisting is enabled, with this configuration, Spark will not be robust to one bad node. Decrease spark.blacklist.task.maxTaskAttemptsPerNode, increase spark.task.maxFailures, or disable blacklisting with spark.blacklist.enabled. If you try to trigger the following job from shell of e

Running SQL on stream is super simple with SQL Stream Builder, via Flink SQL. I recorded a live demo and this article is to provide supporting artifacts to run it.   Demo Flink SSB Credit Card Fraud Demo Prerequisite Quick setup Note If you experience any challenges with running SSB, as a sanity check run the following on the cluster     flink-yarn-session -d \ -D security.kerberos.login.keytab=<keytab_filename>.keytab \ -D security.kerberos.login.principal=<workload_username>     If the above works as a sanity check, kill the yarn job.   yarn application -appStates RUNNING -listThat will display all running yarn apps. Your app will show up in this list. Then run this to kill the appyarn application -kill application_xxxx   customer_inferences table   CREATE TEMPORARY TABLE customer_inferences ( `customer_id` INT, `account_number` STRING, `center_inferred_lat` FLOAT, `center_inferred_lon` FLOAT, `max_inferred_distance` STRING,

Prerequisites  Base cluster with Hive on Tez, Zookeeper, Spark services installed  DDL files copied to the base cluster Steps Using beeline Kinit with hive user to base cluster host where Zookeeper is installed. We can get Zookeeper using CM client services.Example: Execute beeline command on Zookeeper host. Sample beeline URL: beeline -f {data.sql} -u 'jdbc:hive2://{zookeeper_server}:2181/default;" \ f"serviceDiscoveryMode=zooKeeper;ssl=true;sslTrustStore={crt.jks};trustStorePassword={password} Using Impala shell   Kinit with impala user to base cluster host where Zookeeper/impala-server is installed. We can get Zookeeper, impala server using CM client services  Execute impala-shell command on Zookeeper/Impala host. Sample impala-shell command:  impala-shell -i {impala_server}:25003 -d default -k --ssl -f {data}.sql

I often encounter the following question - "How to connect a Java application to CDP Kafka?". The good news is, it's simple.   Public docs: Connecting Kafka clients to Data Hub provisioned clusters   CDP Kafka, by default, is secure and governed. That means CDP Kafka uses TLS and CDP workload users for security and authorization. CDP Kafka supports all Apache Kafka authentication capabilities. Again, by default, CDP Kafka is automatically secured <SASL_SSL> and governed by Apache Ranger.   Prerequisites CDP Public Cloud with a Kafka DataHub instance. Connectivity requirements To connect to CDP Kafka, the following is required: CDP TrustStore CDP Workload Username/Password Kafka Broker Name JaaS File Java Example   Properties properties = new Properties(); properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "<KAFKA-BROKER>:9093,<KAFKA-BROKER>:9093,<KAFKA-BROKER>:9093"); properties.put(ProducerConfig.KEY_SERIALI

The purpose of this article is to explain what Process Groups and Remote Process Groups (RPGs) are and how input and output ports are used to move FlowFiles between them. Process groups are a valuable addition to any complex dataflow. They give DataFlow Managers (DFMs) the ability to group a set of processors on to their own imbedded canvas. Remote Process groups allow a DFM to treat another NiFi instance or cluster as just another process group in the larger dataflow picture. Simply being able to build flows on different canvases is nice, but what if I need to move NiFi FlowFiles between these canvases? This is where input and output ports come in to play. They allow you move FlowFiles between these canvases that are either local to a single NiFi or between the canvases of complete different NiFi instances/clusters. Embedded Process Groups: Lets start by talking about the simplest use of multiple embedded canvases through process groups. When you started NiFi for the very first time

Recently I came around an interesting problem: how to use boto to get data from a secure bucket in a Jupyter notebook in Cloudera Machine Learning. The missing piece was: I needed to get my code integrated with my AWS permissions given by IDBroker.Since CML already authenticated me to Kerberos, all I need was getting the goods from IDBroker. In this article, I will show you pseudo code on how to get these access keys both in bash and python.Note: Special thanks to @Former Member to whom I owe this article and many more things.  Find your IDBroker URL Regardless of the method, you will need to get the URL for your IDBroker host. This is done simply in the management console of your datalake. The following is an example: Getting Access Keys in bash After you are connected to one of your cluster's node and ensure you kinit, run the following:IDBROKER_DT="$(curl -s --negotiate -u: "https:/[IDBROKER_HOST]:8444/gateway/dt/knoxtoken/api/v1/token")"

This article describes the method to remove Ranger Plugin status records for unused services. For example, the records are underlined in the following screenshot. There is no menu option to remove such records in the web user interface. The only way to remove them is to manually delete the records from x_plugin_info table directly from the Ranger database.   Once connected to the Ranger database, run the following:select * from x_plugin_info; Identify the record ID for the record you want to remove.delete from x_plugin_info where id = Record_ID;

Many times, Support need to have a look at the Microsoft AD tree for AD internal configuration (e.g. CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration, <domain>). The problem with this part of the AD tree is - it is not visible by default in the AD directory structure. This article will help the reader to explore the hidden AD tree. Any change in AD should be carried out by the customer’s AD team or admin. Instructions Here are the steps: Log on to the AD server Run the ADSIEdit.exe program from Start > Run menu. Select the top-most entry of the left side, and then, select Action > Connect to... from the menu. In the next dialog box, select Select a well known Naming Context radio button, and select Configuration from the drop-down menu. Click OK and ADSI Edit will now show CN=Configuration and its sub-tree. We can now traverse to CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration, <domain> part easily as seen above. Example: For Mic

HiveServer2 is a thrift server which is a thin Service layer to interact with the HDP cluster in a seamless fashion. It supports both JDBC and ODBC driver to provide a SQL layer to query the data. An incoming SQL query is converted to either TEZ or MR job, the results are fetched and send back to client. No heavy lifting work is done inside the HS2. It just acts as a place to have the TEZ/MR driver, scan metadata infor and apply ranger policy for authorization. Configurations :TEZ parameters Tez Application Master Waiting Period (in seconds) -- Specifies the amount of time in seconds that the Tez Application Master (AM) waits for a DAG to be submitted before shutting down. For example, to set the waiting period to 15 minutes (15 minutes x 60 seconds per minute = 900 seconds):tez.session.am.dag.submit.timeout.secs=900 Enable Tez Container Reuse -- This configuration parameter determines whether Tez will reuse the same container to run multiple queries. Enabling this parameter improves

Since Ranger 0.5 there has been the ability to summarize audit events that differ only by timestamp to reduce the amount of events logged in a busy system. When enabled, if a Ranger plugin logs consecutive audit events that differ only by timestamp it will coalesce all such events in to a single event and set 'event_count' to the number of events logged and 'event_dur_ms' to the time difference in milliseconds between the first and last event.To enable this feature you must set the following properties in the Ranger plugin's configuration:Configuration nameNotesxasecure.audit.provider.summary.enabledTo enable summarization set this property to true. This would cause audit messages to be summarized before they are sent to various sinks.By default it is set to false i.e. audit summarization is disabled.xasecure.audit.provider.queue.sizeIf unspecified this value defaults to 1048576, i.e. the queue is sized to store 1M (1024 * 1024) messages.Note the difference in property name that contr

Objective With the release of NiFi Registry 0.2.0, flow contents can now be stored under a Git directory using the new GitFlowPersistenceProvider. This tutorial walks you through how to configure this provider in NiFi Registry so that versioned flows in NiFi are automatically saved to a Git repository. A video version of this tutorial can be seen here: https://youtu.be/kK7eVppg9Aw Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi Registry 0.2.0 Apache NiFi 1.7.1 GitFlowPersistenceProvider Git Configuration First, create a new GitHub repo: then clone it locally using the git clone command (e.g. git clone https://github.com/andrewmlim/versioned_flows.git): Next, go to GitHub’s “Developer settings” and create a new “Personal access token”: NiFi Registry Configuration In the ./conf/providers.xml file, configure the following properties: Set org.apache.nifi.registry.provider.

When you install Cloudera Manager you can configure the mail server you will use with the Alert Publisher. However, if you need to change these settings, you can do so under the Alert Publisher section of the Management Services configuration tab. Through Alert Publisher we can get the alerts for a different type of service status like Bad, Warning, Good. Under the Alert Publisher role of the Cloudera Manager Management Service, you can configure email or SNMP delivery of alert notifications and you can also configure a custom script that runs in response to an alert. The information on how to configure the alert can be found in the official documentation of Cloudera. Please check the provided link [1].However, we will discuss some of the common issues faced while configuring or receiving the alert delivery. Common Issues Over Alert Publisher-> 1- Sometimes we are not able to receive the alert to the SMTP/SNMP server end. While checking the alert-publisher logs, the b