The all new Cloudera Data Engineering Experience   I recently had the opportunity to work with Cloudera Data Engineering to stream data from Kafka.  It's quite interesting how I was able to deploy code without much worry about how to configure the back end components.     Demonstration This demo will pull from the Twitter API using NiFi, write to payload to a Kafka topic named "twitter".  Spark Streaming on Cloudera Data Engineering Experience CDE will pull from the twitter topic, extract the text field from the payload (which is the tweet itself) and write back to another Kafka topic named "tweet"   The following is an example of a twitter payload.  The objective is to extract only the text field: What is Cloudera Data Engineering? Cloudera Data Engineering (CDE) is a serverless service for Cloudera Data Platform that allows you to submit Spark jobs to an auto-scaling cluster. CDE enables you to spend more time on your applications, and less time on infrastructure. How do I begin with Cloudera Data Engineering (CDE)? Complete setup instructions here.   Prerequisites Access to a CDE Some understanding of Apache Spark Access to a Kafka cluster In this demo, I use Cloudera DataHub, Streamings Messaging for rapid deployment of a Kafka Cluster on AWS An IDE I use Intellij I do provide the jar later on in this article Twitter API developer Access: https://developer.twitter.com/en/portal/dashboard Setting up a twitter stream I use Apache NiFi deployed via Cloudera DataHub on AWS Source Code I posted all my source code here. If you're not interested in building the jar, that's fine.  I’ve made the job Jar available here. Oc t26, 2020 update - I added source code for how to connect CDE to Kafka DH available here.  Users should be able to run the code as is without need for jaas or keytab.  Kafka Setup This article is focused on Spark Structured Streaming with CDE.  I'll be super brief here Create two Kafka topics twitter This topic is used to ingest the firehose data from twitter API tweet This topic is used post tweet extraction performed via Spark Structured streaming NiFi Setup This article is focused on Spark Structured Streaming with CDE.  I'll be super brief here. Use the GetTwitter processor (which requires twitter api developer account, free) and write to the Kafka twitter topic       Spark Code (Scala) Load up the Spark code on your machine from here: https://github.com/sunileman/spark-kafka-streaming Fire off a sbt clean and package A new jar will be available under target: spark-kafka-streaming_2.11-1.0.jar The jar is available here   What does the code do? It will pull from the source Kafka topic (twitter), extract the text value from the payload (which is the tweet itself) and write to the target topic (tweet) CDE Assuming CDE access is available, navigate to virtual clusters->View Jobs Click on Create Job: Job Details Name Job Name Spark Application File This is the jar created from the sbt package: spark-kafka-streaming_2.11-1.0.jar Another option is to simply provide the URL where the jar available https://sunileman.s3.amazonaws.com/CDE/spark-kafka-streaming_2.11-1.0.jar Main Class  com.cloudera.examples.KafkaStreamExample Arguments arg1 Source Kafka topic: twitter arg2 Target Kafka topic: tweet arg3 Kafka brokers: kafka1:9092,kafka2:9092,kafka3:9092 From here jobs can be created and run or simply created.  Click on Create and Run to view the job run: To view the metrics about the streaming: At this point, only the text (tweet) from the twitter payload is being written to the tweet Kafka topic.   That's it! You now have a spark structure stream running on CDE fully autoscaled.  Enjoy ... View more

  Image Courtesy: k9s I recently ran into a scenario where I needed to gather Hive logs on the new Data Warehouse Experience on AWS.   The "old" way of fetching logs was to SSH into the nodes. Data Warehouse Experience is now deployed on K8s, so SSHing is off the table. Therefore a tool like K9s is key. This is a raw article to quickly demonstrate how to use K9s to fetch Data Warehouse Experience logs which are deployed on AWS K8s Prerequisites Data Warehouse Experience K9s installed on your machine  AWS ARN (instructions provided below) AWS configure (CLI) pointing to your AWS env.  Simply type AWS configure via CLI and point to the correct AWS subscription AWS ARN Your AWS ARN is required to successfully connect K9s to CDW(DW-X) On AWS, go to IAM > Users > Search for your user name: Click on your username to fetch the ARN: Kubeconfig Connecting to DW-X using K9s requires kubeconfig. DW-X makes this available under DW-X-> Environments > Your Environment > Show Kubeconfig.  Click on the copy option and make the contents available within a file in your machine file system.  For example, I stored the kubeconfig contents here:  /Users/sunile.manjee/.k9s/kubeconfig.yml​                 ARN To access K8s from K9s, your ARN will need to be added under Grant Access:                             K9s Now all is set up to connect to DW-X K8s using K9s.   Reference kubeconfig.yml file when using K9s     k9s --kubeconfig /Users/sunile.manjee/.k9s/kubeconfig.yml                         That's it. From here the logs are made available and a ton of other metrics.  For more information on how to use K9s, see k9scli.io ... View more

The EFM (Edge Flow Manager) makes it super simple to write flows for MiNiFi to execute where ever it may be located (laptops, refineries, phones, OpenShift,etc).  All agents (MiNiFi) are assigned an agentClass.  Once the agent is turned on, it will phone home to EFM for run-time instructions. The run-time instructions are set at the Class level.  Meaning all agents within a class, run the same instruction (flow) set. There can be 0 to many Classes.  In this example, I will capture Windows Security Events via MiNiFi and ship them to NiFi over Site2Site   Download MiNiFi MSI and set the classname.  In this example, I set the classname to test6.  This property is set at install time (MSI) or by going directly into minifi.properties. Also, notice the setting nifi.c2.enable=true.  This informs MiNFi that run time flow instructions will be received from EFM.  Start MiNiFi. MiNiFi can be configured to send data to multi endpoint (ie Kafka, NiFi, EventHub, etc). In this example, data will be sent to NiFi over S2S.  On NiFi create an input port: Capture the port ID. This will be used in EFM later on: On EFM, open class test6.  This is where we design the flow for all agents with their class is set to test6: To capture Windows events via MiNiFi, add ConsumeWindowsEventLog processor to the canvas: Configure the process to pull events.  In this example, MiNiFi will listen for Windows Security Events: To send data from MiNiFi to NiFi, add Remote Process Group to the canvas. Provide a NiFi endpoint: Connect ConsumeWindowsEventLog processor to the Remote Process Group. Provide the NiFi Input Port ID captured earlier: Flow is ready to publish: Click on Publish. MiNiFi will phone home at a set interval (nifi.c2.agent.heartbeat.period). Once that occurs, MiNiFi will receive new run time flow instructions.  At that time data will start flowing into NiFi.   The EFM makes it super simple to capture Windows events and universally ship anywhere without the ball and chain of most agent/platform designs.  ... View more