Community Articles

In part 1 of this series we talked about the growing relevance of streaming technologies and covered the need for existing Cloudera customers currently using Apache Flume, to consider moving over to Cloudera DataFlow (CDF).   Cloudera DataFlow is an umbrella term that covers the streaming technologies from Cloudera. CDF is supported on CDH 5 / CDH 6 and HDP 2 / HDP 3. So there is nothing stopping customers adopting Cloudera DataFlow right now so that they are in a supported configuration for when they upgrade to the new Cloudera Data Platform (CDP).   CDF includes the technology to address a number of areas: Edge Flow Management Core Flow Management Stream Processing Streaming Analytics Enterprises Services A good summary of these components can be found in the blog post Introducing Cloudera Data Flow.   Cloudera DataFlow - Data-In-Motion Platform   If you are a traditional Cloudera customer using the Cloudera Distribution of Apache Kafka, there are a number of

Cloudera Data Platform recently introduced Cluster Connectivity Manager (CCM) in tech preview, a feature with which CDP can communicate with workload clusters without the need for inbound network connections to customer workload clusters.   In this article, I will highlight how to automate the setup of CCM for both AWS and Azure including: Cloud Network Setup CDP Environment setup   Before you continue reading: You can find all automation on my github (here) Please refer to the official documentation for more details (here) This is just an example, different deployments require a different set of parameters! Overview   As highlighted in the above image, CCM uses reverse tunneling to communicate back to your CDP dedicated control plane. A few things to note: Each tenant (i.e. customer) will have their own dedicated tunnel using a dedicated Apache Mina server The diagram above describes a setup in AWS. The Azure setup is very similar but does not require public sub

COD - CDE using PhoenixIn this article, we will walk through steps required to be followed to read/ write using Phoenix JDBC to COD (Cloudera Operational Database) from Spark on CDE (Cloudera Data Engineering).AssumptionCOD is already provisioned and database cod-db is created. Refer to this link for the same.CDE is already provisioned and virtual cluster is already created. Refer to this link for the same.CODConfigurationSpark in CDE to be able to talk to COD, it would require the hbase-site.xml config of the COD cluster. Do the following steps for retrieve the same:Go to the COD control plane UI and click on cod-db database.Under the "Connect" tab of COD database, look for HBase Client Configuration URL field. Following is the screenshot for the same.The configuration can be downloaded using the following curl command. curl -o "hbase-config.zip" -u "santhosh" "https://XXXXXX.cldr.work/clouderamanager/api/v41/clusters/cod-hvtur2ovawfr/services/hbase/

In this article, we’ll walk through the steps required to connect a Spark Structured Streaming application to Kafka in CDP Data Engineering Experience (DEX). This article extends the official CDP documentation: Connecting Kafka clients to Data Hub provisioned clusters, to include Spark applications run in Cloudera Data Experience.   Steps Obtain the FreeIPA certificate of your environment: From the CDP home page, navigate to Management Console > Environments Locate and select your environment from the list of available environments Click Actions Select Get FreeIPA Certificate from the drop-down menu. The FreeIPA certificate downloads. Add the FreeIPA certificate to a Java keystore: All clients that you want to connect to the Data Hub provisioned cluster, will need to use the certificate to communicate over TLS. The exact steps of adding the certificate to the truststore depends on the platform and key management software used. For example

In this article, we will walk through the steps required to connect a Spark Structured Streaming application to Kafka in CDP Data Hub. We use two Data Hubs, one with a Data Engineering Template, and another with a Streams Messaging template. Both Data Hubs were created in the same environment. This article extends the official CDP documentation: Connecting Kafka clients to Data Hub provisioned clusters to include Spark applications. Steps1. Obtain the FreeIPA certificate of your environment:From the CDP Home Page, navigate to Management Console > EnvironmentsLocate and select your environment from the list of available environmentsClick ActionsSelect Get FreeIPA Certificate from the drop-down menu. The FreeIPA certificate downloads.2. Add the FreeIPA certificate to the truststore of the client.The certificate needs to be added for all clients that you want to connect to the Data Hub provisioned cluster. The exact steps of adding the certi

 Sometimes you need real CDC and you have access to transaction change logs and you use a tool like QLIK REPLICATE or GoldenGate to pump out records to Kafka and then Flink SQL or NiFi can read them and process them.Other times you need something easier for just some basic changes and inserts to some tables you are interested in receiving new data as events. Apache NiFi can do this easily for you with QueryDatabaseTableRecord. you don't need to know anything but the database connection information, table name and what field may change. NiFi will query, watch state and give you new records. Nothing is hardcoded, parameterize those values and you have a generic Any RDBMS to Any Other Store data pipeline. We are reading as records, which means each FlowFile in NiFi can have thousands of records that we know all the fields, types and schema related information for. This can be ones that NiFi infers the schema or ones we use from a Schema Registry like Cloudera's amazing Open Source Sc

OverviewCDP Public Cloud provides a mechanism for provisioning all of your resources in private subnets (i.e. resources that don't get assigned public IPs). I won't detail the specifics about how each CSP accomplishes this; you can find much more thorough overviews in each CSP's documentation.However, in using this architecture, the problem becomes: how do I interact (UI/API/etc) with these resources if they aren't publicly accessible? In an ideal/ corporate world, some sort of VPN peering would be setup so that merely being on your corporate VPN would allow access to the resources in these subnets.  You may find yourself in the unfortunate situation I do, where, you happen to be using an AWS (or Azure) account that doesn't have the fancy VPN peering enabled. Enter the SOCKS5 proxy. I also won't go over the specifics about how this works, but suffice it to say, we will create a bastion host in a public subnet (with a very narrow security group configuration), create an S

In a previous article, I realized that I saved my flow as a flow file instead of a template, which may make it hard to import for some. So in this article, I will explain how to import a flow to NiFi registry in Datahub (knowing that NiFi registry is secure by default in CDP Datahub).Step 1: Download and configure storesConnect to one of the NiFi machines with the Cloudbreak user and the key you used at deployment: $ ssh -i [path_to_private_key] cloudbreak@[your_nifi_host]​​​ Next, copy and authorize the key and trust stores: $ sudo su$ cp /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_keystore.jks /tmp$ cp /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks /tmp$ chmod a+rw /tmp/cm-auto-host_keystore.jks$ chmod a+rw /tmp/cm-auto-global_truststore.jks​​ Step 2: Create a registry.properties fileGo to /tmp (for instance) and create the following file - registry.properties file: baseUrl=https://fod-nifi-cluster-gateway0.fod-cdp.a465

Smart Stocks with FLaNK (NiFi, Kafka, Flink SQL)I would like to track stocks from IBM and Cloudera frequently during the day using Apache NiFi to read the REST API. After that I have some Streaming Analytics to perform with Apache Flink SQL and I also want permanent fast storage in Apache Kudu queried with Apache Impala. Let's build that application cloud native in seconds in AWS or Azure.Source Code:  https://github.com/tspannhw/SmartStocksTo script Loading Schemas, Tables, Alerts, see scripts/setup.sh:Source Code:  https://github.com/tspannhw/ApacheConAtHome2020Kafka TopicKafka SchemaKudu TableFlink PrepFlink SQL Client RunFlink SQL Client ConfigurationOnce our automated admin has built our Cloud environment and populated it with the goodness of our app, we can bring out continuous SQL. If you know your data, build a schema, share to the registry One unique thing we added was a default value in our Avro schema and making it a logicalT

Here we demonstrate how to run YCSB workloads for HBase performance testing. The YCSB benchmarking toolkit is available here. Download YCSB. To download YCSB > runcurl -O --location https://github.com/brianfrankcooper/YCSB/releases/download/0.17.0/ycsb-0.17.0.tar.gz Untar YCSB. Untar the downloaded file:tar xfvz ycsb-0.17.0.tar.gz Create HBase table. Run HBase shell and create table usertable. To open Shell, use command:hbase shell Create 'usertable', 'cf' hbase(main):001:0> n_splits = 200 # HBase recommends (10 * number of regionservers) hbase(main):002:0> create 'usertable', 'cf', {SPLITS => (1..n_splits).map {|i| "user#{1000+i*(9999-1000)/n_splits}"}} Set Java Home. Check if JAVA_HOME is already set usingecho $JAVA_HOME If Java home is not set, YCSB will give you an error. In Cloudera, path to jre could be at /usr/java/default. Once you find the path, you can set JAVA_HOME by using the following: export JAVA_HOME=<path-to-jre

Post based on KB article:As soons as you realize that you have accidentally deleted the the files/directories in HDFS, immediately force the name node to go in SafeMode (hadoop namenode -safemode enter) to avoid further damage to file system. If possible, shutdown the NN and DataNodesCheck out the core-site.xml, to verify if Trash is enabled (look for configuration parameter ‘fs.trash.interval’. The value greater than zero value indicates number or minutes that data will be kept in Trash)If the number is more than the duration between delete command execution and namenode shutdown/safe mode, then you can recover the files from user’s Trash folder. If Trash is not enabled, contact Hortonworks support immediately. DO NOT restart the namenode and/or bring it out of safe mode. Doing so may delete your data permanently.

In this video, we'll briefly discuss what is Hive locking mechanism, what are the type of locks and how they work. We will also demonstrate with the help of a query about validation on the locks and removal of obsolete Hive locks from backend. Query used: CREATE TABLE ABORTED_TXNS AS SELECT A.TXN_ID AS HL_TXNID FROM TXNS A , HIVE_LOCKS B WHERE A.TXN_ID=B.HL_TXNID AND A.TXN_STATE='a'; DELETE FROM HIVE_LOCKS WHERE HL_TXNID IN (SELECT HL_TXNID FROM ABORTED_TXNS); Reference Link: https://cwiki.apache.org/confluence/display/Hive/Locking     Open YouTube video here

Summary:This article provides the steps for using ConsumeJMS and PublishJMS processors with Weblogic Topics and Queues.Prerequisites:In this article a Weblogic 12c server was used with HDF-3.3.0. Also, for this article the JMS Server, Connection Factory, Queue and Topic have already been added to the Weblogic server. These blogs were used as references for this article, Install Weblogic 12.1.3 server, Basic JMS Demo using WebLogic Queue. Much of the credit for the information here goes to @emattos and @Jay Kumar SenSharma, both provided a great deal of assistance in understanding the Weblogic server and sample java programs for testing.Also, for this you will need the wlfullclient.jar, this can be created by following this guide: Using the WebLogic JarBuilder ToolNote: The JndiJmsConnectionFactoryProvider controller service was introduced in HDF-3.3.0. So, this controller service isn’t available in any earlier versions of HDF.NiFi Setup:In this article we'll use a simple flow to demon

Please follow the steps below to install ambari-shell on your system -Environment setup details -Virtual Box : 5.0.14 OS : CentOS release 6.7 (64-bit) Ambari-shell code : https://github.com/sequenceiq/ambari-shell.git Gradle Version: gradle-1.12 [ https://services.gradle.org/distributions/gradle-2.11-all.zip ] Java Version: 1.7.0_79Installation steps:1.Login using root/superuser 2.Install "git" package using rpm/yum command# yum install -y git[Note: Make sure you have internet access to your system and /etc/yum.repos.d/CentOS-Base.repo in place unless you have local repository configured to download packages] 3.Download Gradle from the link given above and unzip it in a directory. Here i will download code in /opt directory# cd /opt # wget https://services.gradle.org/distributions/gradle-2.11-all.zip # unzip # cd /opt# wget https://services.gradle.org/distributions/gradle-2.11-all.zip# unzip gradle-2.11-all.zip4. Export gradle home path# export PATH=$PATH:/opt/gradle-2.11/bin5. Make su

Experimental: Apache Hadoop Distributed File System (HDFS) is a popular file system in big data world. The HDFS federation provided the flexibility to configure multiple namenodes and improves the overall scalability of the cluster. The ViewFileSystem was introduced to access the federated HDFS cluster in a unified way. However, most of the existing HDFS users did not migrate to it because of the following reasons. It is a new scheme and users have to update their path URIs (for example, Hive stores fully qualified path URIs in its metastore). Also, the mount-table configurations need to be copied to all client nodes. To solve these problems, we enhanced ViewFS and introduced the ViewDistributedFileSystem. Now, users can initialize the filesystem with scheme "hdfs" and configure the mount points. We also introduced central mount-table reading functionality. Users can upload the mount-table configurations into Hadoop compatible file system (for example, HDFS) and configure the pat

Final article of the hybrid cloud series (see parent article here), and it's the funnest one! In this tutorial, we will learn to use Cloudera Viz to create visual apps. We will not go in depth into how to use viz, rather we will import an already existing app, executing on this part of the tutorial flow:PrerequisitesComplete Part 3 of the tutorial series.Step 1: Import Viz App Navigate to your Management console > Data Warehouse > Open Cloudera Viz:You will be redirected to the default examples; Navigate to the DATA tab:Then, click on Default Hive VW (this is your CDW VW):Then, click on Import Visual Artifacts:Finally, import the viz app (link here) as follows:After clicking on Import, you will see a new app:Step 2: Update each maps with Mapbox tokenIn order for the maps to display, you will have to add a mapbox token. I will only detail one of the three dashboard edit here, but you will need to do it for each dashboard.For this, first create an account at https://accou

In HDP 2.x we had hbck tool which was used to identify and fix inconsistencies in hbase tables. However with HBase 2.x and HDP 3.x together this tool is deprecated and its strongly advised to not run it on production clusters as it may lead to data corruption.HBase 2.0 introduces a completely revamped HBase Assignment Manager with Proc V2 framework (State machine). In HDP-2.x, we maintained HBASE_REGION_STATE and TABLE_STATE in zookeeper and Hmaster in-memory ( possibility for inconsistencies) but with HDP-3 it uses MasterProcWAL procedure which is stored in HDFS.These set of procedures are now scheduled and executed by HMaster.Given the changes on internals of new AssignmentManager , the old hbck fix options no longer work, but it can still provide a report about tables/regions states. To fix regions assignment problems that may still occur, a new HBCK 2 has been designed, and shipped independently from hbase build artifacts (and hence not part of HDP 3.x distribution ), as a plan to

Welcome to Part 3 of my article series on how to harness the hybrid cloud series (see parent article here). In this tutorial you will learn to use NiFi to:Pull information from public APIsPush this raw data to secure S3 bucket using SDX integrationCreate Hive tables on top of this data by connecting to CDWThis corresponds to step 3 of the series, as explained below:Note: The anonymized NiFi flow can be found here.PrerequisitesComplete Part 2 of this article seriesA NiFi Datahub in the environment you created for Part 1 and Part 2Step 1: Prepare your NiFi Datahub for JDBC connection to Hive in CDWDownload driver jarNavigate to your Management console > Data Warehouse > find your virtual warehouse and download the driver jar:Upload jar to NiFi nodesNavigate to the Management console > your NiFi datahub > Hardware and note the public IPs of the NiFi nodes:Using these public IPs (here hidden), upload the downloaded jar to all three nodes (see example code below, using your work

What is the Role of Journal nodes in Namenode HA ?I know many of us are aware that Role of Journal nodes is to keep both the Namenodes in sync and avoid hdfs split brain scenario by allowing only Active NN to write into journals. Have you ever wonder how does it works? Here you go!Journal nodes are distributed system to store edits. Active Namenode as a client writes edits to journal nodes and commit only when its replicated to all the journal nodes in a distributed system. Standby NN need to read data from edits to be in sync with Active one. It can read from any of the replica stored on journal nodes.ZKFC will make sure that only one Namenode should be active at a time. However, when a failover occurs, it is still possible that the previous Active NameNode could serve read requests to clients, which may be out of date until that NameNode shuts down when trying to write to the JournalNodes. For this reason, we should configure fencing methods even when using the Quorum Journal Manager

SUMMARY: How to enable Performance logging in Atlas, where we can track each search time like time taken to get any entities or to get linegae info. It helps while debugging if Atlas UI or API is taking long time to get results. We can check which phase is taking long time and debug accordingly. Example :2016-12-20 14:24:02,344|qtp1381713434-59648 - ce3e660e-bdcb-4656-805d-7a99d0b9ddb6|PERF|EntityResource.getEntityDefinition()|4522016-12-20 14:24:02,432|qtp1381713434-59901 - d15c9039-945a-4a87-abf2-017fdde22ad6|PERF|EntityResource.getEntityDefinition()|62016-12-20 14:24:02,553|qtp1381713434-59893 - d9624e31-6c1f-4900-8269-e9f14dfb0a09|PERF|EntityResource.getAuditEvents(03b90ea3-a307-4cfd-ba93-79a2a7cbadf8, null, 26)|1172016-12-20 14:24:02,643|qtp1381713434-59896 - 775b3108-49e4-4c69-af65-b028a21b26b3|PERF|LineageResource.schema(03b90ea3-a307-4cfd-ba93-79a2a7cbadf8)|2072016-12-20 14:24:03,176|qtp1381713434-59894 - 98047e2d-181b-4a41-bdf1-4d273a4cc7a3|PERF|LineageResource.inputsGraph(03b

This article contains Questions & Answers on Cloudera Data Engineering.   Is Cloudera Data Engineering integrated with Cloudera Data Warehouse (CDW) and Cloudera Machine Learning (CML)? Yes, through SDX generated datasets from your Data Engineering pipelines will automatically be accessible in downstream analytics like Data Warehousing and Machine Learning.  In additionally all the benefits of SDX including lineage to secure your data pipelines across your enterprise.    I already have Cloudera Machine Learning (CML) that has Spark capabilities. How is this better? Cloudera Data Engineering is tailor built for data engineers to operationalize their data pipelines. ML is tailored to data scientists who want to develop and operationalize their ML models.  Both services are fully integrated with each other and seamlessly interoperable for however you want to run your data engineering and data science workflows. Because they’re both included with CDP, there’s

What happened?   Starting up a ZooKeeper server in a Kerberized CDP-DC 7.0.3 cluster failed with the logs below.    2020-03-30 12:23:10,251 ERROR org.apache.zookeeper.server.quorum.QuorumPeerMain: Unexpected exception, exiting abnormallyjava.io.IOException: Could not configure server because SASL configuration did not allow the  ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: Message stream modified (41)        at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:243)        at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)        at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:148)        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)        at org.apache.zo

A quick way to get started with CDP and Cloudera Data Warehouse (CDW) is to hydrate your environment with some existing relational tables you have. In this article, we'll walk through ingesting existing relational tables into CDP, creating table definitions on top of this data, and then briefly explore the question of "now what?"   A quick note: In this article, we'll use Apache Sqoop to do some full table copies as well as some simple selects/copies based on a where condition. If you plan on doing anything more complex or have thoughts of setting up something more permanent, you may want to consider another solution (like NiFi). Special thanks to @pvidal for the initial dataset and RDS creation automation. Determine the Source Data I have created an Oracle database in one of my AWS accounts. Keep in mind, whatever your source data, you'll need to make sure that proper ingress is enabled in your CDP environment. I'll skip showing the specifics for my case but suffice to

Generally we are used to use chown/chmod to change permissions. When we try chown/chmod on a directory which contains some x million objects it takes very long sometime even days. So to reduce the time and make the changes in one command instead of 2 chown and chmod, you can use DistCh faster then regular chown and chmod. Below is the command to use DistChhadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-extras.jar org.apache.hadoop.tools.DistCh java org.apache.hadoop.tools.DistCh [OPTIONS] <path:owner:group:permission>The values of owner, group and permission can be empty.Permission is a octal number.OPTIONS:-f <urilist_uri> Use list at <urilist_uri> as src list-i Ignore failures-log <logdir> Write logs to <logdir>Generic options supported are-conf <configuration file> specify an application configuration file-D <property=value> use value for given property-fs <local|namenode:port&g

Welcome to Part 2 of our harness the hybrid cloud series. In this tutorial, we will learn how to use Data Catalog, Atlas, and Ranger to profile and protect sensitive data in CDP Public Cloud, as depicted below: Prerequisites Complete Part 1 of the series Step 1: Launch Data Profiling CDP Data Catalog comes with data profilers out of the box. You can of course customize them, but in our datasets, we will use the standard data profilers. Launch Profiler Cluster Navigate to your CDP Management Console > Data Catalog > Select your environment > Launch Profilers: This will launch a datahub cluster to run the data profiling spark jobs. Wait for the cluster to be built, like in the following screenshot: Verify Profiler execution Navigate back to your Data Catalog > Profilers > Select your env > Cluster Sensitivity Profiler, and verify that profilers have run successfully: Check profiled data Go to Search and find the employees Hive table: In the employees table,