Resources

In a previous article, I realized that I saved my flow as a flow file instead of a template, which may make it hard to import for some. So in this article, I will explain how to import a flow to NiFi registry in Datahub (knowing that NiFi registry is secure by default in CDP Datahub).Step 1: Download and configure storesConnect to one of the NiFi machines with the Cloudbreak user and the key you used at deployment: $ ssh -i [path_to_private_key] cloudbreak@[your_nifi_host]​​​ Next, copy and authorize the key and trust stores: $ sudo su$ cp /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_keystore.jks /tmp$ cp /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks /tmp$ chmod a+rw /tmp/cm-auto-host_keystore.jks$ chmod a+rw /tmp/cm-auto-global_truststore.jks​​ Step 2: Create a registry.properties fileGo to /tmp (for instance) and create the following file - registry.properties file: baseUrl=https://fod-nifi-cluster-gateway0.fod-cdp.a465

Smart Stocks with FLaNK (NiFi, Kafka, Flink SQL)I would like to track stocks from IBM and Cloudera frequently during the day using Apache NiFi to read the REST API. After that I have some Streaming Analytics to perform with Apache Flink SQL and I also want permanent fast storage in Apache Kudu queried with Apache Impala. Let's build that application cloud native in seconds in AWS or Azure.Source Code:  https://github.com/tspannhw/SmartStocksTo script Loading Schemas, Tables, Alerts, see scripts/setup.sh:Source Code:  https://github.com/tspannhw/ApacheConAtHome2020Kafka TopicKafka SchemaKudu TableFlink PrepFlink SQL Client RunFlink SQL Client ConfigurationOnce our automated admin has built our Cloud environment and populated it with the goodness of our app, we can bring out continuous SQL. If you know your data, build a schema, share to the registry One unique thing we added was a default value in our Avro schema and making it a logicalT

Here we demonstrate how to run YCSB workloads for HBase performance testing. The YCSB benchmarking toolkit is available here. Download YCSB. To download YCSB > runcurl -O --location https://github.com/brianfrankcooper/YCSB/releases/download/0.17.0/ycsb-0.17.0.tar.gz Untar YCSB. Untar the downloaded file:tar xfvz ycsb-0.17.0.tar.gz Create HBase table. Run HBase shell and create table usertable. To open Shell, use command:hbase shell Create 'usertable', 'cf' hbase(main):001:0> n_splits = 200 # HBase recommends (10 * number of regionservers) hbase(main):002:0> create 'usertable', 'cf', {SPLITS => (1..n_splits).map {|i| "user#{1000+i*(9999-1000)/n_splits}"}} Set Java Home. Check if JAVA_HOME is already set usingecho $JAVA_HOME If Java home is not set, YCSB will give you an error. In Cloudera, path to jre could be at /usr/java/default. Once you find the path, you can set JAVA_HOME by using the following: export JAVA_HOME=<path-to-jre

Post based on KB article:As soons as you realize that you have accidentally deleted the the files/directories in HDFS, immediately force the name node to go in SafeMode (hadoop namenode -safemode enter) to avoid further damage to file system. If possible, shutdown the NN and DataNodesCheck out the core-site.xml, to verify if Trash is enabled (look for configuration parameter ‘fs.trash.interval’. The value greater than zero value indicates number or minutes that data will be kept in Trash)If the number is more than the duration between delete command execution and namenode shutdown/safe mode, then you can recover the files from user’s Trash folder. If Trash is not enabled, contact Hortonworks support immediately. DO NOT restart the namenode and/or bring it out of safe mode. Doing so may delete your data permanently.

In this video, we'll briefly discuss what is Hive locking mechanism, what are the type of locks and how they work. We will also demonstrate with the help of a query about validation on the locks and removal of obsolete Hive locks from backend. Query used: CREATE TABLE ABORTED_TXNS AS SELECT A.TXN_ID AS HL_TXNID FROM TXNS A , HIVE_LOCKS B WHERE A.TXN_ID=B.HL_TXNID AND A.TXN_STATE='a'; DELETE FROM HIVE_LOCKS WHERE HL_TXNID IN (SELECT HL_TXNID FROM ABORTED_TXNS); Reference Link: https://cwiki.apache.org/confluence/display/Hive/Locking     Open YouTube video here

Summary:This article provides the steps for using ConsumeJMS and PublishJMS processors with Weblogic Topics and Queues.Prerequisites:In this article a Weblogic 12c server was used with HDF-3.3.0. Also, for this article the JMS Server, Connection Factory, Queue and Topic have already been added to the Weblogic server. These blogs were used as references for this article, Install Weblogic 12.1.3 server, Basic JMS Demo using WebLogic Queue. Much of the credit for the information here goes to @emattos and @Jay Kumar SenSharma, both provided a great deal of assistance in understanding the Weblogic server and sample java programs for testing.Also, for this you will need the wlfullclient.jar, this can be created by following this guide: Using the WebLogic JarBuilder ToolNote: The JndiJmsConnectionFactoryProvider controller service was introduced in HDF-3.3.0. So, this controller service isn’t available in any earlier versions of HDF.NiFi Setup:In this article we'll use a simple flow to demon

Please follow the steps below to install ambari-shell on your system -Environment setup details -Virtual Box : 5.0.14 OS : CentOS release 6.7 (64-bit) Ambari-shell code : https://github.com/sequenceiq/ambari-shell.git Gradle Version: gradle-1.12 [ https://services.gradle.org/distributions/gradle-2.11-all.zip ] Java Version: 1.7.0_79Installation steps:1.Login using root/superuser 2.Install "git" package using rpm/yum command# yum install -y git[Note: Make sure you have internet access to your system and /etc/yum.repos.d/CentOS-Base.repo in place unless you have local repository configured to download packages] 3.Download Gradle from the link given above and unzip it in a directory. Here i will download code in /opt directory# cd /opt # wget https://services.gradle.org/distributions/gradle-2.11-all.zip # unzip # cd /opt# wget https://services.gradle.org/distributions/gradle-2.11-all.zip# unzip gradle-2.11-all.zip4. Export gradle home path# export PATH=$PATH:/opt/gradle-2.11/bin5. Make su

Experimental: Apache Hadoop Distributed File System (HDFS) is a popular file system in big data world. The HDFS federation provided the flexibility to configure multiple namenodes and improves the overall scalability of the cluster. The ViewFileSystem was introduced to access the federated HDFS cluster in a unified way. However, most of the existing HDFS users did not migrate to it because of the following reasons. It is a new scheme and users have to update their path URIs (for example, Hive stores fully qualified path URIs in its metastore). Also, the mount-table configurations need to be copied to all client nodes. To solve these problems, we enhanced ViewFS and introduced the ViewDistributedFileSystem. Now, users can initialize the filesystem with scheme "hdfs" and configure the mount points. We also introduced central mount-table reading functionality. Users can upload the mount-table configurations into Hadoop compatible file system (for example, HDFS) and configure the pat

Final article of the hybrid cloud series (see parent article here), and it's the funnest one! In this tutorial, we will learn to use Cloudera Viz to create visual apps. We will not go in depth into how to use viz, rather we will import an already existing app, executing on this part of the tutorial flow:PrerequisitesComplete Part 3 of the tutorial series.Step 1: Import Viz App Navigate to your Management console > Data Warehouse > Open Cloudera Viz:You will be redirected to the default examples; Navigate to the DATA tab:Then, click on Default Hive VW (this is your CDW VW):Then, click on Import Visual Artifacts:Finally, import the viz app (link here) as follows:After clicking on Import, you will see a new app:Step 2: Update each maps with Mapbox tokenIn order for the maps to display, you will have to add a mapbox token. I will only detail one of the three dashboard edit here, but you will need to do it for each dashboard.For this, first create an account at https://accou

In HDP 2.x we had hbck tool which was used to identify and fix inconsistencies in hbase tables. However with HBase 2.x and HDP 3.x together this tool is deprecated and its strongly advised to not run it on production clusters as it may lead to data corruption.HBase 2.0 introduces a completely revamped HBase Assignment Manager with Proc V2 framework (State machine). In HDP-2.x, we maintained HBASE_REGION_STATE and TABLE_STATE in zookeeper and Hmaster in-memory ( possibility for inconsistencies) but with HDP-3 it uses MasterProcWAL procedure which is stored in HDFS.These set of procedures are now scheduled and executed by HMaster.Given the changes on internals of new AssignmentManager , the old hbck fix options no longer work, but it can still provide a report about tables/regions states. To fix regions assignment problems that may still occur, a new HBCK 2 has been designed, and shipped independently from hbase build artifacts (and hence not part of HDP 3.x distribution ), as a plan to

Welcome to Part 3 of my article series on how to harness the hybrid cloud series (see parent article here). In this tutorial you will learn to use NiFi to:Pull information from public APIsPush this raw data to secure S3 bucket using SDX integrationCreate Hive tables on top of this data by connecting to CDWThis corresponds to step 3 of the series, as explained below:Note: The anonymized NiFi flow can be found here.PrerequisitesComplete Part 2 of this article seriesA NiFi Datahub in the environment you created for Part 1 and Part 2Step 1: Prepare your NiFi Datahub for JDBC connection to Hive in CDWDownload driver jarNavigate to your Management console > Data Warehouse > find your virtual warehouse and download the driver jar:Upload jar to NiFi nodesNavigate to the Management console > your NiFi datahub > Hardware and note the public IPs of the NiFi nodes:Using these public IPs (here hidden), upload the downloaded jar to all three nodes (see example code below, using your work

What is the Role of Journal nodes in Namenode HA ?I know many of us are aware that Role of Journal nodes is to keep both the Namenodes in sync and avoid hdfs split brain scenario by allowing only Active NN to write into journals. Have you ever wonder how does it works? Here you go!Journal nodes are distributed system to store edits. Active Namenode as a client writes edits to journal nodes and commit only when its replicated to all the journal nodes in a distributed system. Standby NN need to read data from edits to be in sync with Active one. It can read from any of the replica stored on journal nodes.ZKFC will make sure that only one Namenode should be active at a time. However, when a failover occurs, it is still possible that the previous Active NameNode could serve read requests to clients, which may be out of date until that NameNode shuts down when trying to write to the JournalNodes. For this reason, we should configure fencing methods even when using the Quorum Journal Manager

SUMMARY: How to enable Performance logging in Atlas, where we can track each search time like time taken to get any entities or to get linegae info. It helps while debugging if Atlas UI or API is taking long time to get results. We can check which phase is taking long time and debug accordingly. Example :2016-12-20 14:24:02,344|qtp1381713434-59648 - ce3e660e-bdcb-4656-805d-7a99d0b9ddb6|PERF|EntityResource.getEntityDefinition()|4522016-12-20 14:24:02,432|qtp1381713434-59901 - d15c9039-945a-4a87-abf2-017fdde22ad6|PERF|EntityResource.getEntityDefinition()|62016-12-20 14:24:02,553|qtp1381713434-59893 - d9624e31-6c1f-4900-8269-e9f14dfb0a09|PERF|EntityResource.getAuditEvents(03b90ea3-a307-4cfd-ba93-79a2a7cbadf8, null, 26)|1172016-12-20 14:24:02,643|qtp1381713434-59896 - 775b3108-49e4-4c69-af65-b028a21b26b3|PERF|LineageResource.schema(03b90ea3-a307-4cfd-ba93-79a2a7cbadf8)|2072016-12-20 14:24:03,176|qtp1381713434-59894 - 98047e2d-181b-4a41-bdf1-4d273a4cc7a3|PERF|LineageResource.inputsGraph(03b

This article contains Questions & Answers on Cloudera Data Engineering.   Is Cloudera Data Engineering integrated with Cloudera Data Warehouse (CDW) and Cloudera Machine Learning (CML)? Yes, through SDX generated datasets from your Data Engineering pipelines will automatically be accessible in downstream analytics like Data Warehousing and Machine Learning.  In additionally all the benefits of SDX including lineage to secure your data pipelines across your enterprise.    I already have Cloudera Machine Learning (CML) that has Spark capabilities. How is this better? Cloudera Data Engineering is tailor built for data engineers to operationalize their data pipelines. ML is tailored to data scientists who want to develop and operationalize their ML models.  Both services are fully integrated with each other and seamlessly interoperable for however you want to run your data engineering and data science workflows. Because they’re both included with CDP, there’s

What happened?   Starting up a ZooKeeper server in a Kerberized CDP-DC 7.0.3 cluster failed with the logs below.    2020-03-30 12:23:10,251 ERROR org.apache.zookeeper.server.quorum.QuorumPeerMain: Unexpected exception, exiting abnormallyjava.io.IOException: Could not configure server because SASL configuration did not allow the  ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: Message stream modified (41)        at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:243)        at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)        at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:148)        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)        at org.apache.zo

A quick way to get started with CDP and Cloudera Data Warehouse (CDW) is to hydrate your environment with some existing relational tables you have. In this article, we'll walk through ingesting existing relational tables into CDP, creating table definitions on top of this data, and then briefly explore the question of "now what?"   A quick note: In this article, we'll use Apache Sqoop to do some full table copies as well as some simple selects/copies based on a where condition. If you plan on doing anything more complex or have thoughts of setting up something more permanent, you may want to consider another solution (like NiFi). Special thanks to @pvidal for the initial dataset and RDS creation automation. Determine the Source Data I have created an Oracle database in one of my AWS accounts. Keep in mind, whatever your source data, you'll need to make sure that proper ingress is enabled in your CDP environment. I'll skip showing the specifics for my case but suffice to

Generally we are used to use chown/chmod to change permissions. When we try chown/chmod on a directory which contains some x million objects it takes very long sometime even days. So to reduce the time and make the changes in one command instead of 2 chown and chmod, you can use DistCh faster then regular chown and chmod. Below is the command to use DistChhadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-extras.jar org.apache.hadoop.tools.DistCh java org.apache.hadoop.tools.DistCh [OPTIONS] <path:owner:group:permission>The values of owner, group and permission can be empty.Permission is a octal number.OPTIONS:-f <urilist_uri> Use list at <urilist_uri> as src list-i Ignore failures-log <logdir> Write logs to <logdir>Generic options supported are-conf <configuration file> specify an application configuration file-D <property=value> use value for given property-fs <local|namenode:port&g

Welcome to Part 2 of our harness the hybrid cloud series. In this tutorial, we will learn how to use Data Catalog, Atlas, and Ranger to profile and protect sensitive data in CDP Public Cloud, as depicted below: Prerequisites Complete Part 1 of the series Step 1: Launch Data Profiling CDP Data Catalog comes with data profilers out of the box. You can of course customize them, but in our datasets, we will use the standard data profilers. Launch Profiler Cluster Navigate to your CDP Management Console > Data Catalog > Select your environment > Launch Profilers: This will launch a datahub cluster to run the data profiling spark jobs. Wait for the cluster to be built, like in the following screenshot: Verify Profiler execution Navigate back to your Data Catalog > Profilers > Select your env > Cluster Sensitivity Profiler, and verify that profilers have run successfully: Check profiled data Go to Search and find the employees Hive table: In the employees table,

 [FLaNK] Smart Weather Applications with Flink SQL    Sometimes you want to acquire, route, transform, live query, and analyze all the weather data in the United States while those reports happen. With FLaNK, it's a trivial process to do.   From Kafka to Kudu for Any Schema of Any Type of Data - No Code, Two Steps The Schema Registry has full Swagger-ized Runnable REST API Documentation. Integrate, DevOps, and Migration in a simple script.  Here's your schemas, upload, edit, and compare. Validating Data Against a Schema With Your Approved Level of Tolerance. You want extra fields allowed, you got it: Feed that data to beautiful visual applications running in Cloudera Machine Learning. You like drill-down maps, you got them: Query your data fast with Apache Hue against Apache Kudu tables through Apache Impala: Let's ingest all the US weather stations even though they are a zipped directory of a ton of XML files: Weather Ingest is Easy

 Recently I ran into a scenario requiring to connect my Spark Intellij IDE to Kafka DataHub.   I'm not going to claim the status of a pro at IDE secure setup.  Therefore for novices in the security realm alike, they may find this article useful This article will go through steps setting up an Spark Scala IDE (Intellij)  (with a supplied working code example) to connect securely to a Kafka DataHub over SASL_SSL protocol using PLAIN SASL mechanism.Artifactshttps://github.com/sunileman/spark-kafka-streamingScala Objecthttps://github.com/sunileman/spark-kafka-streaming/blob/master/src/main/scala/KafkaSecureStreamSimpleLocalExample.scalaThe scala object accepts 2 inputsTarget Kafka topicKafka broker(s)Prequequites Kafka DataHub InstancesPermission setup on Ranger to be able to read/write from KafkaIntellij (or similar) with the Scala plugin installedWorkload username and passwordTrustStoreAndre Sousa Dantas De Araujo did a great job explaining (very simply

Objective:- Atlas, by default comes with certain types for Hive, Storm, Falcon etc. However, there might be cases where you would like to capture some custom metadata in Atlas. This can be metadata related to ETL processes, enterprise-operations etc. The article explains how to create custom Atlas types and provides some insight on establishing lineage between said types. Use Case:- Consider a simple Use Case where Raw Textual data is analyzed via a ML process and the results are stored in HDFS. For instance, the raw source data is a dump of access logs on professors and research assistants referring research papers. The ML process would try to come up with recommendations on research papers for further reading for these end users. To capture metadata and lineage for this workflow, we would want to have three custom types in Atlas.a.) ResearchPaperAccessDataset: To capture the metadata for the input dataset. b.) ResearchPaperRecommendationResults: To capture the metadata for t

As explained in this Parent article, here is the first step in our hybrid cloud implementation: replicating bank branches and employee data from an on-prem cluster to CDP Public Cloud.PrerequisitesA CDP Base Cluster (with admin rights and user part of HDFS supergroup)A CDP Public Cloud environment (with admin rights)CDW virtual warehouse attached to Public Cloud environmentNote: you can find the datasets used for this exercise hereStep 1: Register CDP Base as classic clusterStart registrationIn your CDP Public Cloud management console, go to Classic Clusters > Add Cluster > CDH and enter your CDP Base cluster information:You will then see your cluster registration in progress:Install a secure communication tunnelClick on the Files button in Install files and follow the instructions:The following are some example instructions to do this on your CM node.Download from your management console the ssh_tunnel_setup_files.zip archive:Copy it to your CM node  $ s

A continuation article from my IaaS Hadoop performance testing. My previous performance test was on BigStep.ObjectiveTest 1 Terabyte of data using the Tera Suite (TeraGen, TeraSort, and TeraValidate) on similar hardware profiles using core baseline settings across multiple IaaS providing and Hadoop as a Service offerings. Here we will capture EMR performance statistics using EMRFS(s3), an object storage.AWS EMRThe natural next step is to test the Tera suite on AWS EMR which is Amazon's Hadoop as a Service offering. I used EMR with "EMRFS which is an implementation of HDFS which allows EMR clusters to store data on Amazon S3". Object storage with Hadoop has not traditionally performed well. I was very interested in testing the new EMRFS. EMRFS/S3 was chosen as the storage device for this test due to fact much of the s3's (with EMR) allure is around EMR's ability to store and process data directly off s3. Using EMR's local storage (not s3) mayincrease performance. HardwareInstan

I'm back with a new article series like I did previously with news author personality recognition, beast mode quotient, and AI to edge (though this one got recently replaced with identifying Magic: The Gathering cards). In this series, I will showcase how to harness the true power of Cloudera Hybrid cloud capabilities. Throughout the series you will learn how to use CDP Private Cloud Base, Replication Manager, CDP Public Cloud, Nifi, Kafka on data hub, Cloudera Data Warehouse, and Cloudera Viz.Reminder: CDP VisionCDP is designed to seamlessly enable you to deploy any data workloads (data collection, streaming, enrichment, engineering, serving, and AI/ML), on any infrastructure, with the latest engines while maintaining a coherent layer of security and governance (SDX). Case Study: Worldwide BankFor the purpose of this article, I will use an example of a fake bank (Worldwide Bank). Worldwide Bank is a large international bank that leverages a traditional big data architec

A great way to connect Power BI desktop to CDP Public Cloud is using the Cloudera Data Warehouse Experience with Hive LLAP.PrerequisitesCDP Public Cloud environment with Data Warehouse Experience using either Hive LLAP or ImpalaUser will need a CDP login and will need to set up their Workload PasswordFrom their CDP login, they will need the Workload User Name and Workload Password NoteBoth Impala and Hive LLAP Virtual Warehouses will work with the Hive LLAP driver that is included with Power BI.  The steps below will work for both.  It is important to include port 443 when entering the server string in Step 6.Steps to connect Power BI to CDW with Hive LLAPFrom within the DW experience in the CDP Management console, select the Virtual Warehouse you want to use and click the  and select Copy JDBC URL:  From that string, you will only need the bold portion such as the following:jdbc:hive2://hs2-azure-hive-1.env-5iskrk.dwx.cloudera.site/default;trans