Resources

There will be multiple form factors available in the future, for now, I will assume you have an environment that contains 1 datahub with NiFi, and 1 Data Hub with both Impala and Kudu. (The answer still works if all are on the same datahub).   Prerequisites Data Hubs with NiFi and Impala+Kudu Permission to access these (e.g. add a processor, create table)  Know your Workload User Name (CDP Management Console > Your name (bottom left) > Profile) You should have set your Workload Password in the same location Steps to write data from NiFi to Kudu in CDP Public Cloud Unless mentioned otherwise, I have kept everything to its default settings. In Kudu Data Hub Cluster: Gather the FQDN links of the brokers and the used ports. Go to the Data Hub > Click Kudu Master > Click Masters. Combine the RPC addresses together in the following format: host1:port,host2:port,host3:port Example:```master1fqdn.abc:7051,master2fqdn.abc:7051,master3fqdn.abc:7051```​ In HUE

The Cloudera Data Platform (CDP) comes with a wide variety of tools that move data, these are the same in any cloud as well as on-premises.Though there is no formal decision tree, I will summarize the key considerations from my personal perspective. In short, it can be visualized like this:   Steps for finding the right tool to move your dataStaying within Hive and SQL queries suffice? > Hive otherwiseNo complex operations (e.g. joins) > Nifi otherwiseBatch > Spark otherwiseAlready have Kafka Streams/Spark Streaming in use? > Kakfa Streams/Spark Streaming otherwiseFlinkSome notes:If you can use Nifi or a more complex solution, use NifiUse Flink as your streaming engine, unless there is a good reason not to. It is the latest generation of streaming engines.Currently, I do not recommend using Flink for batch processing yet, but that will likely soon changeI did not include tools like Impala, Hbase/Phoenix, Druid as their main purpose is accessing dataThis is a

When running the YARN service check in a Resource Manager HA environment you see that the service check fails - all other functionality is working correctly (restart of services, running jobs etc)When you run the service check you see:stderr: /var/lib/ambari-agent/data/errors-392.txtTraceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py", line 159, in <module> ServiceCheck().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py", line 130, in service_check info_app_url = params.scheme + "://" + rm_webapp_address + "/ws/v1/cluster/apps/" + application_name File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__ raise Fail("Configuration parameter '"

Co-authored by Balazs Gaspar and @steffen.maerkl - more coming soon   The purpose of this article is to describe how to set up Keycloak as a SAML 2.0 Identity Provider for CDP Public Cloud, enabling any user in your organisation to use single sign-on to access CDP and its services, such as Cloudera Machine Learning or Cloudera Data Warehouse.   Keycloak is a popular open source Identity and Access Management (IAM) solution but of course there are many more such solutions, so see this as one example. In an upcoming Cloudera Community post we plan to provide a similar guide for configuring CDP Public Cloud with other popular SSO solutions.   In the examples below we will create users and groups directly in Keycloak. In a real world deployment it is more likely that you would use Keycloak only for SSO and rely on a separate provider (such as Active Directory) to store users and groups via LDAP federation. If you have not been exposed to SAML 2.0 previously, Okta

This article contains Questions & Answers on Cloudera Operational Database (COD). What is the relationship between Data Hub & Data Lake?Data Lake houses SDX (governance & authorization). Data Hub is the actual service that hosts the workload, in this case, the Operational DB. When should I use Cloudera OpDB vs a template in Data Hub?For new apps use Cloudera OpDB (COD) that is self-tuning and auto-improves performance over time. And for replicating on-prem environments to the cloud via lift and shift or for disaster recovery use Data Hub templates. How does Apache Phoenix relate to Apache HBase?Phoenix is an OLTP SQL engine for OpDB. It adds relational capabilities on top of HBase. Phoenix provides a much more familiar programming paradigm and allows our customers to reach production faster. Think of Phoenix as a SQL persona and HBase as a NoSQL persona. Should I use HBase/Phoenix or Apache Kudu for an operational data store (ODS) / Operational database?

This article contains Questions & Answers on Cloudera Data Warehouse (CDW) demos. Which data warehousing engines are available in CDW?Hive for EDW and complex report building and dashboarding, Impala for interactive SQL and ad hoc exploration, Kudu for time-series and Druid for log analytics. How do you create data warehouses?Step 1, create your CDP environment. Step 2, activate the CDW service. Step 3, create your virtual warehouse. Step 4, define tables, load data, run queries, integrate your BI tool, etc. What’s the relationship between a Database Catalog and Virtual Warehouse?For each database catalog there can be one or more virtual warehouses. But each virtual warehouse is isolated from other warehouses while they share the same data and metadata. What’s the CDW query performance in the cloud when using remote storage?CDW uses different levels of caching to offset object storage access latency. This includes a data cache on each query execution node, a que

This article contains Questions & Answers on Cloudera AI. Is it possible to run CML on premises?Yes, Cloudera Machine Learning is available on both CDP Public Cloud as well as CDP Private Cloud. How is the deployment of models managed by CML Private Cloud vs CML Public Cloud?Model deployment functions similarly across both form factors of CML; the models are built into containerized images, and then deployed on top of Kubernetes pods for production-level serving. Is there a level of programming required for a data scientist to use this platform? What languages can developers use?CML enables data scientists to write code in Python, R, or Scala in their editor of choice. Therefore, beginner data scientists can easily run sample code in the workbench, and more experienced data scientists can leverage open source libraries for more complex workloads. Can you run SQL-like queries? E.g. with Spark SQL?Yes, Spark SQL can be run from CML Do pre-built models come ou

Why is it so difficult to vreate a new post. This was the only page that allowed me to post. Why is that? 

This article contains Questions & Answers on Cloudera DataFlow (CDF). What is the difference between Cloudera’s NiFi and Apache NiFi?Cloudera Flow Management (CFM) is based on Apache NiFi but comes with all the additional platform integration that you’ve just seen in the demo. We make sure it works with CDP’s identity management, integrates with Apache Ranger and Apache Atlas. The original creators of Apache NiFi work for Cloudera. Does NiFi come with CDP public cloud or is it an add on?Any CDP Public Cloud customer can start using NiFi by creating Flow Management clusters in CDP Data Hub. You don’t need additional licenses and you will be charged based on how many instance hours your clusters consume What types of read/write data does NiFi support?NiFi supports 400+ processors with many sources/destinations. Do you support cloud-native data sources and sinks?Yes, we support a lot of cloud-native sources and sinks with dedicated processors for AWS, Azure, and GC

This article contains Questions & Answers on CDP Security and Governance (SDX). Does SDX provide governance for all the data in the cluster that’s in the cloud?Yes. Any cluster that has been built with CDP will have governance applied to it, regardless of whether it’s deployed to a public or private cloud. How do we enable end to end governance on CDP?  Is it different for public cloud and private cloud?End-to-end governance is enabled by default in CDP public cloud and private cloud.  Under the hood, Apache Ranger and Apache Atlas are out-of-the-box wired into all the processing engines in CDP (e.g. Apache Hive, Impala, NiFi, Spark et al.). How do you set up SDX?SDX installation is completed when you provision an environment with wire and at-rest encryption preconfigured. Technical Metadata management functionalities are also set up automatically. Business Metadata and data policies must be implemented according to the customer’s context and requirements.&

This article contains Questions & Answers on Cloudera Data Platform (CDP) - private or public cloud. CLOUDERA DATA PLATFORM Which clouds is CDP public supported on?AWS, Azure and, soon, GCP. What’s the difference between Cloudera’s product and cloud providers?Cloudera Data Platform (CDP) is a Platform-as-a-Service (PaaS) that is cloud infrastructure agnostic and easily portable between multiple cloud providers including private solutions such as OpenShift. CDP is both hybrid and multi-cloud from the ground up which means one platform can serve all data lifecycle use cases, independent of location or cloud, with a unified security and governance model.  How do CDP experiences compare to solutions from other cloud service providers?CDP has an SDX layer that stores all policies and metadata for security and governance. This preservation of state is the big differentiating factor, especially when running transient workloads and a variety of experiences. The SDX

To put Kerberos DEBUG logging for HDF managed Schema Registry ( SR ) clusters, do the following depending on your environment: Navigate to Ambari schema registry > configs > advanced > registry-env template Add the below line:export REGISTRY_OPTS="$REGISTRY_OPTS -Dsun.security.krb5.debug=true"Example: Restart SR. You will then be able to view Kerberos DEBUG messages under /var/log/registry (default).     

In this article, we cover the following:  Demonstrate how hive sessions work Clarify some misunderstanding about hive sessions behavior (most of the people, including myself until recently, believe hive sessions and queue allocation work different than the way they actually work) Propose a solution for small-medium environments (up to 200 nodes) that allows concurrent users + multi-tenancy + soft allocation + preemption   Assumptions: Tested with HDP 2.4.0 I'm using configuration suggested in best-practices and docs related to security and hive performance guide, which include following configurations: hive engine (hive.execution.engine) = tez hive do-as (hive.server2.enable.doAs) = false hive default queues (hive.server2.tez.default.queues) = (queue-name1,queue-name2,etc) hive number of sessions (hive.server2.tez.sessions.per.default.queue) = 1 (or up to 4) hive start sessions (hive.server2.tez.initialize.default.sessions) = true hive pre-warm containers (hive.prewarm.en

When NiFi is secured for TLS server authentication, at UI login time first it tries to use TLS certificates if loaded on the browser and then, it tries to use SPNEGO authentication, and finally, it falls back to your configured login provider. If you KERBERISE the cluster via AMBARI and want to use login-providers like LDAP or KERBEROS, it automatically sets the following properties which enable SPNEGO authentication. nifi.kerberos.spnego.keytab.location nifi.kerberos.spnego.principal    Furthermore, SPNEGO properties through AMBARI are greyed out for: COMMAND: From your Amabari manager host, change the setting for NiFi, where the text in red is tailored to your unique environment: nifi.kerberos.spnego.keytab.location to be blank: ./configs.py -a set -s http -l c2288-node1.squadron.support.hortonworks.com -t 8080 -n c2288 -u admin -p AdminPassword -c nifi-properties -k 'nifi.kerberos.spnego.keytab.location' -v '' nifi.kerberos.spnego.principal to be bl

See the webinar for this Prototype here: https://www.cloudera.com/about/events/webinars/building-diverse-anomaly-detection-capabilities-for-business-with-cdp.html  We now have our second Applied Machine Learning Prototypes available (Go here for the previous Prototype - Customer Churn). These are prototypes (or templates as they were called) that will help you build a fully working machine learning example in CML. The Templates will include source data, and walk through various steps:Ingest data into a useful place in CDP (e.g. a Hive Table)Explore the data setCreate a plan to build a modelTrain the modelDeploy the modelBuild and deploy an applicationOnce you have deployed the template and all the CML artifacts that go with it, you can unpick and work it backward to map the process to your own data in your own environment. Our latest Prototype - Fraud Detection - is now available. To get up and running with it, do the following:Log in to your CML workspace and creat

Hi everyone! Is there a way where I can download all the videos on the Cloudera OnDemand app so that I can view it while commuting, instead having to download 1 by 1? Thanks!

Issues with Global IndexesIn Part 1 and Part 2 of this article series we discussed Index internals and some frequently faced issues. In this article we will cover few more index issues in the form of scenarios.Scenario 4 : Index writes are failing, client retries exhaused, handler pool saturated while Index table regions are in transition.Here client is trying to write to data table on server1 which is triggering index update for server2 (via server-to-server RPC) . If Index region is stuck in transition, the index update RPC would also hang and eventually timeout and because of this the RPCs between client and server1 also gets stuck and timed out. Since client would make several retries to write this mutation, it would again lead to handler saturation on region server one causing another “deadlock” like situation.This should be fixed in two steps: Fix all Index RITs first as without this no client index maintenance or server side index rebuild would succeed.As a holistic tuning, keep

If we are working on some cloud environment like "OpenStack" then we might encounter an issue in which after installing the ambari agents we will see that some of the Quick Links in the ambari console is not having the expected hostname. For example if we navigate to Ambari --> HDFS --> "Resource Links" then we may see the incorrect hostnames. Actually those are detected as "public_host_name" by the ambari "/usr/lib/python2.6/site-packages/ambari_agent/hostname.py" script during every restart.So in case of OpenStack it is possible that the "public_host_name" might be different from the "host_name" (`hostname -f`)Here if we see that the Resource Link will show the links with the hostname "ambari1.example.com" (which is `hostname -f` but it is showing "ambari1.novalocal" hence the links are not working.In such case we might find that the ambari "hosts" table in the ambari database has different values for the "public_host_name" and the "host_name"select public_host_name, host_name

This is a quick how-to on integrating spark on zeppelin with kerberised kafka. To do this first we will enable impersonation support in zeppelin for spark. By Impersonation, i mean jobs will be executed / submitted to spark by zeppelin as the user logged in to zeppelin and not the default zeppelin user. In my case i will be using my user "Karthik"Now to enable impersonation while submitting to spark, i would recommend using the livy server. The livy server and the interpreter natively support impersonation. Spark interpreter impersonation is possible, but you may have to configure SSh keys or sudo capability for zeppelin which may not be recommended. You can livy to hdp by going to a host and installing "Spark client" , followed by the livy server.Restart Zeppelin , to enable the livy interpreter. Ensure that livy impersonation and livy proxy permissions are setup correctly in ambari. This is done when you install livy server by default.Log in to zeppelin using your credentials.click t

This article assumes that you're already configured to use Ranger authorization in HDF 3.0.x or earlier and that Ranger instance is syncing the available users/groups with your Directory Server. In order to leverage policies defined with these groups, we'll need to upgrade to HDF 3.1.0 or later and migrate your configuration to utilize a new Ranger authorizer in NiFi.In short, you’ll need to update your authorizers.xml file to configure a userGroupProvider and modify your authorizer to be a ManagedRangerAuthorizer. The idea is that previously an Authorizer was configured which was responsible for making access decisions, managing policies, and managing users/groups. Whether or not the Authorizer supported managing policies and users/groups was optional. The default File base authorizer supported this. The existing Ranger authorizer did not (it only made access decisions).For HDF 3.1 we decoupled these concepts so you can independently configure the Authorizer (makes access decisions),

Here are the best practices for writing an HBase client application for HDP. 1. Use the new HBase-1.0 API's instead of old interfaces. Instead of HTable, use Table, instead of HConnection, use Connection, etc. Also the Connection management has been changed so that the connection lifecycle management is best performed by the client application. Check out these slide decks for more examples:https://www.dropbox.com/s/v1x3djtlp1qg204/HBase%201.0%20API%20Changes%20-%20Meetup%2010_15_2014.pdf?dl=0http://www.slideshare.net/enissoz/meet-hbase-10As well as further examples here:https://github.com/hortonworks/hbase-release/tree/HDP-2.5.3.0-tag/hbase-examples2. Always close the Connection, Table, Scanner and Admin interfaces when you are done. These interfaces implement Closeable, and holds resources on the client side or on the server side (Scanner). Thus properly closing these is very important for best performance. Java's new try syntax is an easy way to auto-close these resources in your cod

Introduction This document describes the details of Knox SSO integration with a SAML identity provider (For the purpose of this article, the IDP is Ping Federate). Here, Ping Federate acts as the identity provider and it has the ability to connect to any authentication source such as AD, LDAP, etc.   The following diagram depicts the process at a high level:    The SAML authentication process has three components  Identity Provider: Identity provider has the flexibility to authenticate users against an authentication source, and sends a SAML response XML back to the service provider Service Provider: Service provider is an application that requires users to authenticate Authentication source:  A directory service such as AD, LDAP. The service provider uses the authentication token and does not need to know the authentication source. As part of this integration, both IDP (Ping Fed) and SP (Knox server) exchange information for the handshake to work successfu

Configure Ldap server on Redhat/Centos :- Check the ldap packages are installed or not on Server with following command #rpm –qa|grep openldap2. If packages are not installed then install the packages with yum command#yum install openldap-* -y 3. Once pacakge are installed then check with following command#rpm –qa |grep openldap4. Create Ldap password with following command#slappasswd [Enter the password and copy the md5 formate password for adding the password into thedatabase file]5. Edit database files for domain#vi /etc/openldap/slapd.d/cn=config/olcDatabase={2}bdb.ldifoldSuffix:dc=example,dc=comolcRootDN:cn=Manager,dc=example,dc=comolcRootPW:copy the password here which isgenerated after set the slpappasswd. #vi /etc/openldap/slpapd.d/cn=config/olcDatabase={1}monitor.ldifolcAccess: {0}to * by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" read by dn.base="cn=manager,dc=example,dc=com" read by * none6.Run the updatedb command to initialize database. Create

Yesterday evening, in a moment of inspiration, I wondered if one could build a game in NiFi (a tool for moving data in realtime). Of course, building a game is quite unnecessary, but it is far from pointless as it can really show how flexible a tool can be with a bit of creativity. A few years ago I played ‘Worlds first game of chess… against PySpark’ but given that NiFi is meant for simpler logic, I decided to start with something different. Building Tic Tac Toe in NiFi My goals were to build something that allowed people across the globe to play in realtime, by using a board (no code). Fortunately, the Graphical User Interface of NiFi was made with realtime interaction of multiple users in mind, so I had a good foundation for this. Furthermore, I wanted NiFi to determine the winner automatically. For this, I leveraged its capabilities in applying logic/business rules. Of course, NiFi wants to apply the logic to something, so to ensure that the winner would be determined in a timely

Introduction This article assumes that KnoxSSO for NiFi UI is enabled and is working as expected.   To configure Knox topology in order to access NiFi Rest API using Knox based NiFi URL, do the following:     Access NiFi UI through Knox SSO based URL such as the following:https://<knox.fqdn>:8443/gateway/<nifi-topology>/nifi-app/nifi/​This helps the users to expose only the Knox host and port (not the NiFi hosts) and also authenticate users via SSO before they successfully log in to NiFi UI. Once the SSO is enabled on NiFi UI, NiFi URL for the UI or the Rest API will automatically redirect to the SSO page. Usually, the NiFi Rest API access involves obtaining Bearer token and using it in subsequent API calls. With Knox proxy-based NiFi endpoint, this Bearer token would not work as Knox does not recognize this token. On top of that, the URL always redirects to the SSO URL.    To establish access to Knox based NiFi URLs, both Knox and NiFi need