Community Articles

The purpose of this article is to explain what Process Groups and Remote Process Groups (RPGs) are and how input and output ports are used to move FlowFiles between them. Process groups are a valuable addition to any complex dataflow. They give DataFlow Managers (DFMs) the ability to group a set of processors on to their own imbedded canvas. Remote Process groups allow a DFM to treat another NiFi instance or cluster as just another process group in the larger dataflow picture. Simply being able to build flows on different canvases is nice, but what if I need to move NiFi FlowFiles between these canvases? This is where input and output ports come in to play. They allow you move FlowFiles between these canvases that are either local to a single NiFi or between the canvases of complete different NiFi instances/clusters. Embedded Process Groups: Lets start by talking about the simplest use of multiple embedded canvases through process groups. When you started NiFi for the very first time

Recently I came around an interesting problem: how to use boto to get data from a secure bucket in a Jupyter notebook in Cloudera Machine Learning. The missing piece was: I needed to get my code integrated with my AWS permissions given by IDBroker.Since CML already authenticated me to Kerberos, all I need was getting the goods from IDBroker. In this article, I will show you pseudo code on how to get these access keys both in bash and python.Note: Special thanks to @Former Member to whom I owe this article and many more things.  Find your IDBroker URL Regardless of the method, you will need to get the URL for your IDBroker host. This is done simply in the management console of your datalake. The following is an example: Getting Access Keys in bash After you are connected to one of your cluster's node and ensure you kinit, run the following:IDBROKER_DT="$(curl -s --negotiate -u: "https:/[IDBROKER_HOST]:8444/gateway/dt/knoxtoken/api/v1/token")"

This article describes the method to remove Ranger Plugin status records for unused services. For example, the records are underlined in the following screenshot. There is no menu option to remove such records in the web user interface. The only way to remove them is to manually delete the records from x_plugin_info table directly from the Ranger database.   Once connected to the Ranger database, run the following:select * from x_plugin_info; Identify the record ID for the record you want to remove.delete from x_plugin_info where id = Record_ID;

Many times, Support need to have a look at the Microsoft AD tree for AD internal configuration (e.g. CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration, <domain>). The problem with this part of the AD tree is - it is not visible by default in the AD directory structure. This article will help the reader to explore the hidden AD tree. Any change in AD should be carried out by the customer’s AD team or admin. Instructions Here are the steps: Log on to the AD server Run the ADSIEdit.exe program from Start > Run menu. Select the top-most entry of the left side, and then, select Action > Connect to... from the menu. In the next dialog box, select Select a well known Naming Context radio button, and select Configuration from the drop-down menu. Click OK and ADSI Edit will now show CN=Configuration and its sub-tree. We can now traverse to CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration, <domain> part easily as seen above. Example: For Mic

HiveServer2 is a thrift server which is a thin Service layer to interact with the HDP cluster in a seamless fashion. It supports both JDBC and ODBC driver to provide a SQL layer to query the data. An incoming SQL query is converted to either TEZ or MR job, the results are fetched and send back to client. No heavy lifting work is done inside the HS2. It just acts as a place to have the TEZ/MR driver, scan metadata infor and apply ranger policy for authorization. Configurations :TEZ parameters Tez Application Master Waiting Period (in seconds) -- Specifies the amount of time in seconds that the Tez Application Master (AM) waits for a DAG to be submitted before shutting down. For example, to set the waiting period to 15 minutes (15 minutes x 60 seconds per minute = 900 seconds):tez.session.am.dag.submit.timeout.secs=900 Enable Tez Container Reuse -- This configuration parameter determines whether Tez will reuse the same container to run multiple queries. Enabling this parameter improves

Since Ranger 0.5 there has been the ability to summarize audit events that differ only by timestamp to reduce the amount of events logged in a busy system. When enabled, if a Ranger plugin logs consecutive audit events that differ only by timestamp it will coalesce all such events in to a single event and set 'event_count' to the number of events logged and 'event_dur_ms' to the time difference in milliseconds between the first and last event.To enable this feature you must set the following properties in the Ranger plugin's configuration:Configuration nameNotesxasecure.audit.provider.summary.enabledTo enable summarization set this property to true. This would cause audit messages to be summarized before they are sent to various sinks.By default it is set to false i.e. audit summarization is disabled.xasecure.audit.provider.queue.sizeIf unspecified this value defaults to 1048576, i.e. the queue is sized to store 1M (1024 * 1024) messages.Note the difference in property name that contr

Objective With the release of NiFi Registry 0.2.0, flow contents can now be stored under a Git directory using the new GitFlowPersistenceProvider. This tutorial walks you through how to configure this provider in NiFi Registry so that versioned flows in NiFi are automatically saved to a Git repository. A video version of this tutorial can be seen here: https://youtu.be/kK7eVppg9Aw Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi Registry 0.2.0 Apache NiFi 1.7.1 GitFlowPersistenceProvider Git Configuration First, create a new GitHub repo: then clone it locally using the git clone command (e.g. git clone https://github.com/andrewmlim/versioned_flows.git): Next, go to GitHub’s “Developer settings” and create a new “Personal access token”: NiFi Registry Configuration In the ./conf/providers.xml file, configure the following properties: Set org.apache.nifi.registry.provider.

When you install Cloudera Manager you can configure the mail server you will use with the Alert Publisher. However, if you need to change these settings, you can do so under the Alert Publisher section of the Management Services configuration tab. Through Alert Publisher we can get the alerts for a different type of service status like Bad, Warning, Good. Under the Alert Publisher role of the Cloudera Manager Management Service, you can configure email or SNMP delivery of alert notifications and you can also configure a custom script that runs in response to an alert. The information on how to configure the alert can be found in the official documentation of Cloudera. Please check the provided link [1].However, we will discuss some of the common issues faced while configuring or receiving the alert delivery. Common Issues Over Alert Publisher-> 1- Sometimes we are not able to receive the alert to the SMTP/SNMP server end. While checking the alert-publisher logs, the b

What is CRN?CRN is an acronym for Customer Resource Number and this is a Cloudera-specific identifier for an environment. The CRN uniquely identifies the environment as well as the CDP control plane that is used to create it.Where can I find the CRN?It is located prominently on the Details page of each CDP environment. Steps to find the CRN for an environment:Click Environments from the CDP Management Console in the left panel next to Dashboard Type in the name of the environment in the search bar. In the screenshot here, the search key is "orrep-aw-env"Click on the "Name" of the environment. This should show the details of the environmentClick Data LakeNow, click Copy to Clipboard icon next to "crn: ..." in the header This should copy the CRN to the clipboard. Now you can paste it as and where needed.

In this article, we will learn how to integrate Zeppelin JDBC (Phoenix) interpreter example.   1. Configuring the JDBC (Phoenix) interpreter:Login to Zeppelin UI -> Click on the user name (in my case, admin) at the right-hand corner. It will display a menu > click on Interpreter.   Click on + Create at the right-hand side of the screen.     It will display a popup menu. Enter Interpreter Name as jdbc and select Interpreter Group as jdbc. Then, it will populate Properties in table format.      Click on + button and add the Phoenix-related properties according to your cluster, and click on the Save button. phoenix.driver org.apache.phoenix.jdbc.PhoenixDriver phoenix.url jdbc:phoenix:localhost:2181:/hbase phoenix.user   phoenix.password        2. Creating the Notebook:   Click Notebook dropdown menu in the top left-hand corner and select Create new note and enter Note Name as Phoeni

This article describes various "recipes" on how to accomplish certain tasks with the NiFi processor ExecuteScript, with examples given in Groovy, Jython, Javascript (Nashorn), and JRuby. This is Part 3 in the series, I will be discussing advanced features such as dynamic properties, modules, state management, and accessing/using Controller Services. Part 1 - Introduction to the NiFi API and FlowFiles Getting a flow file from an incoming queue Creating new flow files Working with flow file attributes Transferring a flow file Logging Part 2 - FlowFile I/O and Error Handling Reading from a flow file Writing to a flow file Reading and writing to/from a flow file Error Handling Part 3 - Advanced Features Using Dynamic Properties Adding Modules State Management Accessing Controller Services   Advanced Features The first two articles in the series covered the basics of flow file operations, such as reading/writing attributes and content, as well as retrieving and transferring flow

Components required: jconsole (UI required) jmxterm (for Linux environment - CLI only) Kafka client (java producer/consumer) exposing JMX  Kafka Brokers exposing jmx Steps to get the available metrics (mbeans) available in a Kafka consumer (Environment with a UI available): Add the following JVM properties to your java consumer:-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9090 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false After starting, your consumer uses jconsole to connect to the host:port specified in Step 1. After we add the <producer-hostname>:<JMXPort>, we should be able to see the following in the jconsole UI. Here, we have to click on the mbeans tab and navigate through the available metrics, in this example, we want to see the "records-consumer-rate" which in the program has the following definition "The average number of records consumed per second".In this case, the average

Problem: Trying to use sudo to run commands(as root) in a cluster gives error " Stderr: Pseudo-terminal will not be allocated because stdin is not a terminal. sudo: sorry, you must have a tty to run sudo" This is a common situation where you are login as some other user in a terminal and then sudo to root to perform the task. And you can only ssh from the user you logged in and not from root. Solution: Note: You have to install pssh if it is not already installed. in your cluster.Keep all host names in a file say all_hosts This kind of execution will fail:pssh -i -h all_hosts "sudo whoami;hostname -f" Will fail to execute and give error: Stderr: sudo: sorry, you must have a tty to run sudo OR pssh -i -h all_hosts -x "-t" "sudo whoami;hostname -f" Will fail to execute and give error:Stderr: Pseudo-terminal will not be allocated because stdin is not a terminal. sudo: sorry, you must have a tty to run sudo The correct command should be run like this:pssh -i -h all_hosts -x "-t -t" "sudo

This article will show you how to interact with Atlas APIs in CDP-public to create tags and associate tags with entities (in preparation for use with Ranger's tag based policies)   In Cloudera CDP-public offering, Apache Atlas is a part of SDX DataLake cluster that is created when you create your first Environment: Introduction to Data Lakes Pre-requisites A. First, you will need to find the Atlas endpoint using the Cloudera CDP management console: Accessing Data Lake services Sample Atlas endpoint: https://pse-722-cdp-xxxxx.cloudera.site/pse-722-cdp-dl/cdp-proxy-api/atlas/api/atlas/   B. Next, you will need to set your user's workload password Setting the workload password Now you can use the following sample bash code to interact with Atlas APIs from a CentOS instance outside CDP:   From Atlas endpoint, you can extract the first 2 params below. You will also need to set your username and password: export datalake_name='pse-722-cdp-dl'export lake_ip='pse-722-cdp-xxxxx.

This article applies to HDP 2.5.x and below. For HDP 2.6, please check new article. Zeppelin can be configured to leverage an organization's Active Directory infrastructure for user authentication. By doing this, the existing Active Directory users can login to Zeppelin UI using their Active Directory credentials. This article discuss how to configure this kind of setup. Environment Setup: HDP 2.5 cluster / Sandbox - I'm using HDP 2.5 Sandbox on VirtualBox. Get one from here ! Ambari 2.4+ - I'm using Ambari 2.4.0.0 which comes with HDP 2.5 Sandbox 'Zeppelin Notebook' Service installed in Ambari - With HDP 2.5 Sandbox, it will be Zeppelin version 0.6.0 - If you don't have Zeppelin installed, it can be installed via 'Add Service' option in Ambari Active Directory - I'm using Active Directory 2012 R2 version - Make sure that you have 'working' Active Directory details handy like URI, bind DN/password, search base etc. Configuration Steps: 1. From Ambari Dashboard, navigate to Zeppelin Not

My previous article demonstrated how to use the CDE-managed Airflow instance to read files from CML. To close the loop, here is a short demonstration on how to write files to CML using CDE's managed Airflow instance.   Artifacts Airflow dag Demo  

I wanted to quickly share that we’ve released FIVE new Applied ML Prototypes (AMPs) in Cloudera AI and Cloudera AI Workbench! These AMPs solve a wide range of problems for data scientists and help jumpstart ML / AI projects, enabling them to deliver greater value faster across their organization. Here’s an overview of what was released: Summarization — This project demonstrates four automatic summarization models, including extractive and abstractive techniques. Why you should care: Summarization enables users to quickly extract important information from larger bodies of text. This is useful for any industry looking to accelerate research, competitive analysis, or businesses that need to process and understand large amounts of text information that would be time-prohibitive for a human to do. AutoML with TPot — This project enables automated machine learning on a sample of credit card fraud data. Why you should care: AutoML has the potential to accelerate many repetitive pro

Description NiFI performance comparison between Python and groovy script. This analysis shows the performance impact of programming scripts in NiFi. And also, it has an impact on setting concurrent tasks. Here, generate processor has input, and it is coming out as a attribute. Groovy and Python script read the attributes and replace them with the actual value for target_file_cfg. Below are the sample codes. Objective Translate Python script to groovy script and analyze the performance and set the concurrent task accordingly. Use Case Following is the input where the target file name will replace with the actual value.   input-target_file_cfg — /movement/%C/%D/MOVE%V/%F output-target-filename — /movement/JP/None/MOVE055/xxxxxx.xxxx.x Input target_file_cfg — /movement/%C/%D/MOVE%V/%F country-JP dns-country-code- JP dns-site-number- 055 source-file-country-code-WM source-filename-xxxxxx.xxxx.x target_file_cfg — /movement/%C/%D/MOVE%V/%F Output target-filename — /movement/JP/None/MOV

Introduction of Apache Kafka 3.0 Thanks to a collaborative effort between Cloudera and other parties, Apache Kafka continues to improve rapidly, and now, this has resulted in the release of Apache Kafka 3.0! The top two questions now are: What does this mean for the setup that I have today? Why are we excited about Apache Kafka 3.0? Any challenges we should be aware of? Let us address both of these. What does this mean for the setup that I have today? First of all, there is no need to be concerned about your current footprint. Every time Cloudera announces a release, it also announces an end of support date. The availability of a new version does not detract from this.  Furthermore, the Kafka 2 branch is very much still alive, and we are still planning to release improvements to this for a good period of time. Why are we excited about Apache Kafka 3.0? Any challenges we should be aware of? In short, we see Kafka 3.0 as a good foundation. There are many areas with small improvem

Migrating HDP clusters to CDP has been a journey many customers are going through these days. Migrating Ranger Audits + Atlas collections in infra-solr to CDP has been a challenging task. We hope the steps below will make simplify your journey. Preparation: Sample API calls to get the current status of collections in infra-solr. This is an important step to visualise how big these collections are and as a result, get an idea of how long the migration will take. Note: In the commands below, -k is used if https is used rather than http change http for https if the connection is secure check the port is correct depending on the version of infra-solr being used or if you have customized the port Infra-Solr API queries Gather the list of collections in infra-solr   ### To list collectionscurl --negotiate -u: -k 'http://solr_host:solr_port/solr/admin/collections?action=LIST'     Get the total number of records in your infra-solr collection. Example, ranger_audits

At Cloudera, we believe that data can make what is impossible today, possible tomorrow. There are many good uses of data. With data, we can monitor our business, the overall business, or specific business units. We can segment based on the customer verticals or whether they run in the public or private cloud. We can understand customers better, see usage patterns and main consumption drivers. We can find customer pain points, see where they get stuck, and understand how different bugs affect them. With data, we can discover new market opportunities, and review where we stand compared to the global market. We can track feature adoption, see how new features are picked up and what usage/consumption they generate. With data, we can set better goals, know where we are and where we want to go. And in the end, we can make better decisions.  At Cloudera, we practice what we preach. As the Cloudera gains popularity and more and more customers make it a critical piece of their in

1. We need to know on which database Hive metastore is hosted like my SQL, Oracle? 2. What is the version of jdbc driver used for a database connection?   if Oracle then, we need to know which version it has by this command unzip -p /usr/share/oracle-connector-java.jar | head > jar_version.txt Right now we are using ojdbc8.jar.   Solution: Using the following command, move the existing oracle-connector-java.jar from /usr/share/java/ to /tmp/mv /usr/share/java/oracle-connector-java.jar /tmp/ Using the following command, copy the existing ojdbc8.jar as oracle-connector-java.jarcp /usr/share/java/ojdbc8.jar oracle-connector-java.jar Using the following command, change the permissions to 775chmod 775 oracle-connector-java.jar Restart the Hive service Disclaimer: This article is contributed by an external user. The steps may not be verified by Cloudera and may not be applicable for all use cases and may be very specific to a particular distribution. Please follow with

If any user/admins are not able to view the container logs. And the applications are running with Spark service. Looks like ACLs are not configured. Add this in your clusters:    Go to Cloudera Manager -> Spark -> Configuration -> Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.confspark.acls.enable= truespark.admin.acls=username1,username2spark.admin.acls.groups=group1,group2 Set spark.ui.view.acls=username1,username2spark.ui.view.acls.groups=group1,group2 Set spark.modify.acls=user1,user2spark.modify.acls.groups=group1,group2  If you pass * as value it will give access to all the users. And both user[admin.acls] and groups[admin.acls.groups] accept both username and groups as comma-separated values.   Both user and group lists use a comma-separated list of entries. The wildcard character “*” allows access to anyone, and the single space “ “ allows access to no one.   Regards, Naresh.B   Disclaimer

 Short demo on how to use CDE's new Airflow UI to design pipelines.  Additionally leveraging Airflow triggers to orchestrate a typical job pipeline forking logic (triggers).  A "Typical job pipeline" often encapsulates logic regarding success and failure paths. Artifacts Demo Videohttps://youtu.be/G-53Uf4qrFwGit Repohttps://github.com/sunileman/cde-airflow-branch-operator  

Often a timestamp is found within a phoenix table to support various business taxonomy requirements. Generally I have seen a pattern simply adding a timestamp datatype field. Many may not realize part of the hbase data model rowkey design is the timestamp. When a cell is written in hbase by default it is versioned using timestamp when the cell was created. This is out of the box.Apache Phoenix provides a way of mapping HBase’s native row timestamp to a Phoenix column. This leverages various optimizations which HBase provides for time ranges on the store files as well as various query optimization capabilities built within Phoenix. - https://phoenix.apache.org/rowtimestamp.htmlBased on your use case, set this "column" to a value of your liking. Take advantage of the built in data model design without adding yet another timestamp field. Commonly I find many end up creating secondary index on these additional timestamp fields due to time always being a variable at query time.Lets ta