Q&A

My Production environment has existing 4 data nodes.I just added additional 4 data nodes in the cluster and would like to run the HDFS Rebalance to balance the data across the 8 data nodes. My question is, can I rebalance data but also ingesting and processing new data at the same time? Will it caused any file interrupted if do so?  Thanks.  

I had a flow which needs to be replicated for 100 jobs.one of my jolt transform looks like this. [{ "operation": "default", "spec": { "*": { "id": "#{customer_id_1}", "warehouse_loaddate" : "${now():format('yyyy-MM-dd HH:mm:ss', 'CET')}" } }}] The customer_id_1 is a parameter currently, which is different in all 10 different jobs.  like there are many different parameters in the context like customer_id_2customer_id_3 etc..With this I need to touch my code at 100 different places...But since I know the customer ID is present in my upstream processor.. so I just want to name the parameter customer_id and get the value from upstream dynamically... Is there a way that I can send a attribute to downstream processor and use it in jolt ?Without using parameters from parameter context ? We use #{<id>} and ${<func>} to access parameters and Functions.. but how can i refer an attribute from my upstream processor ? Please help me under

I have installed Zeppelin on my CDP cluster , since by default "anonymous" id is disabled what is the user ID that I can use to login to Zeppelin UI. I tried using :admin = password1user1 = password2user2 = password3But none of them seems to work

Hello,We need to partition 5 hive tables with more than 100 million records each. Tables have multiple columns to partition with integer and string data allowing less than 1000 partitions per column. Could we combine several of these fields to partition the tables? would there be a restriction on the maximum number of combined partitions? Should they have a specific order according to the number of partitions in each table? for example, first partition by year (10 partitions) , then by country (50 partitions) and then by city name (500 partitions) ? Must the combination of partitions be less than a particular number?Best Regards,H.M 

Hi,After tried to login, it is taking me back to reenter the credentials.I have tried all the passwords.Pls help.... 

Hi, Is there any processor to add some text/characters to each line in the flowfile in Nifi. Replace text seems to be more complicated given that the partiton value keeps changing. compute incremental stats mydb.mytable partition (date=2023-07-21);compute incremental stats mydb.mytable partition (name=abc);tocompute incremental stats mydb.mytable partition (date='2023-07-21');compute incremental stats mydb.mytable partition (name='abc'); Note: date=2023-07-21 comes as a complete string from the sys.partitions.part_name table in hive. We will need to add '' to the date.

 Hello All,I am new to NiFi. I have set up NiFi with LDAP authentication and NiFi starts with no issues but I don't know how to log into NiFi as "Initial Admin Identity" user "user_admin" as shown in my setup below. What should I use as the login credentials as NiFi presently rejects user_admin/<password>?nifi.propertiesnifi.security.user.authorizer=file-providernifi.security.allow.anonymous.authentication=falsenifi.security.user.login.identity.provider=ldap-provider login-identity-providers.xml       <identifier>ldap-provider</identifier>        <class>org.apache.nifi.ldap.LdapProvider</class>        <property name="Authentication Strategy">START_TLS</property>         <property name="Manager DN">uid=test_admin,OU=users,DC=abc,DC=com</property>        <property name="Manager Password">*******</proper

I am using ListFile processor to pick up input files to my flow. And it picks up all available files in the directory.Is it possible to configure it to pick only one file, let its processing to be completed and then pick next file. So basically only one file at a time to be processed by system.Or if any processor other than ListFile can be used for this purpose?

Nifi registry key was expired, I have regenerated key for nifi registry ans also nifi nodes.After setting up new key ssl error resolved but now i still can't commit to nifi registry from nifi node.Error log from nifi registry log :2023-09-15 12:18:56,526 ERROR [NiFi Registry Web Server-25] o.a.n.r.web.mapper.ThrowableMapper An unexpected error has occurred: org.apache.nifi.registry.flow.FlowPersistenceException: Git directory ./nifi01_flow is not clean or has uncommitted changes, resolve those changes first to save flow contents.. Returning Internal Server Error response.org.apache.nifi.registry.flow.FlowPersistenceException: Git directory ./nifi01_flow is not clean or has uncommitted changes, resolve those changes first to save flow contents.Note: I did not made any config change other then adding new ketstore and truststore to nifi registry and nodes.

Here is my  login-identity-providers.xml <provider><identifier>ldap-provider</identifier><class>org.apache.nifi.ldap.LdapProvider</class><property name="Authentication Strategy">SIMPLE</property><property name="Manager DN">cn=admin,dc=example,dc=com</property><property name="Manager Password">secret</property><property name="TLS - Keystore"></property><property name="TLS - Keystore Password"></property><property name="TLS - Keystore Type"></property><property name="TLS - Truststore"></property><property name="TLS - Truststore Password"></property><property name="TLS - Truststore Type"></property><property name="TLS - Client Auth"></property><property name="TLS - Protocol"></property><property name="TLS - Shutdown Gracefully"></property><property name="Referral Strategy">FOLLOW</property><pr

Is it possible, or does it make sense to use AWS Load Balancers in front of a NiFi Cluster?We are running a 3 node nifi cluster using v 1.19.1 on aws and I wanted to see if it is possible, or makes sense to use an AWS Load Balancer(s) in front of the nifi cluster.Any guidance, suggestions would be tremendously appreciated.

Hi All ,I planning to upgrade NiFi 1.9.2 to 1.23.0. I read some articles about upgrading.https://community.cloudera.com/t5/Support-Questions/Nifi-Upgrade/m-p/346309If I follow the take special note above , can I directly migrate 1.9.2 to 1.23.0 ? Will there be any problem ?

Hi,I have an input json as below: [ { "orderId": "1234567890", "orderName": "Test1", "orderItems": [ { "orderItemId": "01", "orderItemName": "test1-OrderItem1" }, { "orderItemId": "02", "orderItemName": "test1-OrderItem2" } ] }, { "orderId": "12235", "orderName": "Test2", "orderItems": [ { "orderItemId": "01", "orderItemName": "test2-OrderItem1" } ] }, { "orderId": "12236", "orderName": "Test3", "orderItems": [ { "orderItemId": "01", "orderItemName": "test3-OrderItem1" } ] }] I want to convert it into text form where we want the order details and then item details for each map in the list.Here is the desired output: 1234567890 Test101 test1-OrderItem102 test1-OrderItem212235 Test201 test2-OrderItem112236 Test301 test3-OrderItem1 This is json to csv conversion, and I am using the ConvertRecord processor and below are the reader and

I'm trying to use a "GenerateTableFetch" process, but it always returns empty flow files, with no error.When I turn on trace, it runs the followingGenerateTableFetch[id=9adb6bf3-018a-1000-517b-b3f88a43e39e] Setting CLUSTER State to {}GenerateTableFetch[id=9adb6bf3-018a-1000-517b-b3f88a43e39e] Returning CLUSTER State: StandardStateMap[version=663, values={}]GenerateTableFetch[id=9adb6bf3-018a-1000-517b-b3f88a43e39e] Executing SELECT COUNT(*) FROM TESTTRANSFERGenerateTableFetch[id=9adb6bf3-018a-1000-517b-b3f88a43e39e] Returning CLUSTER State: StandardStateMap[version=663, values={}]GenerateTableFetch[id=9adb6bf3-018a-1000-517b-b3f88a43e39e] Returning CLUSTER State: StandardStateMap[version=663, values={}]So it just runs a COUNT(*) and then exits it seems?When I run the same query against the ORACLE 19c database I get a count of 500kI can't find any info that suggests Oracle isn't supported, but I can't figure out a reason it would just return nothing. 

On a processor, right click -> view status history....there is a data available by name 'Records Processed' in dropdown.For one of the processor in my flow, this is not available.What can be the reason for this? Is it possible that value was so high that system lost track somewhere?

I have a nifi flow, working fine.  It picks file from a source directory, processes it, and file is deleted from source directory once picked up.This all works fine first time. But later on, if I copy another file in source directory, that file is not picked up. I even stop all processors, clean any queue if not empty, and restart processors, but still file is not picked from input directory.What causes this behaviour?As a result, I have to stop-restart nifi, making it a bit cumbersome.PS: Based on some google suggestions, I tried to delete flowfile_repository, but it doesn't allow so with message 'Device or resource busy'.

Hi,I'm seeking guidance on how to set up version control for Apache NiFi on a Windows machine using Bitbucket as the repository. Could you please  provide me the steps how to configure?

Does HDP and Ambari install on Rocky 8?

Hello everybody,One of my disks on a datanode brake down few months ago. I changed it to a new one but my HDFS won't balance datas between disks. So, i have 7 disks with 80-90% of their capacity and 1 disk with only 15-20% of his capacity.I tried couple things, such as balancing HDFS or delete every small files.Furthermore, this difference between disks created a huge amount of blocks in my cluster. (The datanode has 1,043,627 blocks. Warning threshold : 0)Do you have any ideas to fix this ? Thanks,Oscar

while connecting hive to hue Got below error TSocket read 0 bytes (code THRIFTTRANSPORT): TTransportException('TSocket read 0 bytes',)

Good morning everyone! I am having troubles with the PutKudu processor.The Put Processor should write on a KUDU table, previously created by using Impala on HUE, inside the HDFS. Everything is configured but the PutKudu processor keeps showing up this error message. Kudu is configured with just one master (leader). I am thinking that this error could be caused by my one-node configuration as the docu seems to suggest to use a 3 nodes configuration at least. Is this the case ?The error message is too generic to be useful for any kind of debugging activity.The pipeline is composed by two processors: generateflowfile and putkudu.Down below you can have a look at the error messages and at the putkudu properties. If you need more information please let me know.Thank you and have a nice day.  

Hi , Kindly for your support, I'm trying to create a kudu table from impala shell after enabling kerberos but i got the following error:  ERROR: AnalysisException: Cannot analyze Kudu table 'courses_kudu': Error determining if Kudu's integration with the Hive Metastore is enabled: Couldn't find a valid master in (cloudera.ngdw.com:7051). Exceptions received: [org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT)] Kindly find attached steps and error returned.  

Hi everyone,I use NiFi to frequently check if a website is up or down. I get the following response as text in a single Flowfile:HTTP/1.1: 200 Connection establishedProxy-Agent: Fortinet-Proxy/1.0Content-Type: image/jpegConnection: keep-aliveDate: Tue, 12 Sep 2023 14:05:43 GMTServer: ApacheStrict-Transport-Security: max-age=31536000URL: https://test.com How can I turn these lines into Key/Value-Pairs or Attributes, to use them with further processors? My goal is to save these Informations in an mySQL Database where the following fields are given:HTTP/1.1Proxy-AgentContent-TypeConnectionDate...Best regardsMaik

Hello everyone, I am new to docker and I have been researching and working with nifi recently. I have one query for my use case where I need to use code repositories that are outside the docker container (I have already bind them while docker run command) and I can access the folder via exec but I need to access these scripts in the executestreamcommand processor's working directory and command path fields. Even after putting the destination directories into the fields I am getting no such file or directory exists. So, please help me out.

I have some basic questions about nifi cluster.I understand that nifi makes use of multiple nodes to speed up the processing of data.But how do we ensure durability of data processed? Suppose we distribute n flowfiles in a queue to several nodes, with each node getting a subset of n files. Does nifi assign same subset to more than one nodes, so that if one of those processing node goes down, we still have others working on same subset?Second, how is result of processing  done on different nodes aggregated? Is it done on primary node OR there is some other mechanism?