Q&A

 Hi All, Getting error in executing below script thru Execute Script processor.Requirement : upper flow will copy files to some folder.. trying to read files from folder and post them to http processor. For each file i am trying to create a new flowfile and move the same to HTTP processor. But below code failing. Can you please suggest what i am missing.--------------------------------import java.nio.charset.StandardCharsetsdef flowFile = session.get()if(!flowFile) returndef flowFiles = [] as List<FlowFile>def inputStream = session.read(flowFile)def list = []def dir = new File("/root/file_data/poc/output/")dir.eachFileRecurse (FileType.FILES) { file ->       list << file}list.each{ i ->        def newFlowFile = session.create(flowFile)        newFlowFile = session.write(newFlowFile, { outputStream ->             outputStream.write( i.getBytes(StandardCharsets.UT

Hi,I want to  modify the ddl query table name. I am using the replaceText processor to modify the flowfile query table name. Currently I am able to achieve it   because I have one source DB and only one table name I have changed in destination DB.Like sourceDBName: nrpsubscriotion and tableName: status -> destinationDBName: nrpresport and tableName: subscription_status. Now I have two source DB(1. nrpsubscriotion, 2. nrpuserorg) and both have same tableName: status. Now Source DB 2 , tableName in destination DB nrpreport has tableName: user_org_status. I have stuck here , I am not getting how can define a condition to replce a tableName in DDL based on schema name and ddl query able name: It should map like below: 1. SourceDBName: nrpsubscriotion and tableName: status -> destinationDBName: nrpresport and tableName: subscription_status.Flow File input: {"type" : "ddl","timestamp" : 1658314685000,"binlog_gtidset" : "e75d07af-eb37-11ec-9d

Hi,I want to manipulate below string in updateAttribute  processor. I tried replace function etc but it is not working for me. Can anyone suggest to modify below string to change DBName and TableName as per expected output.query = ${query:replace('nrpuserorgdb2306.status', 'nrpreportdb.user_org_status')} query : "ALTER TABLE `nrpuserorgdb2306`.`status` \nADD COLUMN `is_deleted` TINYINT NULL AFTER `code`"Expected output:query : ALTER TABLE `nrpreportdb`.`user_org_status` ADD COLUMN `is_test` TINYINT NULL AFTER `code`

Hi everyone, I'm trying to add Hive service in cluster but an error occurred.This is the log file:2022-07-19 12:42:25,037 main ERROR Log4j2 ConfigurationScheduler attempted to increment scheduled items after startStarting metastore schema initialization to 3.1.3000.7.1.7.1000-141Initialization script hive-schema-3.1.3000.hive.sql2022-07-19 12:42:25,310 main ERROR Log4j2 ConfigurationScheduler attempted to increment scheduled items after start How can i proceed to solve the problem? Thank you

I have configured the NiFi flow file where I am reading mysql binary-log file. for me insert, update and delete flow is working fine but when I alter a table then I am getting error.sourceDBName: nrpsubscriptiondb2306 and tableName: product_types destinationDBName: nrpreportdb and tableName: product_types Alter table nifi error:Record had no (or null) value for Field Containing SQL: query, FlowFile StandardFlowFileRecord[uuid=8ca029c9-70aa-4e53-aec3-d37c15c8cd07,claim=StandardContentClaim [resourceClaim=StandardResourceClaim

Hi everyone, I would like to fetch all the processor groups in the nifi canvas including sub-processor groups usinf nifi-api. Could you please help me on this

Hi, I'm trying to make a kafka consumer working, but I am having this issue about SSL Handshake failed. Any ideas ?  2022-07-18 14:00:45,216 INFO [NiFi Web Server-203] o.a.n.c.s.StandardProcessScheduler Starting ConsumeKafkaRecord_2_6[id=f5ee162d-1006-1181-c1d1-1d8a7293ffb7]2022-07-18 14:00:45,217 INFO [NiFi Web Server-203] o.a.n.controller.StandardProcessorNode Starting ConsumeKafkaRecord_2_6[id=f5ee162d-1006-1181-c1d1-1d8a7293ffb7]2022-07-18 14:00:45,217 INFO [Timer-Driven Process Thread-5] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled ConsumeKafkaRecord_2_6[id=f5ee162d-1006-1181-c1d1-1d8a7293ffb7] to run with 1 threads2022-07-18 14:00:45,219 INFO [Timer-Driven Process Thread-8] o.a.k.clients.consumer.ConsumerConfig ConsumerConfig values:allow.auto.create.topics = trueauto.commit.interval.ms = 5000auto.offset.reset = latestbootstrap.servers = [bootstrap-url:9092]check.crcs = trueclient.dns.lookup = use_all_dns_ipsclient.id = consumer-integration.cubo-transactions-consumer-

Hi,I am trying to integrate Nifi with Pulsar. But unfortunately i cannot find any of Pulsar nar Processors around. I am running Nifi 1.16.3Thanks for your help.

Hello everyone you want to install the trial version of CDP private cloud on CentOS 7 machines on an external volume. Can I proceed using this guide?https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/installation/topics/cdpdc-trial-installation.html Or should I follow this?https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/installation/topics/cdpdc-manually-install-cloudera-software-packages.htmlThank you

Hi folks,My Datalake backup of CDP public always fails with this reason: Solr: Could not find any valid local directory for s3ablock-0001-. Anyone knows how to fix?

Hi everyone.I am getting stale alerts every hr, if the last point of contact between Data nodes and Namenode is more than 30 s we get these alerts.I am not able to find the root cause of this slowness, I have 32 cores system, but when this alert is generated in htop hdfs usage is more but not all cores are 100% utilized.DataNode Health SummaryDataNode Health: [Live=5, Stale=1, Dead=0]Please suggest changes required to resolve this.

anyone know why I have thousands and thousands of:2022-07-01 16:28:59,992 INFO org.apache.atlas.AtlasBaseClient: method=GET path=api/atlas/v2/entity/guid/ contentType=application/json; charset=UTF-8 accept=application/json status=2002022-07-01 16:29:00,017 INFO org.apache.atlas.AtlasBaseClient: method=GET path=api/atlas/v2/entity/guid/ contentType=application/json; charset=UTF-8 accept=application/json status=2002022-07-01 16:29:00,039 INFO org.apache.atlas.AtlasBaseClient: method=GET path=api/atlas/v2/entity/guid/ contentType=application/json; charset=UTF-8 accept=application/json status=2002022-07-01 16:29:00,061 INFO org.apache.atlas.AtlasBaseClient: method=GET path=api/atlas/v2/entity/guid/ contentType=application/json; charset=UTF-8 accept=application/json status=2002022-07-01 16:29:00,085 INFO org.apache.atlas.AtlasBaseClient: method=GET path=api/atlas/v2/entity/guid/ contentType=application/json; charset=UTF-8 accept=application/json status=2002022-07-01 16:29:00,108 INFO org.ap

I have a flow in which I make a call using InvokeHTTP. The http call returns a JSON response.This JSON response looks like this:{ "token_type": "Bearer", "expires_in": 3599, "ext_expires_in": 3599, "access_token": "eyJ0eXAiOiJKV1QiLCJub25jZSI6InAzVWN1NmRBcTNPRmVGbDM5cTVoZE9EaUpUSFRHT2hVVWdYTUNLMEsyVjQiLCJhbGciOiJSUzI1NiIsIng1dCI6IjJaUXBKM1VwYmpBWVhZR2FYRUpsOGxWMFRPSSIsImtpZCI6IjJaUXBKM1VwYmpBWVhZR2FYRUpsOGxWMFRPSSJ9.eyJhdWQiOiJodHRwczovL2dyYXBoLm1pY3Jvc29mdC5jb20iLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC9iZWJhN2VlNi05ZWRhLTQzZTEtOTY5OS0yMDIxYzVmOTI5NDEvIiwiaWF0IjoxNjU3NTUzNzI3LCJuYmYiOjE2NTc1NTM3MjcsImV4cCI6MTY1NzU1NzYyNywiYWlvIjoiRTJaZ1lMalI4cjV6ZWRSMHRxUkErWFdIV2QrZkJnQT0iLCJhcHBfZGlzcGxheW5hbWUiOiJhcHAtZm9yLW5pZmkiLCJhcHBpZCI6ImRlMjg0MjNmLTgxYmMtNDc5Ni1hYTU3LTIyYTM5NmJkNDliYSIsImFwcGlkYWNyIjoiMSIsImlkcCI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0L2JlYmE3ZWU2LTllZGEtNDNlMS05Njk5LTIwMjFjNWY5Mjk0MS8iLCJpZHR5cCI6ImFwcCIsIm9pZCI6IjAwN2EwNjkyLTg5ZjYtNDlmYi04ZDFhLTQwZGE3YjcwZjgwNSIsInJoIjoiMC5BVGdB

I have a huge dataset and I want to split it into partitioned table, but I'm wondering, the second column is date in int and I wonder is it possible to split table based on this, because I don't want to create three more columns with year, month and day. 

Hello, I have successfully migrated the h2 database of registry to Postgres sql but it did not migrate the users.xml and authorizations.xml to postgres. Could you please help in pointing me in right direction on what setting needs to be changed to get migrate the users and authorizations data aswell to Postgres SQL.@MattWho ThanksMeeran

How to write a dataframe (pandas,spark) with python from CML (Workbench or NoteBook) to a Cloudera Datahubcluster? Is it possible without pyspark? Thanks!

Hello all,I do not know how to configure multi users on 3 nodes Nifi Cluster:Currently, i use the following to generate client & server certificates using TLS toolkit:./tls-toolkit.sh standalone -B password -C 'CN=nifiadmin, OU=NIFI' -n 'node1,node2,node3' --nifiDnPrefix 'CN=' --nifiDnSuffix ', OU=NIFI' -o /shared_folder/vm_shared/certs_100722/ -K password -P password -S password I want to configure 3 users for this 3 nodes cluster.Your help would be greatly appreciated.Thank you.

I am trying to design a flow, where I get file details from a Sharepoint site's directory.For this I need to make 4 or more REST calls:1. Get access token from microsoft.2. Use the access token from previous JSON, to get site details.3. Use the site id from previous JSON, to get drive details.4. Use the drive id from previous response JSON , to list drive children. What would be the best approach to implement this in NiFi? Thanks 

Hi , im unable to connect impala shell ( kerberised cluster ) with below: command  on Edge node impala-shell -i hostname.abc.com -d default -kcase scenario :  we have two Edge node A and B , able to connect on edgenode A  and unable to connect on edgenode  B with same credentials. checked with different user is it also same as working  on Edge node A and edgenode B . observed that both have different python version  A has 2.7.5 as default and B has 3.7.0 is this the cause of connection , if yes how to rectify. ERROR screenshot shared. 

Hello, I have configured the nifi registry with an aws load balancerI have configured both type of load balancer (application(https)/ tcp) but none of them accessible from nifi nodes and not listing the buckets list.Below are the errors I am getting:with AWS network type load balancer:with application (https) load balancer: but if I add an individual registry node, every thing is working fine.Issue is with the load balancer only!! Please help what change needs to be done on the load balancer side.

Hello Experts, I am using org.apache.nifi.util.TestRunners in Nifi custom processor unit test cases, and code something like below.. Spoiler (Highlight to read)testRunner = TestRunners.newTestRunner(O9QueueRouting.class);....MockFlowFile ff = this.testRunner.enqueue(new ByteArrayInputStream(ffContents.getBytes(StandardCharsets.UTF_8)));Map<String, String> attrs = new HashMap<String, String>();attrs.put("routingheader", "headervalue");attrs.put("kafka.topic", "topic1");ff.putAttributes(attrs);testRunner.run();testRunner = TestRunners.newTestRunner(O9QueueRouting.class);....MockFlowFile ff = this.testRunner.enqueue(new ByteArrayInputStream(ffContents.getBytes(StandardCharsets.UTF_8)));Map<String, String> attrs = new HashMap<String, String>();attrs.put("routingheader", "headervalue");attrs.put("kafka.topic", "topic1");ff.putAttributes(attrs);testRunner.run();It was working earlier now started giving error like below -[ERROR] O9QueueRoutingTest.test:7

Hello!  Attached below is a flow I have created to perform a simple sorting task with a given CSV file 1. Getfile processor brings the target CSV file into the flow 2. QueryRecord processor executes SQL commands and send the flowfile two ways3. Each flow file is saved into Postgresql database with PutDatabaseRecord processors.  What I am trying to do is. 1. Keep a log of all SQL commands that are executed on Queryrecord processor and keep timestamp (When they were executed) of them 2. Save the log into a Postgresql table named 'validation_log" that has 3 columns. (CSV file name, SQL command, Timestamp) How can I make this work? 

Hi guys,  I need your assistance on querying hbase rest with special characters in the rowkey. Hi, I have the following rowkey and data in Hbase:  ROW COLUMN+CELL \xF2member-00002 column=cf:age, timestamp=2022-07-12T10:16:11.986, value=20    I am querying using the REST interface (version 0.0.3) as:  GET 'http://localhost:8080/member/%F2member-00002'  but always getting Not found.  What should be the correct url encoded rowkey? Thanks!

Change Company Name When I created my registration I chose the wrong option

Hello, I'm using nifi in version 1.13.1 and there got no problems but with upgrade to 1.16.2 got following issue with Merge Content Processor: Flow files are merged by flow file name. When 1500 messages come in then its just stopping with merging. When I stop the processor manually and start it again he will again pull 1500 messages and stop again. Implementation is old, works fine on 1.13.1 but with 1.16.2 something has change and I don't know where to find root cause. Any idea what I should check ?