Hello, you can see other solution here: https://community.cloudera.com/t5/Support-Questions/Nifi-Execute-Groovy-Script/m-p/344265#M234128 ... View more

@Techie123  Different processor are built to execute differently when scheduled to execute by different Apache NiFi contributors. Some ways to overcome this are to possibly: Adjust the run duration so that processor runs longer when scheduled to process more FlowFiles or produce more FlowFile from an internal queue in a single scheduled execution. Configure your cron to execute multiple times within a short period of time. so maybe every second for 15 seconds within a specific hour and minute. Combination of both the above. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

Hi MattWho,   I am not able to access the NiFi UI. There its saying "This site can’t be reached". Below is the error that I am getting:     2023-07-31 14:01:24,921 ERROR [External Resource Provider Service - NAR Auto-Loader Provider - NAR Auto-Loader Provider - b1ef45bd-ef3f-4866-8f9e-5b4b1e38df8a] o.a.n.f.r.ConflictResolvingExternalResourceProviderWorker Error during polling for external resources javax.net.ssl.SSLException: readHandshakeRecord at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1309) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:435) at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1567) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1495) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352) at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:409) at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:273) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:300) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:675) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:674) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:422) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:318) at org.apache.nifi.registry.client.impl.JerseyBundleVersionClient.lambda$getBundleVersions$2(JerseyBundleVersionClient.java:173) at org.apache.nifi.registry.client.impl.AbstractJerseyClient.executeAction(AbstractJerseyClient.java:103) at org.apache.nifi.registry.client.impl.JerseyBundleVersionClient.getBundleVersions(JerseyBundleVersionClient.java:156) at org.apache.nifi.registry.extension.NiFiRegistryExtensionRegistry.getExtensionBundleMetadata(NiFiRegistryExtensionRegistry.java:85) at org.apache.nifi.registry.extension.NiFiRegistryExternalResourceProvider.listResources(NiFiRegistryExternalResourceProvider.java:69) at org.apache.nifi.flow.resource.ConflictResolvingExternalResourceProviderWorker.poll(ConflictResolvingExternalResourceProviderWorker.java:113) at org.apache.nifi.flow.resource.ConflictResolvingExternalResourceProviderWorker.run(ConflictResolvingExternalResourceProviderWorker.java:86) at java.lang.Thread.run(Thread.java:748) Suppressed: java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81) at sun.security.ssl.TransportContext.fatal(TransportContext.java:355) at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:438) ... 28 common frames omitted Caused by: java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at sun.security.ssl.SSLSocketOutputRecord.flush(SSLSocketOutputRecord.java:251) at sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:89) at sun.security.ssl.CertificateVerify$T12CertificateVerifyProducer.produce(CertificateVerify.java:769) at sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:421) at sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(ServerHelloDone.java:182) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:152) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1392) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1300) ... 29 common frames omitted 2023-07-31 14:06:24,946 ERROR [External Resource Provider Service - NAR Auto-Loader Provider - NAR Auto-Loader Provider - b1ef45bd-ef3f-4866-8f9e-5b4b1e38df8a] o.a.n.f.r.ConflictResolvingExternalResourceProviderWorker Error during polling for external resources javax.net.ssl.SSLException: readHandshakeRecord at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1309) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:435) at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1567) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1495) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352) at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:409) at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:273) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:300) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:675) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:674) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:422) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:318) at org.apache.nifi.registry.client.impl.JerseyBundleVersionClient.lambda$getBundleVersions$2(JerseyBundleVersionClient.java:173) at org.apache.nifi.registry.client.impl.AbstractJerseyClient.executeAction(AbstractJerseyClient.java:103) at org.apache.nifi.registry.client.impl.JerseyBundleVersionClient.getBundleVersions(JerseyBundleVersionClient.java:156) at org.apache.nifi.registry.extension.NiFiRegistryExtensionRegistry.getExtensionBundleMetadata(NiFiRegistryExtensionRegistry.java:85) at org.apache.nifi.registry.extension.NiFiRegistryExternalResourceProvider.listResources(NiFiRegistryExternalResourceProvider.java:69)   Can you please help? Thank you.   ... View more

First, the incoming Data should be proper JSON that can be easily parsed and processed...i.e. something like this [ [ { "ID": "100000", "Date": "2022-09-22", "Start Time": "08:00", "End Time": "14:00" }, { "ID": "100001", "Date": "2022-09-02", "Start Time": "08:00", "End Time": "14:00" }, { "ID": "100002", "Date": "2022-09-02", "Start Time": "08:00", "End Time": "14:00" } ], { "res_data": [ { "Record": "Record 2", "Description": "Invalid values for ID or Date" }, { "Record": "Record 3", "Description": "Invalid values for ID or Date" } ] } ]   Then Groovy code like this: import groovy.json.JsonOutput import groovy.json.JsonSlurper JsonSlurper jsonSlurper = new JsonSlurper() JsonOutput jsonOutput = new JsonOutput() List data = jsonSlurper.parseText(''' [ [ { "ID": "100000", "Date": "2022-09-22", "Start Time": "08:00", "End Time": "14:00" }, { "ID": "100001", "Date": "2022-09-02", "Start Time": "08:00", "End Time": "14:00" }, { "ID": "100002", "Date": "2022-09-02", "Start Time": "08:00", "End Time": "14:00" } ], { "res_data": [ { "Record": "Record 2", "Description": "Invalid values for ID or Date" }, { "Record": "Record 3", "Description": "Invalid values for ID or Date" } ] } ] ''') println(data) println("=" * 80) List<Map<String, String>> transformedList = [] data[0].eachWithIndex { map, index -> Map<String, Object> record = data[1]["res_data"].find { it["Record"] == "Record ${index + 1}" } if (record) { map.putAll(record) } transformedList.add(map) } println(jsonOutput.prettyPrint(jsonOutput.toJson(transformedList)))   Creates an output like this: [[[ID:100000, Date:2022-09-22, Start Time:08:00, End Time:14:00], [ID:100001, Date:2022-09-02, Start Time:08:00, End Time:14:00], [ID:100002, Date:2022-09-02, Start Time:08:00, End Time:14:00]], [res_data:[[Record:Record 2, Description:Invalid values for ID or Date], [Record:Record 3, Description:Invalid values for ID or Date]]]] ================================================================================ [ { "ID": "100000", "Date": "2022-09-22", "Start Time": "08:00", "End Time": "14:00" }, { "ID": "100001", "Date": "2022-09-02", "Start Time": "08:00", "End Time": "14:00", "Record": "Record 2", "Description": "Invalid values for ID or Date" }, { "ID": "100002", "Date": "2022-09-02", "Start Time": "08:00", "End Time": "14:00", "Record": "Record 3", "Description": "Invalid values for ID or Date" } ] ... View more

This is Groovy code that achieves what you've mentioned and should be able to adapt to your scripted processor. import groovy.json.JsonOutput import groovy.json.JsonSlurper JsonSlurper jsonSlurper = new JsonSlurper() JsonOutput jsonOutput = new JsonOutput() Map<String, String> data = jsonSlurper.parseText('''{ "Message": "\nRecord 1:\nRequired data is missing. \n\nRecord 2:\nprocessing failed\n" }''') println(data) println("=" * 80) List<String> records = data.Message.split("\\nRecord\\s\\d+:\\n").findAll { it.trim() != "" } println(records) println("=" * 80) Map<String, String> messages = [:] records.eachWithIndex { message, index -> messages["Record ${index + 1}"] = message.trim() } println(jsonOutput.toJson(messages)) This is the output: [Message: Record 1: Required data is missing. Record 2: processing failed ] ================================================================================ [Required data is missing. , processing failed ] ================================================================================ {"Record 1":"Required data is missing.","Record 2":"processing failed"}   ... View more

@Techie123  What is observed in the endpoint logs for these transactions? Can you share the complete stack trace from the NiFi logs? Can you share your invokeHTTP processor configurations? Can you share your NiFi version? Thanks, Matt ... View more

Yes, I am doing the same. And how to maintain the order of the dynamic properties in Invokehttp processor as the order is also important else it is saying bad request. Whenever i am adding those, these are automatically arranging the alphabetical orders instead the order on which i am adding  ... View more

@Techie123 Well, like i said,  you have to learn the aws side of providing access to a bucket.  A public bucket starting point will show you what you have to do, inside of the bucket config, to allow other systems to access that bucket.   For example starting from public open bucket, to whatever access control level you ultimately need to have.   Getting lost in that space is not necessarily a "nifi" thing....  so my recommendation is to build nifi with public bucket,  THEN when it works, start testing the deeper access requirements.   The controller service configuration provides multiple ways to access a bucket and a bunch of settings.       Make sure you have a working access/key credentials tested directly in the processor before moving to the Controller Service.     ... View more