Member since
06-14-2022
62
Posts
2
Kudos Received
0
Solutions
05-21-2025
10:06 PM
Hi, can you please help? @MattWho
... View more
04-24-2025
03:56 AM
I am facing the same issue. The processor is configured rto get only 10 messages at a time. I have over 500 messages in the inbox, all unread. If I "Run once" the ConsumeIMAP processor it get 10 messages every time. If I "run" the processor (regrdless what is the interval and if it's TimeDriven or CronDriven) it only get 1 message each time, but when I stop the processor it release 9 extra messages . I don't know what happens with the rest of the emails when the process executed multiple intervals, I don't know if anything is lost. The processor has no configuration for run duration: My processor configuration:
... View more
11-06-2024
05:31 AM
Hello, you can see other solution here: https://community.cloudera.com/t5/Support-Questions/Nifi-Execute-Groovy-Script/m-p/344265#M234128
... View more
07-31-2023
08:26 AM
Hi MattWho, I am not able to access the NiFi UI. There its saying "This site can’t be reached". Below is the error that I am getting: 2023-07-31 14:01:24,921 ERROR [External Resource Provider Service - NAR Auto-Loader Provider - NAR Auto-Loader Provider - b1ef45bd-ef3f-4866-8f9e-5b4b1e38df8a] o.a.n.f.r.ConflictResolvingExternalResourceProviderWorker Error during polling for external resources javax.net.ssl.SSLException: readHandshakeRecord at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1309) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:435) at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1567) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1495) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352) at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:409) at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:273) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:300) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:675) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:674) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:422) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:318) at org.apache.nifi.registry.client.impl.JerseyBundleVersionClient.lambda$getBundleVersions$2(JerseyBundleVersionClient.java:173) at org.apache.nifi.registry.client.impl.AbstractJerseyClient.executeAction(AbstractJerseyClient.java:103) at org.apache.nifi.registry.client.impl.JerseyBundleVersionClient.getBundleVersions(JerseyBundleVersionClient.java:156) at org.apache.nifi.registry.extension.NiFiRegistryExtensionRegistry.getExtensionBundleMetadata(NiFiRegistryExtensionRegistry.java:85) at org.apache.nifi.registry.extension.NiFiRegistryExternalResourceProvider.listResources(NiFiRegistryExternalResourceProvider.java:69) at org.apache.nifi.flow.resource.ConflictResolvingExternalResourceProviderWorker.poll(ConflictResolvingExternalResourceProviderWorker.java:113) at org.apache.nifi.flow.resource.ConflictResolvingExternalResourceProviderWorker.run(ConflictResolvingExternalResourceProviderWorker.java:86) at java.lang.Thread.run(Thread.java:748) Suppressed: java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81) at sun.security.ssl.TransportContext.fatal(TransportContext.java:355) at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:438) ... 28 common frames omitted Caused by: java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at sun.security.ssl.SSLSocketOutputRecord.flush(SSLSocketOutputRecord.java:251) at sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:89) at sun.security.ssl.CertificateVerify$T12CertificateVerifyProducer.produce(CertificateVerify.java:769) at sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:421) at sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(ServerHelloDone.java:182) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:152) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1392) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1300) ... 29 common frames omitted 2023-07-31 14:06:24,946 ERROR [External Resource Provider Service - NAR Auto-Loader Provider - NAR Auto-Loader Provider - b1ef45bd-ef3f-4866-8f9e-5b4b1e38df8a] o.a.n.f.r.ConflictResolvingExternalResourceProviderWorker Error during polling for external resources javax.net.ssl.SSLException: readHandshakeRecord at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1309) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:435) at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1567) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1495) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352) at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:409) at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:273) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:300) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:675) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:674) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:422) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:318) at org.apache.nifi.registry.client.impl.JerseyBundleVersionClient.lambda$getBundleVersions$2(JerseyBundleVersionClient.java:173) at org.apache.nifi.registry.client.impl.AbstractJerseyClient.executeAction(AbstractJerseyClient.java:103) at org.apache.nifi.registry.client.impl.JerseyBundleVersionClient.getBundleVersions(JerseyBundleVersionClient.java:156) at org.apache.nifi.registry.extension.NiFiRegistryExtensionRegistry.getExtensionBundleMetadata(NiFiRegistryExtensionRegistry.java:85) at org.apache.nifi.registry.extension.NiFiRegistryExternalResourceProvider.listResources(NiFiRegistryExternalResourceProvider.java:69) Can you please help? Thank you.
... View more
06-24-2023
05:55 PM
First, the incoming Data should be proper JSON that can be easily parsed and processed...i.e. something like this [
[
{
"ID": "100000",
"Date": "2022-09-22",
"Start Time": "08:00",
"End Time": "14:00"
},
{
"ID": "100001",
"Date": "2022-09-02",
"Start Time": "08:00",
"End Time": "14:00"
},
{
"ID": "100002",
"Date": "2022-09-02",
"Start Time": "08:00",
"End Time": "14:00"
}
],
{
"res_data": [
{
"Record": "Record 2",
"Description": "Invalid values for ID or Date"
},
{
"Record": "Record 3",
"Description": "Invalid values for ID or Date"
}
]
}
] Then Groovy code like this: import groovy.json.JsonOutput
import groovy.json.JsonSlurper
JsonSlurper jsonSlurper = new JsonSlurper()
JsonOutput jsonOutput = new JsonOutput()
List data = jsonSlurper.parseText('''
[
[
{
"ID": "100000",
"Date": "2022-09-22",
"Start Time": "08:00",
"End Time": "14:00"
},
{
"ID": "100001",
"Date": "2022-09-02",
"Start Time": "08:00",
"End Time": "14:00"
},
{
"ID": "100002",
"Date": "2022-09-02",
"Start Time": "08:00",
"End Time": "14:00"
}
],
{
"res_data": [
{
"Record": "Record 2",
"Description": "Invalid values for ID or Date"
},
{
"Record": "Record 3",
"Description": "Invalid values for ID or Date"
}
]
}
]
''')
println(data)
println("=" * 80)
List<Map<String, String>> transformedList = []
data[0].eachWithIndex { map, index ->
Map<String, Object> record = data[1]["res_data"].find { it["Record"] == "Record ${index + 1}" }
if (record) {
map.putAll(record)
}
transformedList.add(map)
}
println(jsonOutput.prettyPrint(jsonOutput.toJson(transformedList))) Creates an output like this: [[[ID:100000, Date:2022-09-22, Start Time:08:00, End Time:14:00], [ID:100001, Date:2022-09-02, Start Time:08:00, End Time:14:00], [ID:100002, Date:2022-09-02, Start Time:08:00, End Time:14:00]], [res_data:[[Record:Record 2, Description:Invalid values for ID or Date], [Record:Record 3, Description:Invalid values for ID or Date]]]]
================================================================================
[
{
"ID": "100000",
"Date": "2022-09-22",
"Start Time": "08:00",
"End Time": "14:00"
},
{
"ID": "100001",
"Date": "2022-09-02",
"Start Time": "08:00",
"End Time": "14:00",
"Record": "Record 2",
"Description": "Invalid values for ID or Date"
},
{
"ID": "100002",
"Date": "2022-09-02",
"Start Time": "08:00",
"End Time": "14:00",
"Record": "Record 3",
"Description": "Invalid values for ID or Date"
}
]
... View more
06-24-2023
05:18 PM
This is Groovy code that achieves what you've mentioned and should be able to adapt to your scripted processor. import groovy.json.JsonOutput
import groovy.json.JsonSlurper
JsonSlurper jsonSlurper = new JsonSlurper()
JsonOutput jsonOutput = new JsonOutput()
Map<String, String> data = jsonSlurper.parseText('''{
"Message": "\nRecord 1:\nRequired data is missing. \n\nRecord 2:\nprocessing failed\n"
}''')
println(data)
println("=" * 80)
List<String> records = data.Message.split("\\nRecord\\s\\d+:\\n").findAll { it.trim() != "" }
println(records)
println("=" * 80)
Map<String, String> messages = [:]
records.eachWithIndex { message, index ->
messages["Record ${index + 1}"] = message.trim()
}
println(jsonOutput.toJson(messages)) This is the output: [Message:
Record 1:
Required data is missing.
Record 2:
processing failed
]
================================================================================
[Required data is missing.
, processing failed
]
================================================================================
{"Record 1":"Required data is missing.","Record 2":"processing failed"}
... View more
03-21-2023
11:49 AM
@Techie123 What is observed in the endpoint logs for these transactions? Can you share the complete stack trace from the NiFi logs? Can you share your invokeHTTP processor configurations? Can you share your NiFi version? Thanks, Matt
... View more
03-21-2023
06:16 AM
Yes, I am doing the same. And how to maintain the order of the dynamic properties in Invokehttp processor as the order is also important else it is saying bad request. Whenever i am adding those, these are automatically arranging the alphabetical orders instead the order on which i am adding
... View more
02-09-2023
06:22 AM
1 Kudo
@Techie123 Well, like i said, you have to learn the aws side of providing access to a bucket. A public bucket starting point will show you what you have to do, inside of the bucket config, to allow other systems to access that bucket. For example starting from public open bucket, to whatever access control level you ultimately need to have. Getting lost in that space is not necessarily a "nifi" thing.... so my recommendation is to build nifi with public bucket, THEN when it works, start testing the deeper access requirements. The controller service configuration provides multiple ways to access a bucket and a bunch of settings. Make sure you have a working access/key credentials tested directly in the processor before moving to the Controller Service.
... View more
01-10-2023
01:50 AM
Hello @Techie123, can you tell a bit more about what you want to achive? Do you want just use the first or second object of 'data'? Then you have to change: "data": {
"0": { "data": {
"1": { Or how do you want to progress - can it that the numbers are not similiar? We dont know how your results can be look like Greetings
... View more