Member since
07-30-2019
2901
Posts
1438
Kudos Received
844
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
18 | 04-17-2024 11:30 AM | |
26 | 04-16-2024 05:36 AM | |
26 | 04-15-2024 05:31 AM | |
118 | 04-03-2024 05:59 AM | |
127 | 04-02-2024 01:22 PM |
08-10-2023
06:52 AM
1 Kudo
@Anderosn In-between your SplitJson and PuSQL processors are you rebalancing the FlowFile across multiple nodes in a NiFi cluster? Are you routing any of the split Json messges down a different dataflow path that does not lead to this pusSQL processor? The reason I ask is because the splitJson processor will write the following FlowFile attributes to each new FlowFile created (each split): The fragment.identifier value and fragment.count are used by the putSQL processor when "Support FragmentTransactions" is set to "true" (default). This means that, if not all split jsons are present at this putSQL and located on the same node of the NiFi cluster, the FlowFiles part of the same fragment.identifier will not be processed and remain on the inbound connection to the PutSQL. I'd start my listing the connection and checking these attributes to verify the fragment.count is "10", the fragment.identifier has same value on all 10, and fragment.index value shows numbers 1 to 10 across those 10 FlowFiles. If making sure all fragments are processed in same transaction is not a requirement for your dataflow, try changing "Support Fragmented Transactions" to false and see if these 10 FlowFiles get successfully executed by your putSQL processor. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
08-09-2023
01:33 AM
Hey Matt! I am truly grateful for your answer. You made me realize what I was doing wrong and I learned a lot while reading trough your explanation. I am going to revise the documentation again, as I am realizing that I did not had some concepts crystal clear before. Thanks again and I wish you a good day.
... View more
08-08-2023
12:01 PM
Yep! Had moved on to that after your response and written up a JIRA for my team to build the needed tool. Thanks again for the response. My plan is the below > Find all the processors which may have state, then check if each has state, and if so, clear it. Find processors of used types which can contain state: e.g., QueryDatabaseTableRecord For each, pull the state by the ID and check if it has any stored state if totalEntryCount of cluster or local state is non-zero If so, call the clear state endpoint Query URL: /nifi-api/flow/search-results?q=QueryDatabaseTableRecord Pull Processor State: /nifi-api/processors/41584e33-adc6-171d-0000-0000581caccb/state Clear Processor State: POST /nifi-api/processors/41584e33-adc6-171d-0000-0000581caccb/state/clear-requests
... View more
08-08-2023
08:54 AM
Sure. I had to make some extract it from a larger PG and make some adjustments due to key code being in a parameter. But the remainder of the parameters are really placeholders for your environment information. The main complication might be that I used a few external controller services that you will have to replace. But their names in the externalControllerServices JSON should help you replace them.
... View more
08-07-2023
11:10 AM
@sam_s0ni Apache NiFi never intended with the context of the NiFi framework to allow parameters to be dynamically accessed within NiFi Expression Language (NEL). The EvaluateElString() exposed a bug in which this was possible while never intended. That bug was addressed in apache NiFi 1.19 thus resulting in what you are seeing. As an alternative, you could store your schemas in an external database and pull those schemas back dynamically based on a FlowFile attribute instead of using Parameter contexts to store those schemas. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
08-07-2023
04:01 AM
Thanks MattWho, Yeah We figure out that the causing of issue the special character. Thank you Very Much
... View more
08-04-2023
01:37 PM
@MWM MergeContent is probably not the processor you want to use to merge your JSON files as it will simply concatenate them together instead of creating one larger json. You should use the MergeRecord processor instead. In this processor you can specify a JsonTreeReader to read your source Json files and then use a CSVRecordSetWriter to output merged CSV file(s). The rest of the MergeRecord processor configuration will control how many source Json records are added in to a single output CSV. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
08-04-2023
01:30 PM
@samrathal Is your Windows server hosting a SFTP server that you can connect to? If not, I'd expect GetSFTP/FetchSFTP to fail to connect. If you can't reach your Windows file from command line on the server where NiFi resides, you are not going to be able to reach via a NiFi dataflow. 1. Setting up a windows share that you mount to your linux box would allow you to use ListFile and FetchFile processors. 2. Setting up an SFTP server on your Windows server would allow you to use ListSFTP and FetchSFTP processors. 3. Setting up SMB on your Windows server would allow you to use the ListSMB and FetchSMB processors. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
07-31-2023
08:26 AM
Hi MattWho, I am not able to access the NiFi UI. There its saying "This site can’t be reached". Below is the error that I am getting: 2023-07-31 14:01:24,921 ERROR [External Resource Provider Service - NAR Auto-Loader Provider - NAR Auto-Loader Provider - b1ef45bd-ef3f-4866-8f9e-5b4b1e38df8a] o.a.n.f.r.ConflictResolvingExternalResourceProviderWorker Error during polling for external resources javax.net.ssl.SSLException: readHandshakeRecord at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1309) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:435) at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1567) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1495) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352) at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:409) at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:273) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:300) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:675) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:674) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:422) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:318) at org.apache.nifi.registry.client.impl.JerseyBundleVersionClient.lambda$getBundleVersions$2(JerseyBundleVersionClient.java:173) at org.apache.nifi.registry.client.impl.AbstractJerseyClient.executeAction(AbstractJerseyClient.java:103) at org.apache.nifi.registry.client.impl.JerseyBundleVersionClient.getBundleVersions(JerseyBundleVersionClient.java:156) at org.apache.nifi.registry.extension.NiFiRegistryExtensionRegistry.getExtensionBundleMetadata(NiFiRegistryExtensionRegistry.java:85) at org.apache.nifi.registry.extension.NiFiRegistryExternalResourceProvider.listResources(NiFiRegistryExternalResourceProvider.java:69) at org.apache.nifi.flow.resource.ConflictResolvingExternalResourceProviderWorker.poll(ConflictResolvingExternalResourceProviderWorker.java:113) at org.apache.nifi.flow.resource.ConflictResolvingExternalResourceProviderWorker.run(ConflictResolvingExternalResourceProviderWorker.java:86) at java.lang.Thread.run(Thread.java:748) Suppressed: java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81) at sun.security.ssl.TransportContext.fatal(TransportContext.java:355) at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:438) ... 28 common frames omitted Caused by: java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at sun.security.ssl.SSLSocketOutputRecord.flush(SSLSocketOutputRecord.java:251) at sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:89) at sun.security.ssl.CertificateVerify$T12CertificateVerifyProducer.produce(CertificateVerify.java:769) at sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:421) at sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(ServerHelloDone.java:182) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:152) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1392) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1300) ... 29 common frames omitted 2023-07-31 14:06:24,946 ERROR [External Resource Provider Service - NAR Auto-Loader Provider - NAR Auto-Loader Provider - b1ef45bd-ef3f-4866-8f9e-5b4b1e38df8a] o.a.n.f.r.ConflictResolvingExternalResourceProviderWorker Error during polling for external resources javax.net.ssl.SSLException: readHandshakeRecord at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1309) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:435) at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1567) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1495) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352) at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:409) at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:273) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:300) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:675) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:674) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:422) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:318) at org.apache.nifi.registry.client.impl.JerseyBundleVersionClient.lambda$getBundleVersions$2(JerseyBundleVersionClient.java:173) at org.apache.nifi.registry.client.impl.AbstractJerseyClient.executeAction(AbstractJerseyClient.java:103) at org.apache.nifi.registry.client.impl.JerseyBundleVersionClient.getBundleVersions(JerseyBundleVersionClient.java:156) at org.apache.nifi.registry.extension.NiFiRegistryExtensionRegistry.getExtensionBundleMetadata(NiFiRegistryExtensionRegistry.java:85) at org.apache.nifi.registry.extension.NiFiRegistryExternalResourceProvider.listResources(NiFiRegistryExternalResourceProvider.java:69) Can you please help? Thank you.
... View more