Member since
02-13-2020
9
Posts
0
Kudos Received
0
Solutions
02-09-2021
02:00 AM
Hello @MattWho Thank you for your awesome response as always. Yes our NiFi is in 5 node cluster and GetSFTP is configured to run only on primary node. After further analysis we only could arrive at one conclusion, as soon as we pull the file the source system is moving the file to a different location. Now, we are able to pick many files without exception but only for certain rare cases we are encountering this exception. Not sure if this is because of network latency, could this be possible ? but the jsch library is throwing the exception. Yes, since FetchSFTP provides us failure output we could add a retry if possible, so yes we are moving from GetSFTP to List and Fetch combo. Thank you
... View more
02-01-2021
09:05 PM
The GetSFTP processor is throwing the FlowFileAccessException when trying to read a file from a remote directory. If I look at the error stack trace I can see that at StreamUtils.copy is where IOException has happened. Is it something like file was moved when actually reading it ? Can you guys please help me understand why this issue occurring and how I can avoid it ? o.a.nifi.processors.standard.GetSFTP GetSFTP[id=7d5fe4b5-7b67-3779-8de7-00e903c20c34] Unable to retrieve file FILENAME due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.io.IOException: error: org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from com.jcraft.jsch.ChannelSftp$2@6e1e8c5e for StandardFlowFileRecord[uuid =a50179f7-b631-4aa8-9e74-a2dca44d89bd,claim=,offset=0,name=a50179f7-b631-4aa8-9e74-a2dca44d89bd,size=0] due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.io.IOException: errororg.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from com.jcraft.jsch.ChannelSftp$2@6e1e8c5e for StandardFlowFileRecord[uuid=a50179f7- b631-4aa8-9e74-a2dca44d89bd,claim=,offset=0,name=a50179f7-b631-4aa8-9e74-a2dca44d89bd,size=0] due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.io.IOException: error at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:3045) at org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:194) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:205) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.io.IOException: error at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:3038) ... 12 common frames omitted Caused by: java.io.IOException: error at com.jcraft.jsch.ChannelSftp$2.close(ChannelSftp.java:1532) at com.jcraft.jsch.ChannelSftp$2.read(ChannelSftp.java:1444) at com.jcraft.jsch.ChannelSftp$2.read(ChannelSftp.java:1364) at org.apache.nifi.controller.repository.io.TaskTerminationInputStream.read(TaskTerminationInputStream.java:62) at org.apache.nifi.stream.io.StreamUtils.copy(StreamUtils.java:35) at org.apache.nifi.controller.repository.FileSystemRepository.importFrom(FileSystemRepository.java:744) at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:3035) ... 12 common frames omitted
... View more
Labels:
- Labels:
-
Apache MiNiFi
-
Apache NiFi
09-13-2020
05:22 AM
I am using MiNiFi(0.5.0) to pick and transfer files from a Linux machine to NiFi(1.9.1). In few cases i can see duplicate files being transferred to NiFi. Flow is setup as below GetFile ->LogAttribute -> PutFile(archive) -> RemoteProcessGroup Log 1: minifi-app.log:2020-09-13 06:09:00,760 INFO [Timer-Driven Process Thread-4] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=Input_Port_GES,targets=https://nifi1.myorg.com:7071/nifi] Successfully sent [StandardFlowFileRecord[uuid=345d9b6d-e9f7-4dd8-ad9a-a9d66fdfd902,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1599937413340-1, container=default, section=1], offset=1073154, length=237],offset=0,name=RandomFile1154.txt,size=237]] (237 bytes) to nifi://nifi1.myorg.com:7074 in 48 milliseconds at a rate of 4.74 KB/sec Log 2 : minifi-app.log:2020-09-13 06:09:01,910 INFO [Timer-Driven Process Thread-5] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=Input_Port_GES,targets=https://nifi1.myorg.com:7071/nifi] Successfully sent [StandardFlowFileRecord[uuid=f74eb941-a233-4f9e-86ff-07723940f012,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1599937413340-1, container=default, section=1], offset=1109014, length=237],offset=0,name=RandomFile1154.txt,size=237], StandardFlowFileRecord[uuid=522a4350-4cab-476c-a087-a3793101412e,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1599937413340-1, container=default, section=1], offset=1074338, length=235],offset=0,name=RandomFile1346.txt,size=235]] (472 bytes) to nifi://nifi1.myorg.com:7074 in 30 milliseconds at a rate of 14.9 KB/sec In the above log its seen that RandomFile1154.txt file is transferred once at 2020-09-13 06:09:00,760 and then again at 2020-09-13 06:09:01,910 along with RandomFile1346.txt. I went through the StandardRemoteGroupPort code and i can see that once the transfer is successful, session is committed and it should not be available for next transfer. I have added the logs to see if my GetFile picked the file twice, but this is not the case, the log printed only once. Please share your thoughts on this
... View more
Labels:
- Labels:
-
Apache MiNiFi
-
Apache NiFi
03-28-2020
07:28 PM
@MattWho It looks like there are modifications with respect the certificate validation in 1.11.3. I installed the 1.9.1 version in parallel on different port and i was able to authenticate successfully by using existing keystore and truststore files.
... View more
03-25-2020
06:32 AM
@MattWho I tried running the cluster with only one node and it's still giving me the same error WARN [Replicate Request Thread-1] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/current-user to nifisitapp01.xxxxx.com:7081 due to javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target so now its confirmed to be a client to node issue. I don't understand how come the curl command is working curl --insecure -v --cert-type P12 --cert cert.p12:[passkey] https://nifisitapp01.XXXXX.com:7081/nifi/ My keystore has server certificate with 3 levels as mentioned below, so its a complete certificate. Level 1 : Owner : nifisitapp01 Issuer : Intermediate Level 2: Owner : Intermediate Issuer : Owner Level 3: Owner : Owner Issuer : Owner The trust store contains all these 3 level certificates and my client certificate and owner(localhost) certificate as well.
... View more
03-24-2020
04:29 AM
Upgrading to NiFi version - 1.11.3 Issue - SSL Mutual authentication issue Currently using NiFi - 1.9.0 and implemented SSL mutual authentication and its working fine. I am trying to upgrade to a newer version and i have installed 1.11.3 version in the parallel mode on the same server where 1.9.0 is installed in different port. For the newer version i am using the same keystore and truststore along with the client certificates, basically i have just copied these from 1.9.0 version. when i am trying to access the webgui i am getting javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target But when i run the crul command i am getting the response from the URL curl --insecure -v --cert-type P12 --cert cert.p12:[passkey] https://nifisitapp01.XXXXX.com:7081/nifi/ output * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * successfully set certificate verify locations: * CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt CApath: none * TLSv1.2 (OUT), TLS header, Certificate Status (22): } [5 bytes data] * TLSv1.2 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.2 (IN), TLS handshake, Server hello (2): { [85 bytes data] * TLSv1.2 (IN), TLS handshake, Certificate (11): { [4337 bytes data] * TLSv1.2 (IN), TLS handshake, Server key exchange (12): { [333 bytes data] * TLSv1.2 (IN), TLS handshake, Request CERT (13): { [1240 bytes data] * TLSv1.2 (IN), TLS handshake, Server finished (14): { [4 bytes data] * TLSv1.2 (OUT), TLS handshake, Certificate (11): } [1706 bytes data] * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): } [70 bytes data] * TLSv1.2 (OUT), TLS handshake, CERT verify (15): } [264 bytes data] * TLSv1.2 (OUT), TLS change cipher, Client hello (1): } [1 bytes data] * TLSv1.2 (OUT), TLS handshake, Finished (20): } [16 bytes data] * TLSv1.2 (IN), TLS change cipher, Client hello (1): { [1 bytes data] * TLSv1.2 (IN), TLS handshake, Finished (20): { [16 bytes data] * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-SHA384 * ALPN, server did not agree to a protocol * Server certificate: * subject: C=XX; ST=XXXX; L=XXXX; O=SOME ORGANIZATION; OU=X OPERATIONAL UNIT; CN=nifisitapp01.XXXXX.com * start date: Jul 23 06:36:29 2019 GMT * expire date: Jul 21 06:36:29 2024 GMT * issuer: DC=COM; DC=XCOMPANY; DC=UAT; CN=XCompany Issuing CA * SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway. } [5 bytes data] > GET /nifi/ HTTP/1.1 > Host: nifisitapp01.XXXXX.com:7081 > User-Agent: curl/7.47.1 > Accept: */* > { [5 bytes data] < HTTP/1.1 200 OK < Date: Tue, 24 Mar 2020 10:27:57 GMT < X-Frame-Options: SAMEORIGIN < Content-Security-Policy: frame-ancestors 'self' < X-XSS-Protection: 1; mode=block < Strict-Transport-Security: max-age=31540000 < Content-Type: text/html;charset=utf-8 < Vary: Accept-Encoding, User-Agent < Transfer-Encoding: chunked < Server: Jetty(9.4.19.v20190610) < { [16037 bytes data] I am not able to figure out what is actually missing, is it that browser i not passing the right certificate to the server ? Please share your inputs. Thank you
... View more
Labels:
- Labels:
-
Apache NiFi
03-04-2020
01:41 AM
We use NiFi mainly for file transfer, processing these files etc. The requirement is to pull reports from NiFi which has the statistics such as how many files were processed from NiFi, how many of them are successfully transferred to destination, how many are failed and waiting in different queues. The NiFi rest api documentation(https://nifi.apache.org/docs/nifi-docs/rest-api/index.html) has the API's to pull this information, but i did not find any info on how to use these API's. Is the API's exposed out of the box or should it be enabled somehow. Please share your suggestions on this. If there is any other way to extract the data statistics from NiFi please share them as well.
... View more
Labels:
- Labels:
-
Apache MiNiFi
-
Apache NiFi
02-13-2020
05:52 AM
Thank you for sharing your thoughts around this integration, it really really helps. I will try to do a POC based out of your suggestion
... View more
02-13-2020
03:04 AM
We want to store our sensitive information such as passwords, private keys in Vault and retrieve it from Vault on need basis, kindly share your inputs which is much needed.
... View more
Labels:
- Labels:
-
Apache NiFi