Member since
04-09-2018
8
Posts
1
Kudos Received
0
Solutions
03-20-2019
05:54 PM
Hi, I want to use nifi InvokeHTTP processor to get data from a remote URL. First, I have tested the following URL with CURL from CLI of my nifi host, retrieving me the file I want so it is OK : curl --insecure --user <USER>:<PASSWORD> https://<MY_REMOTE_URL>; The same URL with GET method set in InvokeHTTP processor gives me a SSLHandshakeException (user/password are respectively set in <Basic Authentication Username> and <Basic Authentication Password>) Routing to Failure due to exception: sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target This error is skipped with CURL thanks to --insecure option, disabling SSL certificate verification. So I would like to know if a workaround exists with nifi to disable SSL certificate verification ? (I currently cannot get a certificate allowing me accessing to remote host). Or is it possible to provide some options to InvokeHTTP processor ? (I know we can set headers using attributes, but what about options like "--insecure" or "-k" ?) The only way I have found at the moment to achieve what I want is encapsulating the curl call into an ExecuteScript processor, but this solution is not totally satisfying for me. Thanks Benjamin
... View more
Labels:
- Labels:
-
Apache NiFi
07-10-2018
01:04 PM
Thank you for your answer, The second hash is performed after a PutHDFS and not a ListHDFS (I have edited my post, sorry for the mistake). If I understand you well, this check is not useful because PutHDFS and FetchFile processors are already able to catch corruption errors reliably? I would compare both hashes with NiFi expression language (:equals function) inside a RouteOnAttribute processor.
... View more
07-10-2018
07:40 AM
Hi, I work on a NiFi flow getting data from a FTP server, sending this data to HDFS.
I have to add to this flow the ability of checking data integrity between files fetched on the FTP, and files written on HDFS. To do this, I use the HashContent NiFi processor with MD5 algorithm to compute MD5 hash of flowfiles from start and end of the flow (I can get MD5 hash of each file on FTP, second MD5 is computed after a PutHDFS retrieving files having been written).
Finally, I compare both values and consider data integrity is OK if they are equal. Do you have a general advice about this practice?
Is this kind of check really useful with NiFi? Thanks, Benjamin
... View more
Labels:
- Labels:
-
Apache NiFi
05-04-2018
09:58 AM
I have a NiFi job starting with a GetFile processor configured to run on primary node to avoid duplicate flowfiles. After a NiFi node restart, causing a Primary node re-election, the GetFile processor has created two flowfiles (one expected and one duplicate). I suppose that the GetFile was configured to run with the pre-election primary node and post-election one too, causing those two flowfiles (both flowfiles were processed by different nodes according to NiFi data provenance). Is there a way to avoid this behavior, and is it a NiFi bug ? NB: While the job was running, coordinator and primary nodes were the same. Thanks a lot.
... View more
Labels:
- Labels:
-
Apache NiFi
04-09-2018
03:41 PM
1 Kudo
Hi, I need to perform site-to-site transmission within a NiFi dataflow with large files (over 2GB). The transmission does not work, I have logs like "Awaiting transferDataLatch has been timeout" I have tried to split the file using SegmentContent processor from source side (fragments are well sent to target cluster), and then I use MergeContent processor from target side to build the original file, but merge never starts because all fragments are not processed by the same node. Does someone have a solution to send large files via site-to-site, or an idea to make the segment/merge workaround working?
... View more
Labels:
- Labels:
-
Apache NiFi