Member since
04-29-2024
14
Posts
2
Kudos Received
0
Solutions
11-30-2024
11:42 AM
1 Kudo
Hi @Vikas-Nifi , I think can avoid a lot of overhead such as writing the data to the DB for just doing the transformation and assigning the fixed width (unless you need to store the data in the DB). You can use processors like QueryRecord, UpdateRecord to do the needed transformation of data in bulk vs one record at a time and one field at a time. In QueryRecord you can use SQL like function based on apache calcite sql syntax to make transformation or derive new columns just as if you are doing mysql query. UpadateRecord also you can use Nifi Record Path to traverse fields and apply functions in bulk vs one record at a time. There is also a FreeFormTextRecordSetWriter service that you can use to create custom format as an output. For example in the following dataflow, Im using ConvertRecord process with CSVReader and FreeFormTextRecordSetWriter to produce desired out: The GenerateFlowFile processor is used to create the input CSV in flowfile: The ConvertRecord is configured as follows: The CSVReader you can use default configuration. The FreeFormTextRecordSetWriter is configured as follows: In the Text Property you can use the columns\fields names as listed in the input and provided to the reader . You can also use Nifi Expression Language to do proper formatting and transformation to the written data as follows: ${DATE:replace('-',''):append(${CARD_TYPE}):padRight(28,' ')}${CUST_NAME:padRight(20,' ')}${PAYMENT_AMOUNT:padRight(10,' ')}${PAYMENT_TYPE:padRight(10,' ')} This will produce the following output: 20241129Visa Test1 0.01 Credit Card
20241129Master Test2 10.0 Credit Card
20241129American Express Test3 500.0 Credit Card I know this not 100% what you need but it should give you an idea what you need to do to get the desired output. Hope that helps and if it does, please accept the solution. Let me know if you have any other questions. Thanks
... View more
09-26-2024
05:22 AM
1 Kudo
@Vikas-Nifi Your dataflow is working as designed. You have your listFile producing three FlowFiles (1 for each file listed). Each of those FlowFiles the trigger the execution of your FetchFile which you have configured to fetch the content of only one those files. If you only want to fetch "test_1.txt", you need to either configure the listFile to only list file "test_1.txt" or you need to add a RouteOnAttribute processor between your listFile and FetchFile so that you are only routing the listed FlowFile with ${filename:equals{'test_1.txt')} to the FetchFile and auto-terminating the other listed files. The first option of only listing the file you want to fetch the content for is the better option unless there is more to your use case then you have shared. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
08-30-2024
10:21 AM
1 Kudo
Hi, I am using putSlack in NiFi 1.26.0. Where i have configured following: Webhook URL Webhook text Channel SSL Context service which has slack cert. (JKS) Getting below error while running the putSlack in DEV server. "failed to open connection. [java.net.unknownhosexception: hooks.slack.com] Could any one please help to fix this.
... View more
Labels:
- Labels:
-
Apache NiFi
07-02-2024
07:33 AM
@Vikas-Nifi the following error is directly related to failure to establish certificate trust in the TLS exchange between NiFi's putSlack processor and your slack server: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target " The putSlack processor utilizes the StandardRestrictedSSLContextService to define keystore and truststore files the putSlack processor will use. The truststore must contain the complete trustchain for the target slack server's serverAuth certificate. You can use: openssl s_client -connect <companyName.slack.com>:443 -showcerts to get an output of all public certs included with the serverAuth cert. I noticed with my slack endpoint that was not the complete trust chain (root CA certificate for ISRG Root X1 was missing from the chain). You can download the missing rootCA public cert directly from let's encrypt and add it to the truststore set in the StandardRestrictedSSLContextService. https://letsencrypt.org/certificates/ https://letsencrypt.org/certs/isrgrootx1.pem https://letsencrypt.org/certs/isrg-root-x2.pem You might also want to make sure all intermediate CAs are also added and not just the intermediate returned by the openssl command just in case server changes that you get directed to. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
05-30-2024
06:17 AM
@Vikas-Nifi @ckumar is 100% correct. Only fields explicitly marked as supporting NiFi Expression Language (NEL) can support a NEL expression "${schedule}". I am however curious about your use case as to why you would even being trying to do this. From what you shared you are extracting a cron schedule from the json content of some FlowFiles traversing an EvaluateJsonPath processor. That "schedule" is added on to the NiFi FlowFile as a FlowFile attribute (key=value pair). This would not make that key=value pair accessible to any other NiFi component unless that FlowFile containing the FlowFile attribute was processed by that other component. However, in your shared dataflow you do not mention that EvaluateJsonPath connects to your invokeHTTP processor via an inbound dataflow connection (Keep in mind that even if you did do this, it does not change the fact that the run schedule property does not support NEL). I just wanted to clarify how FlowFile attributes are and can be used. Also keep in mind that the "run schedule" is a scheduler only. The run schedule set on a processor controls when the NiFi controller will schedule the execution of the processors code. It does not mean that they the processor will immediately execute at time of scheduling (It may be delayed on execution waiting for an available execution thread from the thread pool). All scheduled components share a thread pool and NiFi framework will also handle assigning threads to next scheduled component as thread become available. So the NiFi framework needs to know the scheduling for a component when it is started; otherwise, NiFi would never know when to schedule it to execute. Unless a component property has an explicit tool tip that tells you it support NEL, then it does not. For NiFi processor components, you will find that only some processor specific properties within the "PROPERTIES" tab support NEL. This is not only available through property tooltips, but also in the processors documentation. Examples: Even when NEL is supported there is a scope. It may support FlowFile attributes, Variable Registry (going away in NiFi 2.x releases), or both. Thank you, Matt
... View more