Member since
04-29-2024
14
Posts
2
Kudos Received
0
Solutions
11-29-2024
05:26 AM
Below is the thing that I am trying for: Input file sample: (Which will have 40k+ records, with a header and data segments, with "|" separated columns.) DATE|CUST_ID|CUST_NAME|PAYMENT_AMOUNT|PAYMENT_TYPE|CARD_TYPE 2024-11-29|123|Test1|0.0100|Credit Card|Visa 2024-11-29|456|Test2|10.00|Credit Card|Master 2024-11-29|789|Test3|500.00|Credit Card|American Express Expected output: This should be in a file with column having fixed length and sent to the remote server using the 'putSFTP' processor. 20241129Visa Test1 0.01 Credit 20241129Master Test2 10.00 Credit 20241129American Express Test3 500.00 Credit I am storing the input data in a database table using the 'putDatabaseRecord' processor and retrieving it using the 'executeSQL' processor (using SQL query adding fixed length for columns) convertAvroToJSOn output: [ { "date": "20241129", "card_type": "Master ", "payment_amount": "10.00 ", "payment_type": "Credit ", "cust_name": "Test1 " }, { "date": "20241129", "card_type": "Visa ", "payment_amount": "0.01 ", "payment_type": "Credit ", "cust_name": "Test2 " } ] SplitJSON to split the JSON and evaluateJSON path to get each attribute value Update attribute to place value in require output pattern Merge al the content and put it in a file.
... View more
11-29-2024
05:19 AM
@SAMSAL Below is the thing that I am trying for: Input file sample: (Which will have 40k+ records, with a header and data segments, with "|" separated columns.) DATE|CUST_ID|CUST_NAME|PAYMENT_AMOUNT|PAYMENT_TYPE|CARD_TYPE 2024-11-29|123|Test1|0.0100|Credit Card|Visa 2024-11-29|456|Test2|10.00|Credit Card|Master 2024-11-29|789|Test3|500.00|Credit Card|American Express ********************************************************************************** Expected output: This should be in a file with column having fixed length and sent to the remote server using the 'putSFTP' processor. 20241129Visa Test1 0.01 Credit 20241129Master Test2 10.00 Credit 20241129American Express Test3 500.00 Credit ************************************************************************************* I am storing the input data in a database table using the 'putDatabaseRecord' processor and retrieving it using the 'executeSQL' processor (using SQL query adding fixed length for columns) convertAvroToJSOn output: [ { "date": "20241129", "card_type": "Master ", "payment_amount": "10.00 ", "payment_type": "Credit ", "cust_name": "Test1 " }, { "date": "20241129", "card_type": "Visa ", "payment_amount": "0.01 ", "payment_type": "Credit ", "cust_name": "Test2 " } ] SplitJSON to split the JSON and evaluateJSON path to get each attribute value Update attribute to place value in require output pattern Merge all the content and put it in a file.
... View more
11-29-2024
05:15 AM
@SAMSAL Below is the thing that I am trying for: Input file sample: (Which will have 40k+ records, with a header and data segments, with "|" separated columns.) DATE|CUST_ID|CUST_NAME|PAYMENT_AMOUNT|PAYMENT_TYPE|CARD_TYPE 2024-11-29|123|Test1|0.0100|Credit Card|Visa 2024-11-29|456|Test2|10.00|Credit Card|Master 2024-11-29|789|Test3|500.00|Credit Card|American Express ****************************************************************************** Expected output: This should be in output file with column having fixed length and sent to the remote server using the 'putSFTP' processor. 20241129Visa Test1 0.01 Credit 20241129Master Test2 10.00 Credit 20241129American Express Test3 500.00 Credit *********************************************************************************** I am storing the input data in a database table using the 'putDatabaseRecord' processor and retrieving it using the 'executeSQL' processor (using SQL query adding fixed length for columns) convertAvroToJSOn output: [ { "date": "20241129", "card_type": "Master ", "payment_amount": "10.00 ", "payment_type": "Credit ", "cust_name": "Test1 " }, { "date": "20241129", "card_type": "Visa ", "payment_amount": "0.01 ", "payment_type": "Credit ", "cust_name": "Test2 " } ] SplitJSON to split the JSON and evaluateJSON path to get each attribute value Update attribute to place value in require output pattern Merge al the content and put it in a file.
... View more
11-29-2024
05:14 AM
@SAMSAL Below is the thing that I am trying for: Input file sample: (Which will have 40k+ records, with a header and data segments, with "|" separated columns.) DATE|CUST_ID|CUST_NAME|PAYMENT_AMOUNT|PAYMENT_TYPE|CARD_TYPE 2024-11-29|123|Test1|0.0100|Credit Card|Visa 2024-11-29|456|Test2|10.00|Credit Card|Master 2024-11-29|789|Test3|500.00|Credit Card|American Express ****************************************************************************************** Expected output: This should be in a file with column having fixed length and sent to the remote server using the 'putSFTP' processor. 20241129Visa Test1 0.01 Credit 20241129Master Test2 10.00 Credit 20241129American Express Test3 500.00 Credit ******************************************************************************** I am storing the input data in a database table using the 'putDatabaseRecord' processor and retrieving it using the 'executeSQL' processor (using SQL query adding fixed length for columns) convertAvroToJSOn output: [ { "date": "20241129", "card_type": "Master ", "payment_amount": "10.00 ", "payment_type": "Credit ", "cust_name": "Test1 " }, { "date": "20241129", "card_type": "Visa ", "payment_amount": "0.01 ", "payment_type": "Credit ", "cust_name": "Test2 " } ] SplitJSON to split the JSON and evaluateJSON path to get each attribute value Update attribute to place value in require output pattern Merge al the content and put it in a file.
... View more
11-29-2024
04:58 AM
@SAMSAL : Below is the thing that I am trying for: ------------------------------------------------------------------- Input file sample: (Which will have 40k+ records, with a header and data segments, with "|" separated columns.) DATE|CUST_ID|CUST_NAME|PAYMENT_AMOUNT|PAYMENT_TYPE|CARD_TYPE 2024-11-29|123|Test1|0.0100|Credit Card|Visa 2024-11-29|456|Test2|10.00|Credit Card|Master 2024-11-29|789|Test3|500.00|Credit Card|American Express ------------------------------------------------------------------------- Expected output: This should be in a file with column having fixed length and sent to the remote server using the 'putSFTP' processor. 20241129Visa Test1 0.01 Credit 20241129Master Test2 10.00 Credit 20241129American Express Test3 500.00 Credit -------------------------------------------------------------------------- I am storing the input data in a database table using the 'putDatabaseRecord' processor and retrieving it using the 'executeSQL' processor (using SQL query adding fixed length for columns) convertAvroToJSOn output: [ { "date": "20241129", "card_type": "Master ", "payment_amount": "10.00 ", "payment_type": "Credit ", "cust_name": "Test1 " }, { "date": "20241129", "card_type": "Visa ", "payment_amount": "0.01 ", "payment_type": "Credit ", "cust_name": "Test2 " } ] SplitJSON to split the JSON and evaluateJSON path to get each attribute value Update attribute to place value in require output pattern ( (date)(card_type)(cust_name)(payment_amount)(payment_type) ) Merge all the content into single file and put it in a file.
... View more
11-28-2024
04:49 AM
I have a text file which needs to be fetched from remote server through SFTP. File will be having 26k+ rows with | separated values like below: I have used below processor's to do the data manipulation. But facing “Processing halted.Out of memory heap” error after splitJSON. Please help with the better approach to handle this case.
... View more
Labels:
- Labels:
-
Apache NiFi
09-26-2024
03:22 AM
Hi, I am using NiFi 1.26.0 and having a flow with combination of ListFile and FetchFile. Basically i have three files in my local directory (test_1.txt, test_2.txt,test_3.txt). I have to fetch only one file (ex:test_1.txt) for further process. Based on regex listFile list's all three files. But facing issue in fetch file. Instead of fetching only one file, it tried to fetch all three files listed by listFile processor and giving output as three files instead of one file, but content of all three files are same as content of file mentioned in fetchFile configuration (test_1.txt) In data provenance could see it is trying to fetch the files which is not mentioned in fetchFile configuration and after fetching the files it modifies the content. How can i fix this. Please help.
... View more
Labels:
- Labels:
-
Apache NiFi
08-30-2024
10:21 AM
1 Kudo
Hi, I am using putSlack in NiFi 1.26.0. Where i have configured following: Webhook URL Webhook text Channel SSL Context service which has slack cert. (JKS) Getting below error while running the putSlack in DEV server. "failed to open connection. [java.net.unknownhosexception: hooks.slack.com] Could any one please help to fix this.
... View more
Labels:
- Labels:
-
Apache NiFi
06-26-2024
06:23 AM
Hi, Getting below error while calling Slack channel from Nifi. Used putSlack Processor. "o.apache.nifi.processors.slack.PutSlack : Failed to open connection javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target " I have added slack cert in to my truststore. I am trying to get it tested in local (haven't deployed in to environment server). Added the channel name and webhookURL (which generated under "companyName.slack.com") Is there anything i need to add it in local truststore to work?. Please help @MattWho @SAMSAL @cotopaul
... View more
Labels:
- Labels:
-
Apache NiFi