Member since
05-30-2023
38
Posts
3
Kudos Received
0
Solutions
04-02-2024
11:31 AM
2 Kudos
If you meant Storing Apache NiFi Versioned Flows in a Git Repository hosted on Azure DevOps, then the steps should be similar to the steps you followed to connect to Github. The appropriate flowPersistenceProvider is GitFlowPersistenceProvider. You can follow the steps as mentioned here: Storing Apache NiFi Versioned Flows in a Git Repos... - Cloudera Community - 248713 Or the video: https://youtu.be/kK7eVppg9Aw Only changes you would have to do is to generate PAT in Azure DevOps with permissions on code read, write, modify. P.S. The solution is only available for Git under Azure DevOps, and not for TFS.
... View more
03-26-2024
09:07 PM
2 Kudos
Hi @Kiranq , This a basic case of data enrichment and there are multiple ways how you can handle this. If for example instead of the function you have a table with a unique id then you can use LookupRecord with DatabaseRecordLookupService or even SimpleDatabaseLookupService . All you have to do here is basically specify the path where you want the new value to be inserted as dynamic property. However since you have a function instead of simple table\view , then the second option is to use ForkEnrich\JoinEnrich processors . The nice thing about those processors is that you have multiple strategies how you can join the data together as you can read here: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apache.nifi.processors.standard.JoinEnrichment/additionalDetails.html Let me know which one works for you better and let me know if you face any issues regarding using either. If you find this is helpful please accept solution Thanks
... View more
02-02-2024
01:32 PM
1 Kudo
@Kiranq all components (processors, controller services, reporting tasks, etc) execute as teh user that owns the running NiFi service. So that NiFi service user would need to be able to execute the local system command. The user who authenticates themselves to access the NiFi UI is not the user used to run the components. Have you tried as the NiFi service user to execute your python code from command line? If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
10-19-2023
04:50 AM
I resolved the issue. I had DepartmentId in my avro schema however, I was getting DepartmentID in the input. Schemas are case sensitive. Now, that the string values are appended in the DepartmentID column, I am using RouteOnContent processors and using ${DepartmentID:equals('Undefined')} to match if the DepartmentID' values are of undefined or null type. It is not identifying the "Undefined" values. is the regex statement correct? Content of flow file is: [ { "TenantId" : "FF369226-0EDA-4D0D-B48C-6B27F404ECCB", "DepartmentID" : "Undefined", "Name": "xyz" } ]
... View more
10-03-2023
08:12 AM
@Kiranq, What I found is that the JoltTransformRecord expects only single record to work with hence the name. I noticed when I try to pass an array I was getting the error "...error transforming the first record", however if I pass just one json record it works. If you have an array of json\csv and you are looking to split and process each record individually then I would suggest that you split the records before the JoltTransformRecord. If you dont want to split the array then I recommend using JoltTranformJson first and then use Convert Record processor to convert to CSV.
... View more
09-22-2023
08:56 AM
@Kiranq The NiFi rest-api endpoint for obtaining token is not rest-api/access. It is rest-api/access/token https://<nifi-hostname>:<nifi-port>/nifi-api/access/token and data passed like: --data-raw 'username=<username>&password=<user password>' Then the token needs to be passed with every subsequent rest-api request. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-16-2023
01:15 PM
As far as I know, NiFi leverage Jython and would currently limit you to using Python 2.7 compatible code and only modules written in pure Python. Home | Jython
... View more
09-06-2023
08:04 AM
@Kiranq Why are you using ExecuteScript? You can setup a DBCP (DataBaseConnectionPool) controller service with your sql connection and driver file. Make sure that jdbc driver is found on all nifi hosts. Then, you are able to use any processors that reference a DBCP Controller Service. For example: ExecuteSql.
... View more
08-31-2023
09:12 AM
If you are getting multiple records in one Json array , then you probably need to use SplitJosn Processor to get each record individually , then extract the values you need using EvaluateJsonPath from each record then do the PutSQL.
... View more
08-29-2023
08:39 AM
Can you help me with a similar issue? My code: import json import collections import java.io import ast from org.apache.commons.io import IOUtils from java.nio.charset import StandardCharsets from org.apache.nifi.processor.io import StreamCallback class ModJSON(StreamCallback) : def __init__(self): pass def process(self, inputStream, outputStream): text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) obj = ast.literal_eval(text) mail = obj['mail'] outputStream.write(bytearray(json.dumps(obj['mail'], indent=4).encode('utf-8'))) outputStream.write(bytearray(json.dumps(obj['id'], indent=4).encode('utf-8'))) flowFile = session.get() if (flowFile != None): flowFile = session.write(flowFile, ModJSON()) flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename').split('.')[0]+'_translated.json') flowFile = session.putAttribute(flowFile, "mail", mail) session.transfer(flowFile, REL_SUCCESS) session.commit() I want to get "mail" value and create a new attribute for it. However, mail variable is not accessible when I insert it into the putattribute command. Any feedback?
... View more