Member since
01-27-2023
229
Posts
73
Kudos Received
45
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
210 | 02-23-2024 01:14 AM | |
275 | 01-26-2024 01:31 AM | |
234 | 11-22-2023 12:28 AM | |
507 | 11-22-2023 12:10 AM | |
672 | 11-06-2023 12:44 AM |
07-03-2023
07:27 AM
@devanand007, I recommend you to provide a better description of your flow, if you would like to receive an answer to your question. It would really help to know the NiFi Version and your Flow's logic. From where are you getting the data? How does it look before reaching ParseEvtx? What properties you have defined in ParseEvtx? And so on.
... View more
06-26-2023
12:02 AM
@Carson, Like @joseomjr wrote (but did not gave the entire link by mistake), you should take a look at the following Article as it describes exactly what you need --> https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-3/ta-p/249148 Basically, you add the property in your NiFi Processor and reference it to your parameter value and afterwards you call it in your script: myValue1 = myProperty1.getValue()
... View more
06-25-2023
11:53 PM
@Wpq, I do not think that NiFi supports the full syntax of Java regular expressions directly, like you are trying to do, especially on attributes and using NiFi's Expression Language. What I would recommend you to try is to use NiFi's EL to replace that entire regex, something like: ${ip:startsWith('10.') or ip:startsWith('127.') or ip:startsWith('169.254.') or (ip:startsWith('172.') and ip:substring(4, 6):matches('1[6-9]|2[0-9]|3[0-1]')) or ip:startsWith('192.168.')} The above code is not 100% correct, I am more than certain, but you can extrapolate from that example and rewrite your query somehow like that. The NiFi EL functions you should use are: OR: ${ fileSize:lt(64):or( ${fileSize:gt(128)} )} StartsWith: ${ filename:startsWith('fizz') } Matches: ${ filename:matches('fizz.*txt') } And: ${ fileSize:gt(64):and( ${fileSize:lt(128)} )}
... View more
06-21-2023
07:46 AM
@drewski7, in this case, have a look at @steven-matison 's answer because that is your solution to your problem.
... View more
06-21-2023
05:58 AM
@drewski7, UpdateRecord works as fast as you design it to work 🙂 For example, using UpdateRecord, I manage to generate 6 columns on a FlowFile, with more than 200k lines, in less than 7 seconds. For AVRO Files of 100MB, doing pretty much the same action will take around 15-20 seconds. If you are using UpdateRecord to generate 100x columns and each of this columns is using a lookup to check something else or if it uses may functions on multiple columns, it is normal that it will take a long time to process. Besides that, if you are using the UpdateRecord on flowfiles with millions of rows, again, it will take longer to process. So, in order to make your flow faster, you first need to identify where is the bottleneck. First things first, check the type of the file you are reading and the type of the file you are writing into. Each type has its pluses and minuses. Next, I suggest you to take a look on the number of rows in each flowfile --> processing 1M rows is slower than processing 500k rows. Afterwards, you should further check the functions you are applying in UpdateRecord and see if you can optimize them in any way.
... View more
06-20-2023
09:32 AM
sorry but I am not understanding what you are trying to do. First of all, GenerateFlowFile does not accept incoming connections, meaning that you cannot use it in your flow, especially if you are dependent by some other actions. What are you trying to do with GenerateFlowFile exactly? We are trying make it as one flow file, with what ever we loaded on the per/daily load. So you are executing your flow once per day at a specific hour, or? How do you know what should be added in your flow file? What exactly are you trying to achieve? Are you extracting something from your database? Unfortunately you have described your use case very vaguely. If you do require assistance, I strongly recommend you to provide a more detailed description of what you are doing, what you are trying to achieve, what you tried and why it failed. If coming back to your original post, once you saved your data in your database using PutDatabaseRecord, you can further go with a success queue in your next processor and do whatever you need. In this way, once the data is saved in your database, you can call your stored procedure as expected.
... View more
06-20-2023
09:23 AM
@Samya, You have an timeout error, meaning that your action took longer than expected to finish and hence the timeout. I suggest you to set the processor on DEBUG and see what gets written in your logs. It could be that what you have set within your ADLSCredentialsControllerService (where you defined your identity) does not have all the required roles to perform this action 🙂 It could also be that you are trying to maybe create a folder with some strange characters and ADLS does not allow you to do that. Nevertheless, to further debug your situation, set your log level for the processor on DEBUG and see what it writes as output in nifi-app.logs. You can start your analysis from there.
... View more
06-20-2023
12:16 AM
@cotopaul, tagging myself because I am struggling with a similar issue and was not quite able to figure it out myself ... and maybe I will get some hints from some of the answers.
... View more
06-19-2023
11:44 PM
@eykf, I never tried to send messages from NiFi to MS Teams but upon reading the documentation, I am not quite sure you need a Webhook URL at all. MS Teams provides the Graph API to be able to integrate MS Teams into external application: https://learn.microsoft.com/en-us/graph/api/resources/teams-api-overview?view=graph-rest-1.0 Now, assuming that you have the channel and the users already created, my first guess is that you would need only to send the message so basically all you have to do is create the API POST as described in the following link: https://learn.microsoft.com/en-us/graph/api/chatmessage-post?view=graph-rest-1.0&tabs=http And here are the parameters in case you need a more complex message, than the one described in the above link: https://learn.microsoft.com/en-us/graph/api/resources/chatmessage?view=graph-rest-1.0
... View more
06-19-2023
11:35 PM
@Fanxxx, First of all, make sure that nothing is running on the port 2182, as this is the port used by the embedded Zookeeper by default. It could be that you changed it, but my initial feeling is that you left it as it is. Secondly, make sure that you wrote the correct zookeeper connection string within state-management.xml and within each nifi.properties.
... View more