Member since
07-21-2020
14
Posts
5
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
58 | 02-08-2021 11:44 PM |
02-10-2021
08:29 AM
1 Kudo
Hello, I try to work out some automated deployment situation in NiFi (no UI usage, just cli toolkit or alternatives). Let us asume, you have a nifi flow under a load which need to be updated. There are some data in in-between connection queues. What happens when you try to update the flow? Will some data be lost or will the update be refused? Are there any solution for procedure like this. 1) stop the data receiving processor 2) check if the success connection queue is empty 3) apply steps 1 and 2 to the next processor till the last processor is stopped. 4) update the flow ignoring any data in failure or error connection queues Do you think it is reasonable approach? Is there something ready to use? Best regards Jaro
... View more
Labels:
02-08-2021
11:44 PM
1 Kudo
OK,, I got it. I missed a parameter. It should be nifi set-param -u http://localhost:8080 -pcid 828f0882-0177-1000-aabd-d680a070c5c5 -pn sample.URL -pv newvalue
... View more
02-08-2021
12:27 PM
Hello, I have tried to use nifi cli toolkit (nifi 1.11.4) to set parameter like this. However, I can't see any effect. What have I doing wrong? #> nifi get-param-context -u http://localhost:8080 -pcid 828f0882-0177-1000-aabd-d680a070c5c5 # Name Value Sensitive Description - ------------- ------------- --------- ----------- 1 sample.Passwd ******** true 2 sample.Schema SAMPLE001 false 3 sample.URL jdbc://sample false 4 sample.User dbsample false #> nifi set-param -u http://localhost:8080 -pcid 828f0882-0177-1000-aabd-d680a070c5c5 -pn sample.URL newvalue Waiting for update request to complete... #> nifi get-param-context -u http://localhost:8080 -pcid 828f0882-0177-1000-aabd-d680a070c5c5 # Name Value Sensitive Description - ------------- ------------- --------- ----------- 1 sample.Passwd ******** true 2 sample.Schema SAMPLE001 false 3 sample.URL jdbc://sample false 4 sample.User dbsample false Best regards Jaro Jaro
... View more
Labels:
02-07-2021
11:55 AM
Hello, I have a question regarding the concept of parameter context. The admin guide says : A Process Group can only be assigned one Parameter Context, while a given Parameter Context can be assigned to multiple Process Groups. OK, let's assume you have internal kafka or database, and some external system you connect to. You might want to have a parameter "group" for the internal kafka connection, and another parameter contexts for each external system connection. However, your flow, clearly a part a process group, reads data from external system and stores data internally. Thus you need actually 2 connections in the flow, it means 2 parameter contexts, which is not possible. To solve that, you assign the connections (controller services) to the parent group and the parameters. Finally, for connections which are common to many flows, like your kafka, databases, and HttpSSL you land in the parent NiFi Flow, having parameter context which contains a bunch of pretty unrelated parameters (for kafka, databases, ..). What is see, there is hardly a possibility to reuse parameter context as they cannot be combined and thus you have 1:1 relationship between par. context and group just like for variables assigned to a processor group. Did I miss something? What is actually the big advantage of parameters over variables? Best regards Jaro
... View more
Labels:
12-04-2020
05:09 AM
Hello, Does nipyapi package works with windows 10 (64bit)? I have tried to install nipyapi on two machines, one with winpython (manual installation of downloaded packages) another with anaconda (installation with pip). Both failed, obviously due a number of missing obscure dependencies like libxml2. I have only a little knowledge, but the installation looked like a tough stuff under windows 10. Is nipyapi recommended at all or are there better alternatives? Best regards Jaro
... View more
Labels:
09-24-2020
12:59 PM
Hi, @MattWho we actually already implemented prototypes of such custom processors and the basic principles are clear. By processor logic, I did not mean the structure of the processor like relationships. The structure is static in our case. By the processor logic, I refer, the code which is called in "onTrigger" in "StreamCallback.process". In my case, the logic can be parametrized by some configuration data (These configuration data is provided by a service or database tables). The configuration data is static during the whole processor run and must be provided during instantiation of the business logic object. You might see it as lookup, which must not happen during the processing "onTrigger", but "onScheduled" (sorry I wrote in "init" previously, that was not right)
Best regards Jaro
... View more
- Tags:
- apache nifi
09-24-2020
03:58 AM
Hello, we consider to implement a custom processor for some complex transformation logic on kafka streams. The flow is straightforward : input topic -> our processor -> output topics The processor logic needs to be initialized by data provided through a rest service and/or database records. What is the NiFi way to implement the initialization? As we implement the processor, we can do lots of stuff in e.g. init method, however, we would prefer to use NiFi infrastructure reasonably. Could you please recommend some examples, blogs etc. for similar solution. Best regards Jaro
... View more
Labels:
09-15-2020
09:51 AM
1 Kudo
Actually, both replies can be considered as valid. I confirmed that one, which better fits to my use case.
... View more
09-08-2020
12:26 AM
1 Kudo
It sounds like your testing solution is exceeding the inbound capabilities of the flow tuning (nifi config, processor/queue config) Correct assessment. It has showed that the pipeline was not properly sized for the amount of data, which lead to a back-pressure in the ingest component
... View more
08-28-2020
08:12 AM
Well, EvaluateJsonPath sounds promising but it looks like that EvaluateJsonPath is somehow limitated. It is unable to extract json objects, only single attributes, which makes it impossible to use for slightly more complex data. Let's suppose this. Input: { whatever known or unknown json nested structures here } Desired output { "nifi-filename": "2c8a7845-8567-4eed-97ee-4dcd1c18668e", "nifi-content": { whatever known or unknown json possibly structures or sub-structures of the input message here } }
... View more
- Tags:
- apache nifi
08-27-2020
02:13 AM
Hello, I am puzzelling for some time if there is possibility to refer the both attributes and content of flow file and transform the content by combining both. Example: Assume a flow file with uuid=2c8a7845-8567-4eed-97ee-4dcd1c18668e and the json content { "data":"some business data" } The result of the transformation should be a content like { "nifi-filename": "2c8a7845-8567-4eed-97ee-4dcd1c18668e", "nifi-content": { "data": "some business data" } } How can I reach that.? Best regards Jaro
... View more
Labels:
08-14-2020
04:51 AM
Hello, I tested ListenHTTP on a single node under some load generated by jmeter. I observed that for the one or two client threads the ListenHTTP works fine, however, having 10 client threads leads to 60% request refused by 503,Service Unavailable. Is there any parameter, which helps to come accross the issue? Also the component does not indicate any problems in the GUI. Best regards Jaro
... View more
- Tags:
- listenhttp
- NiFi
Labels:
07-21-2020
04:08 AM
1 Kudo
Hi, being pretty new to Nifi, I am strugeling to extract interface monitoring data Assuming a very simple ingest flow like: HttpListener -> KafkaPublisher How can I get throughput information, i.e. number of records put (e.g.) every minute to a topic? There is NifiSummary -> Processors -> Status History, which might be useful. Can the statistics be accessed programatically and how? Best regards Jaro
... View more
Labels: