Member since
10-09-2017
6
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1338 | 10-09-2017 10:57 AM |
02-02-2018
03:31 PM
Hi @Abdelkrim Hadjidj, thanks for your answer. I've begun looking onto your suggestions and I think we can probably build something suitable using a combination of the features you mentioned. Next week I’ll try experimenting with this. Thanks again. Dave.
... View more
02-01-2018
02:17 PM
The system we are building will be multi-tenant and involve several NiFi Flows. To make this manageable we want the ability to define a single NiFi flow as a parameterized template, of which multiple instances can be started and stopped from the command-line with varying parameters. One example flow might be a file pump: It consumes files of records from a watched folder, converts the contents to a matching Avro schema, writes to a Kafka topic for that record type. Having multiple record types, we need several instances of this flow running with the only differences being source folder, Avro schema name & Kafka topic name, all of which are simple parameters. Similarly, the same NiFi flows will be used across the many tenants in the system, but started under different credentials to ensure there’s no possibility of ‘data crossing’ between tenants. Parameters might include the tenant ID, possibly derived from the credentials. Requires ability to define a NiFi template containing parameters start any number of instances of that template with defined parameters from command-line monitor running of the resulting flows. Note: Such flows should be read-only in the UI as it's too easy to mistakenly edit and break them. stop a running instance from command-line I can’t find anything in NiFi like these capabilities. Yes, there are templates and process groups, but manually building these via the UI is arduous and prone to error, and there’s no obvious way to generalize a flow (analogous to a class) and create it multiple times with different parameters (instance of a class). The nearest I saw was this, from Andrew Grande of Hortonworks: https://github.com/aperepel/nifi-api-deploy But it looks unofficial and not maintained (“Note: this was originally created for NiFi 0.x. REST APIs and concepts have changed significantly in NiFi 1.x”) Is there anything more suitable or official out there? Or will we need to build a tool akin to (or starting from) Andrew’s, to drive the REST API? Thanks, Dave.
... View more
Labels:
- Labels:
-
Apache NiFi
10-16-2017
10:36 AM
I agree with Bjorn Olsen and Timothy Spann, ability to edit or remove mistaken schemas would be valuable. I've already messed up my sandbox by getting the forward/backward compatibility wrong on one schema, and marking others with group 'Test' and 'test' when I wanted them in the same group for example.
... View more
10-09-2017
10:57 AM
1 Kudo
Hi Beejal, if you're using a local sandbox, have you tried 127.0.0.1 instead of sandbox-hdf.hortonworks.com? Also, my sandbox VM is hosting Ambari on port 8080 so the address becomes http://127.0.0.1:8080.
... View more