Member since
08-21-2024
7
Posts
4
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
475 | 09-20-2024 07:17 AM |
09-20-2024
07:49 AM
1 Kudo
Hey @SAMSAL Amazing response - thank you! I'll go over a couple of things you have mentioned, firstly the overwriting... I was wrong about that one. When I converted it to CSV instead using the 2 sheets, it actually appends the information from sheet 2 to the end of sheet 1 (leaving 2 rows as a gap first) so in your example above you would actually end up with something like this in CSV (when reimported into XL): I'm not sure if I can work with that, as I don't see any way in any of the readers to be able to "split" that (or only reading from row 2 to X for example) whilst keeping the flow etc and just using some of the values The other issue is with the Avro you have set up, because we don't know what the column headers will be (field names) I'm unable to define a schema in Avro for this as they can add and remove these columns at will - the only thing I will be able to rely on is that both the sheets will have identical columns. When I convert direct to JSON, I get something more like this: [ {
"column_0" : "ID",
"column_1" : "NAME",
"column_2" : "ADDRESS"
}, {
"column_0" : "1",
"column_1" : "Sam",
"column_2" : "TX"
}, {
"column_0" : "2",
"column_1" : "Ali",
"column_2" : "FL"
}] Again, not sure this is something I can use in this format but I'm just not familiar enough with the NiFi elements at this stage I guess I'll try the enrichment fork method though and see if there is some mileage in that and then I can feed back
... View more
09-20-2024
07:17 AM
1 Kudo
Hi @MattWho Thanks for the response, much appreciated - what I was looking at doing was simply moving the Registry from one of our servers which we had set up previously into another server which we were using for production - so not keeping 2 registries but instead using the new one and getting rid of the old one. What I didn't want to do was lose everything from the old one... However, this turned out to be WAY easier than I thought lol Basically we were deploying through Ansible and I had missed some configuration values and files when I was deploying which meant it was actually set to use files instead of a git repo for the storage system. Once I found this issue, updated it to the git repo and presto! all worked as expected with everything available in the new registry server from the git repository and all set up. Thanks for the information though that's also very helpful!
... View more
08-22-2024
12:46 AM
Hey @SAMSAL - thanks for the detailed reply, and yep you hit the nail on the head ha ha. I've used a few of those links already, and as you said I was close, but not quite there. I grasp the concept, it's just the syntax of everything that was throwing me off. That being said, the JSLT example you posted does seem to make more sense to me also, so that's definitely something I'll look into. I spoke to a guy called Paul Lakus in the NiFi slack channel and he basically gave me almost exactly the same solution that you have posted and so I managed to work out how it was being done, but still a huge learning curve for me heh. Anyway this now seems to be working as intended, which I'm extremely grateful for so thank you for your input and the steer towards JSLT - definitely something to look into. Thanks!!
... View more