Member since
12-05-2017
2
Posts
0
Kudos Received
0
Solutions
07-18-2018
10:41 AM
XML is a complex format as it is a tree structure, but with a XSD it is possible to safely convert the XML file. But XSD files also supports various formats with makes it a bit more complicated. Any how I wrote a few articles about this topic a few weeks ago and also published source code for doing the conversion. Here is a few direct links that might be useful for you. XML2CSV git repo nifi-xml XML2CSV article XML to tables (xml2csv) with NiFi and Groovy Part 2 of 2 If you also is interested in avro forma have a look at this XML2Avro git repo xml2avro XML2Avro article XML2Avro NiFi processor
... View more
12-06-2017
02:44 PM
It sounds like you are trying to convert a job-based batch use case into a real-time streaming use case. You will have to re-think what your requirements and goals are for your process, as NiFi doesn't operate in jobs but in streaming data that is constantly coming in. In NiFi, you may decide to implement your workflow the following way. Set your processor to monitor an HDFS directory for new files and load them into "Y DB" whenever new files are found with a scheduling interval of 1 minute (or 5, or 10). You would use the ListFiles processor to monitor for new files in a directory. That would accomplish your first 2 jobs. For the third job, you could use the QueryDatabaseTable to scan for changes every so often (say, 1 minute) and load those changes into "Z DB". This would move your processing from a batch-oriented nature to a real-time streaming workflow.
... View more