I want to execute a spark job using NiFi. At the moment NiFi does not support capturing provenance information outside NiFi. Is it possible to add additional information to the existing provenance information for example if I have a workflow that ends at a Spark job, can I add additional provenance information to the provenance information that was attached by NiFi.
I do know for cross component provenance/Lineage Atlas provides some support. At the moment it does not support Spark. I want to add some additional information regarding the job start time and end time etc while executing the Spark Job. Later I want to send back the results of the spark job to NiFi.
would it be possible to add to the existing Nifi provenance information so that when the data is ingested back in I know what happened in Spark.