@Kok Ching Hoo Even you don't need to use jolt transform processor to get only the entry array as the flow file content. We can achieve the same result by using split json processor in more easy way. Configure split json processor as JsonPath Expression $.*.*.*.* By using above json path expression it doesn't matter even if header value changed,array entry has been changed to entry1,exit ... etc until you are having same structure of the json message(same dependency will be applicable by using jolt also), this method will work we are going to split the array it self and then use the splits relation to connect to the next processors. If you want to do dynamically without any dependencies on the attribute names that are going to be defined in the incoming json message/object then go with this approach. ... View more

@Kok Ching Hoo For Method1:- in split json processor Use JsonPath Expression like $.['XMLFile_2234.DAILY'].dataset_12232.entry now we are escaping period in XMLFile_2234.DAILY. For method2:- Increase the below properties values in you extract text processor as per your flowfile size and capture group length. Maximum Buffer Size 1 MB Maximum Capture Group Length 1024 ... View more

@Sai Krishna Makineni Option1:- In hbase as we can have lookup joins by denormalizing all the table while storing into Hbase. Let's take you are having employee table as employeeid 1, name hcc, salary 1000, departmentid 100 and we need to have department name needs to be added to the employee record. so we need to have departments table will having departmentid as rowkey let's consider we are having departmentid 100 departmentname kb, Once we got employee record in avro format then use convertAvroToJson processor, we can extract all the json values and keep as attributes and by using fetchHbase processor we can look for departmentid and extract the departmentname keep it as attribute. then by using AttributesToJson processor we can recreate the final record as follows {"employeeid" :"1", "name": "hcc", "salary" :"1000","departmentid": "100","departmentname" :"kb"} For more reference:- http://hbase.apache.org/0.94/book/joins.html Option2:- Phoenix:- You can create phoenix table will uses indexing on top of Hbase table,queries will be performed way faster compared to hive as you can compare performance between Hive-Hbase vs Phoenix-Hbase. https://phoenix.apache.org/performance.html https://stackoverflow.com/questions/29405629/apache-phoenix-vs-hive-spark https://www.quora.com/How-does-Apache-Phoenix-reduce-latency-over-Hive https://stackoverflow.com/questions/28360888/apache-phoenix-vs-hbase-native-api Option3:- Spark On Hbase:- Even we can load Hbase tables directly by not creating any phoenix (or) hive tables on top into spark by using Spark on Hbase connector. By using this connector you can load all the required tables into spark then perform all joins in spark. Please refer below link for more details https://hortonworks.com/blog/spark-hbase-dataframe-based-hbase-connector/ ... View more

@Pavan M As you are not transferring any of the flowfiles to REL_FAILURE, Transfer the else flowfiles to failure relation and auto terminate the failure relation, else: session.transfer(flowFile, REL_FAILURE) Auto terminate failure relation (or) You can use session.remove to remove the flowfile else: session.remove(flowFile) by using any of the above ways you can achieve same result as you are expecting. ... View more

@Kok Ching Hoo if you want to split the json content on entry array then you don't have to use regex at all. Method1:- Use Split Json processor with below configs:- JsonPath Expression $.XMLfile_2234.dataset_12232.entry Then use split relation from splitjson processor to connect to the next processor. Input Json content:- input flowfile content that feeding into split json processor. {"XMLfile_2234": {"xsi:schemaLocation": "http://xml.mscibarra.com/random.xsd","dataset_12232": {"entry": [{"record_date": "2017-03-01","country": "USA","funds": "100"},{"record_date": "2018-03-01","country": "Chile","funds": "10000"}]}}} Output from Splitjson processor:- flowfile1:- {"record_date":"2017-03-01","country":"USA","funds":"100"} flowfile2:- {"record_date":"2018-03-01","country":"Chile","funds":"10000"} (or) Method2:- you can use Extract text processor to extract the entry array and keep that as attribute then use ReplaceText processor to overwrite the existing content of the flowfile with new array attribute value, then use splitjson processor to split the array. Extract text configs:- Add new property to the extract text processor by clicking + sign at top right corner and then add the below property array "entry": (.*]) Now we are going to extract all the entry array message and keep that into array attribute to the flowfile. Then use Replacetext processor to replace the contents of flowfile with array message value. Replacetext configs:- Replacement Value ${array} Replacement Strategy Always Replace Change the above property values in the replace text processor, in this processor we are writing array attribute value as contents of the flowfile. now we are going to have entry array message as our flowfile content so we can use split json processor to split the array into individual messages. Input flowfile content:- {"XMLfile_2234": {"xsi:schemaLocation": "http://xml.mscibarra.com/random.xsd","dataset_12232": {"entry": [{"record_date": "2017-03-01","country": "USA","funds": "100"},{"record_date": "2018-03-01","country": "Chile","funds": "10000"}]}}} output flowfile content:- [{"record_date": "2017-03-01","country": "USA","funds": "100"},{"record_date": "2018-03-01","country": "Chile","funds": "10000"}] as you can notice the output flowfile content has been changed in this processor. SplitJson processor:- JsonPath Expression $.* Flow:- 1.Extract text processor 2.Replace Text processor 3.SplitJson processor By following both methods output would be the same by using method 1 would be easy to complete this task. ... View more

@Pavan M Could you please see my edited answer. let me know if you still having issues .. ... View more

@hema moger Yes, you are right. Once you successfully fetch any file from sftp server then data provenance will be updated accordingly. ... View more