@aman mittal Yes, it's possible. Take a look into the below sample flow Flow overview: 1.SelectHiveQL //to list tables from specific database in avro format HiveQL Select Query show tables from default //to list all tables from default database 2.ConvertAvroToJson //to convert the list of tables from avro format to json format 3.SplitJson //split each table into individual flowfiles 4.EvaluateJsonPath //extract tab_name value and keep as attribute to the flowfile 5.RemoteProcessorGroup //as you are going to do for 3k tables it's better to use RPG for distributing the work. if you don't want to use RPG then skip both 5,6 processors feed success relationship from 4 to 7. 6.InputPort //get the RPG flowfiles 7.SelectHiveQL //to pull data from the hive tables 8.EncryptContent 9.RouteOnAttribute //as selecthiveql processor writes query.input.tables attribute, so based on this attribute and NiFi expression language add two properties in the processor. Example: azure ${query.input.tables:startsWith("a")} //only tablenames starts with a gcloud ${query.input.tables:startsWith("e"):or(${query.input.tables:startsWith("a")})} //we are going to route table names starts with e(or)a to gcloud Feed the gcloud relationship to PutGCSobject processor and azure relationship to PutAzureBlobStorage processor. Refer to this link for NiFi expression language and make your expression that can route only the required tables to azure,gcs. In addition i have used only single database to list all the tables but if your 3k tables are coming from different databases then use GenerateFlowfile processor and add all the list of databases.Extract each database name as attribute --> feed the success relationship to SelectHiveQL processor. Refer to this link dynamically pass database attribute to first select hiveql processor. Reference flow.xml load-hivetables-to-azure-gcs195751.xml - If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues. ... View more

@Winnie Philip Query Record processor dynamic properties allows expression language, so we can use Distributed Cache for this case. Flow1:Load id's into DistributedCache: Use PutDistributedMapCache server and load all your required id's into Cache. Flow2: Fetch id's from DistributeCache: In the original flow use FetchDistributedMapCache processor to fetch the cached data and keep them as attribute, use the same attribute name in your QueryRecord processor. QueryRecord dynamic property query would be SELECT * FROM FLOWFILE where VBEOID in (${fetch_distributed_cache_attribute_name}) You may have to change MaxCacheEntrysize property in PutDistributedMapCache and Max Length To Put In Attribute property in FetchDistributedMapCache processors as per the size data going to cache and retrieve from cache. Sample Flow: Refer to this link for configuring and usage of Put/FetchDistributedCache processors. By using this method we are not hard coding the values in QueryRecord processor based on the attribute value from the FetchDistributedCache we are running query in QueryRecord processor dynamically. ... View more