Member since
02-28-2023
3
Posts
0
Kudos Received
0
Solutions
02-28-2023
08:03 AM
That should help. Can you give us a example to apply the same.
... View more
02-28-2023
05:16 AM
The folder structure is like /A/B, /A/C, /A/D, /A/D,..... Like this A is the root directory and there are 6000 subdirectories. We only need to read from 200 specific sub directories. Creating 200 ListHdfs Processors seems to be cluttered. Is there any workaround for this?
... View more
02-28-2023
01:56 AM
We have around 6k tables in HDFS. All these tables are stored as sub-dirctories under a parent directory. As of now we're passing the root directory and recursive subdirectories as true in LISTHdfs processor. And later using routeonattribute processor to filter out the relevant tables using the path name. But the problem here is LISTHDFS is taking larger time just to list out as it needs to go through 6K directories. Is there is anyway possible to pass only required directories in listhdfs processor or any workaround available for this?
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
-
HDFS