Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to define input directory path in listFile processor if multiple folders exists ?

how to define input directory path in listFile processor if multiple folders exists ?

New Contributor

Hi Team,

I need to read files from multiple folders. In listFile processor, i have given like below

input Directory : / homepath / [customer_*] / inbox /

This is causing invalid processor. how to use expression language to ready multiple folder paths.

or

if i give /homepath/ only input directory, it is reading all the folders along with inbox folder under customer_* directory.

local Directory path :

/homepath/customer_A/inbox/

/homepath/customer_B/inbox/

/homepath/customer_C/inbox/

/homepath/customer_D/inbox/

can anyone suggest me how to use expression language to define the input directory path in listFile processor.

Thanks,
Rangareddy Y

6 REPLIES 6
Highlighted

Re: how to define input directory path in listFile processor if multiple folders exists ?

Expert Contributor

This should work (no brackets):

/homepath/customer_*/inbox/

Another solution would be to create a separate folder for example: /listFiles/. Then configure your processor to hit that folder with Recurse Subdirectories TRUE. Then inside that folder create symlinks to the real folders you want:

ln -s /homepath/customer_A/inbox/ /listFiles/customer_A_inbox/
ln -s /homepath/customer_B/inbox/ /listFiles/customer_B_inbox/
ln -s /homepath/customer_C/inbox/ /listFiles/customer_C_inbox/
ln -s /homepath/customer_D/inbox/ /listFiles/customer_D_inbox/

If this answer is helpful, please choose Accept to mark your question as answered.

Re: how to define input directory path in listFile processor if multiple folders exists ?

New Contributor

@Steven Matison

/homepath/customer_*/inbox/ is giving error. processor is invalid and getting directory does not exist error.

Re: how to define input directory path in listFile processor if multiple folders exists ?

Rising Star

Yes you can do this by using regex expression Below regex expression should suffice your need

/homepath/customer_*/inbox/

If there are only specific number of customers then you can user something like

/homepath/customer_[0-9]/inbox/

/homepath/customer_[A-Z]/inbox/

Note: Please upvote and accept this answer if you found it useful

Re: how to define input directory path in listFile processor if multiple folders exists ?

New Contributor

Used the same regex expression as input directory property value. processor is invalid and getting directory does not exist error.

Re: how to define input directory path in listFile processor if multiple folders exists ?

New Contributor

Thanks for the reply...

sorry given wrong directory names..there is no name matching in folder names...in this case, how to give the path..like below

/homepath/ranga/inbox/

/homepath/krishna/inbox/

/homepath/kaja/inbox/

/homepath/jilebhi/inbox/

/homepath/laddu/inbox/

/homepath/bundhi/inbox/


like 100+ folders are there...i have used multiple listFile processor based on folder size files.

lets assume, 3 listFile processors i used in my flow and want to configure below paths.

in first listFile processor, i want to point (ranga, krishna) folder paths,
in second listFile processor, i want to point (kaja, jilebhi) folder paths,

in third listFile processor, i want to point (laddu, bundhi) folder paths,

can you please suggest me how to achieve above case...

Thanks,
Rangareddy Y

Re: how to define input directory path in listFile processor if multiple folders exists ?

New Contributor

Thanks for the reply...

sorry given wrong directory names..there is no name matching in folder names...in this case, how to give the path..like below

/homepath/ranga/inbox/

/homepath/krishna/inbox/

/homepath/kaja/inbox/

/homepath/jilebhi/inbox/

/homepath/laddu/inbox/

/homepath/bundhi/inbox/


like 100+ folders are there...i have used multiple listFile processor based on folder size files.

lets assume, 3 listFile processors i used in my flow and want to configure below paths.

in first listFile processor, i want to point (ranga, krishna) folder paths,
in second listFile processor, i want to point (kaja, jilebhi) folder paths,

in third listFile processor, i want to point (laddu, bundhi) folder paths,

can you please suggest me how to achieve above case...

Thanks,
Rangareddy Y

Don't have an account?
Coming from Hortonworks? Activate your account here