Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Groovy: How to get certain files from HDFS based on filePattern

Highlighted

Groovy: How to get certain files from HDFS based on filePattern

New Contributor

I want to get only those files from HDFS directory which contains certain name ( i have put several response file with name like this 2017-090-0.1, 2017-090-0.2,2017-090-0.3 and etc. now i want to be sure that certain amount of flowFile is put in hdfs ( I MEAN IF I SEND 3 REQUEST WITH RELATED NAME I WILL HAVE TO CHECK IF I HAVE PUT THREE RESPONSE IN HDFS ) for this case i use this code below, BUT IT DEOSN'T FETCH FILES there are several subejects i am interested in :

  1. Is it possbile to make this task without groovy code by nifi capabilities?
  2. WHAT SHOULD I CHANGE TO MAKE THIS CODE WORK?
    import org.apache.commons.io.IOUtils
    import java.nio.charset.StandardCharsets
    def flowFile= session.get(1);//  i  gues  it  will   return    list  of  
    flowfiles
    def name="";
    def  count=0;
    def  value=0;
    def  amount=0;
    List<FlowFile> flowFiles = new ArrayList<FlowFile>();
    for(def  n in  flowFile){
    name=n.getAttribute("realName")
    count=n.getAttribute("count")
      value=count as  Number
      value=Math.round(value)
    }
    session.remove(flowFile)
    def findFileRecursive( String directoryName, String filePattern) {
    def fileFound
    def directory = new File(directoryName)
    if (directory.isDirectory()){
    def findFilenameClosure = { if (filePattern.matcher(it.name).find()){  fileFound = it } }
    directory.eachFileRecurse(findFilenameClosure)
    }
    amount++;
    flowFiles.add(fileFound);
    return fileFound
    }
    String filePattern=filePattern.contains(name)
    String  directoryName="/group/test/userDate";
    findFileRecursive(directoryName,filePattern);
    
    if(amount==count){
    for(def m in flowFiles){
    session.transfer(m,REL_SUCCESS);
    }
    
    }
Don't have an account?
Coming from Hortonworks? Activate your account here