Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ExecuteScript does not finish subprocess module of python

Highlighted

ExecuteScript does not finish subprocess module of python

New Contributor

Hello,

I am using the ExecuteScript processor to execute a shell command. Therefore I tried several functions of the subprocess library (like check_output, check_call, call, Popen etc.), the os library (os.system(cmd)) and the commands library. But they all have the same behavior: It seems that the ExecuteScript Processor does not wait for the subprocess to finish the command, even if I added the .wait() function to do so...

Here is my code:

import subprocess
import os
import time
import commands
import cmd
flowFile = session.get() 
if (flowFile != None):
    filename = flowFile.getAttribute('filename')
    file_id = filename.split('/')[-1]
    route= flowFile.getAttribute('RouteOnAttribute.Route')
    storePath = '/home/user/input/' + route
    log.info('Filename: ' + filename)
    command= 'aws s3 cp s3://<s3_bucket>/'+filename+ ' ' + storePath 
    flowFile  = session.putAttribute(flowFile,'cmd',command)
    # I tried the following commands:
    #ret = commands.getstatusoutput(cmd)
    #ret = subprocess.Popen(cmd,env=os.environ)
    # p = subprocess.check_output([c for c in cmd.split(' ')])
    # p = subprocess.call([c for c in cmd.split(' ')])
    #ret.wait()
    p = subprocess.check_call([c for c in cmd.split(' ')])
    flowFile = session.putAttribute(flowFile,'storePath',storePath + '/' +file_id)
    #flowFile = session.putAttribute(flowFile,'returncode',str(ret[0]))
    #flowFile = session.putAttribute(flowFile,'ouput',str(ret[1]))
if os.path.isfile(storePath + '/' + file_id):
    session.transfer(flowFile, REL_SUCCESS)
else:
    session.transfer(flowFile, REL_FAILURE)

Is there a way to explicity wait for the subprocess to finish its task? I also copied the the 'cmd' attribute value and executed it on the shell without any problems.

Kind regards,

Jan

Don't have an account?
Coming from Hortonworks? Activate your account here