Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Write to ADLS through Python Script in NIFI

Highlighted

Write to ADLS through Python Script in NIFI

New Contributor

All, I am trying to write contents oh HTTP response in JSON to ADLS. The JSON is modified for few fields using a Python script in before writing to ADLS. My idea is to once the JSON is modified, connect to ADLS through the Python script itself and write the data. I just wrote the script and am able to write the modified JSON response into my local disk. But I am not sure how can I modify the outputstream.write so that it can write to ADLS. Below is the code.

class PyStreamCallback(StreamCallback): def __init__(self): pass def process(self, inputStream, outputStream): text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) obj = json.loads(text) outputStream.write(bytearray(json.dumps(obj).encode('utf-8')))

flowFile = session.get() if (flowFile != None): flowFile = session.write(flowFile,PyStreamCallback()) session.transfer(flowFile, REL_SUCCESS)

Any lead is appreciated as I am stuck. I have the credentials needed for connecting to Data Lake.