Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

storing JSON records in S3

storing JSON records in S3

Expert Contributor

My nifi workflow is like this:

1. Generate Table Fetch

2. Extract Text

3. ExecuteSQL

4. SplitAVRO

5.AvroToJSON

6. Put S3

Now, in, put s3, object key by default in ${filename}.

It appears to me that records that were in the same file are getting overwritten. How do I handle this?

Ofcourse I need them all, preferably in the same file that were in same file before splitting.

1 REPLY 1

Re: storing JSON records in S3

Super Guru

@Simran Kaur,

if the flowfiles are overwriting then you need to use object key as

${UUID()}

instead ${filename}. This UUID is uniquely generated you will not have overwriting issues anymore.

Don't have an account?
Coming from Hortonworks? Activate your account here