Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi PutSNS - Size Exceeds 256KB

Solved Go to solution
Highlighted

Nifi PutSNS - Size Exceeds 256KB

New Contributor

I'm using a PutSNS processor which puts JSON (from API call) into SNS, in some cases JSON is more than 256KB, and I get "Size exceeds AWS SNS Limit of 256KB". Is there a way to chunk this into smaller sets?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Nifi PutSNS - Size Exceeds 256KB

@MKS_AWS There are a few ways to break up JSON within a flowfile (splitJson,QueryRecord).  However if its just one giant blob of JSON you may not find that very useful.    Perhaps you can share some sample json to that effect.


Check out this way to do SNS up to 2gb per payload:

 

https://github.com/awslabs/amazon-sns-java-extended-client-lib

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

View solution in original post

1 REPLY 1

Re: Nifi PutSNS - Size Exceeds 256KB

@MKS_AWS There are a few ways to break up JSON within a flowfile (splitJson,QueryRecord).  However if its just one giant blob of JSON you may not find that very useful.    Perhaps you can share some sample json to that effect.


Check out this way to do SNS up to 2gb per payload:

 

https://github.com/awslabs/amazon-sns-java-extended-client-lib

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here