Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to get read sqs headers

Highlighted

How to get read sqs headers

New Contributor

I am using GetSQS processor in my usecase, in the incoming sqs messsage there are some message attributes(or headers) specified.

How can I read those message attributes from the getsqs processor ?

7 REPLIES 7
Highlighted

Re: How to get read sqs headers

Expert Contributor

If you're SQS response is XML, use the SplitXML processor and XPath to create fields from the SQS response. An example of using the SplitXML processor is here:

https://community.hortonworks.com/articles/25720/parsing-xml-logs-with-nifi-part-1-of-3.html

Highlighted

Re: How to get read sqs headers

New Contributor

I am not sure that helps, let me elaborate a little bit more

SQS message has 2 parts

Header/Metadata/Message Attributes - These are key value pairs with metadata of the message.

Body/Content/Payload - this could be any data i put in the payload xml, json etc

If I understand correctly, the SplitXML would apply to the payload of the message if it were xml payload. Not sure how it will be useful to extract the header values.

I need to be able to extract the metadata from the sqs.

The PutSQS on the other hand, provides dynamic attributes for exactly this purpose to add message atrributes to the sqs message.

Highlighted

Re: How to get read sqs headers

Super Guru

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.aws.sqs.GetSQS/

Here is the documented source code for that processor:

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/...

it grabs all the attributes:

for (final Map.Entry<String, String> entry : message.getAttributes().entrySet()) {
attributes.put("sqs." + entry.getKey(), entry.getValue());

You will see these in the attributes in the flow file

and these

for (final Map.Entry<String, MessageAttributeValue> entry : message.getMessageAttributes().entrySet()) {
attributes.put("sqs." + entry.getKey(), entry.getValue().getStringValue());

It also puts the body into the flowfile

message.getBody().getBytes

It's using the standard com.amazonaws.services.sqs.AmazonSQSClient so it should be grabbing all the data the same as any SQS client.

Highlighted

Re: How to get read sqs headers

New Contributor

that certainly makes sense, now I am trying to get that attribute like so >>

I have a message attribute 'foo' in my sqs message.

Here's what I am trying to do, get the sqs message , update the attribute 'filename=${foo}' and PutFile

GetSQS -> UpdateAttribute(filename = ${foo}.txt) -> PutFile

I am expecting that a file foo.txt will be created from d payload of the message and put in the folder specified by PutFile.

However the ${foo} attribute is always empty.

Highlighted

Re: How to get read sqs headers

New Contributor

Doesnt seem to work. I enlisted all the attributes using AttributesToJson() and my message attribute(sqs.foo) is not there

{"sqs.ApproximateReceiveCount":"1" "sqs.SentTimestamp":"1476305577165" "path":"./" "filename":"377732797336013" "sqs.receipt.handle":"ABCCCblah" "sqs.ApproximateFirstReceiveTimestamp":"1476305578287" "hash.algorithm":"md5" "uuid":"065fe8cb-84e9-4c64-a973-64f6587b0128" "sqs.message.id":"c22161c4-62d0-4c24-ab81-f50c68c740d4" "sqs.SenderId":"AIDAJQF5U6RNP4URGKOPW" "hash.value":"6813ed62f2481488e664bd6e1a814192"}

Highlighted

Re: How to get read sqs headers

Super Guru

UpdateAttribute will only grab attributes. foo is part of the payload, you can't pull out pieces of the payload. It's not FlowFile attribute. SQS Message Attribute is not a FlowFile Attribute. It's part of the FlowFile Content/Payload. You won't see it in NIFI FlowFile Attributes.

If you payload/content/message is JSON, you can parse it with JSONPATH and make Foo an attribute that way.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.EvaluateJsonPa...

Highlighted

Re: How to get read sqs headers

New Contributor

I also had the issue that GetSQS was not preserving any of my message attributes. Looks like the problem is that the initial message request needs to specify which message attributes to retrieve, or "All" for all of them:

https://stackoverflow.com/questions/25107605/amazon-sqs-java-sdk-cannot-receive-message-attributes

The processor source code is specifying "All" for standard attributes, but not custom message attributes, so I don't think message.getMessageAttributes will ever grab anything.

Don't have an account?
Coming from Hortonworks? Activate your account here