question Re: ConsumePop3: Parsing Messages with Nifi 1.0.0 in Support Questions

ConsumePop3: Parsing Messages with Nifi 1.0.0

TimothySpann — Thu, 20 Oct 2016 02:07:02 GMT

When I route to ExtractEmailHeaders and Attachments, they always fail. Is there a specific type of email server or extra processing required?

Example email with no attachments

Apache NiFi Tutorials


The following tutorials show you how to work with different
processors, create dataflows and display data in various dashboards.
There’s also an intro to MiniFi which is a smaller NiFi to run on
embedded or tiny devices.


Using Apache NiFi with Apache Kafka Best Practices and Tutorial
Using Apache NiFi with Sensors Tutorial
Change Data Capture (CDC) with Apache NiFi
An Introduction to Apache NiFi 1.0.0
Apache NiFi 1.0.0 Crash Course
DataFlow with Apache NiFi (includes Video)
Real-Time Streaming Recommendations with NiFi, Kafka and Spark
Setting up Apache NiFi 1.0.0 Zero Master Clusters
Getting Started with HDF 2.0
Creating a Kibana Dashboard of Twitter Data Pushed with NiFi
NiFi For Geoenrichment and Routing of Log Messages
Running Google Vision From Apache NiFi
Getting Started with MiniFi
How to Add Kerberos Authentication to Apache NiFi 1.0.0
Converting CSV Data to AVRO with Apache NiFi

nifi-app.log:2016-10-19 18:56:09,054 ERROR [Timer-Driven Process Thread-6] o.a.n.p.email.ExtractEmailAttachments
nifi-app.log:javax.mail.MessagingException: Message failed RFC2822 validation
nifi-app.log:	at org.apache.nifi.processors.email.ExtractEmailAttachments$1.process(ExtractEmailAttachments.java:138) ~[nifi-email-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]
nifi-app.log:	at org.apache.nifi.processors.email.ExtractEmailAttachments.onTrigger(ExtractEmailAttachments.java:124) [nifi-email-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]
nifi-app.log:2016-10-19 18:56:09,054 ERROR [Timer-Driven Process Thread-1] o.a.n.p.email.ExtractEmailHeaders
nifi-app.log:javax.mail.MessagingException: Message failed RFC2822 validation
nifi-app.log:	at org.apache.nifi.processors.email.ExtractEmailHeaders$1.process(ExtractEmailHeaders.java:160) ~[nifi-email-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]
nifi-app.log:	at org.apache.nifi.processors.email.ExtractEmailHeaders.onTrigger(ExtractEmailHeaders.java:146) [nifi-email-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]
nifi-app.log:2016-10-19 18:56:09,056 ERROR [Timer-Driven Process Thread-10] o.a.n.p.email.ExtractEmailAttachments ExtractEmailAttachments[id=31a043b1-d7a6-1806-80ef-3040b8bc0bce] Could not parse the flowfile StandardFlowFileRecord[uuid=a963b2b3-4cb1-4155-a251-b0ae2089299a,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1476903369035-1627, container=default, section=603], offset=16199, length=45886],offset=0,name=1994148034455292,size=45886] as an email, treating as failure: javax.mail.MessagingException: Message failed RFC2822 validation
nifi-app.log:2016-10-19 18:56:09,056 ERROR [Timer-Driven Process Thread-4] o.a.n.p.email.ExtractEmailHeaders ExtractEmailHeaders[id=31a043b2-d7a6-1806-e673-2e2b86f5c83e] Could not parse the flowfile StandardFlowFileRecord[uuid=eed4f0e7-d63f-4f5a-9b1c-986087bc896e,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1476903369035-1627, container=default, section=603], offset=16199, length=45886],offset=0,name=1994148034455292,size=45886] as an email, treating as failure: javax.mail.MessagingException: Message failed RFC2822 validation
nifi-app.log:2016-10-19 18:56:09,057 ERROR [Timer-Driven Process Thread-10] o.a.n.p.email.ExtractEmailAttachments
nifi-app.log:javax.mail.MessagingException: Message failed RFC2822 validation
nifi-app.log:	at org.apache.nifi.processors.email.ExtractEmailAttachments$1.process(ExtractEmailAttachments.java:138) ~[nifi-email-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]
nifi-app.log:	at org.apache.nifi.processors.email.ExtractEmailAttachments.onTrigger(ExtractEmailAttachments.java:124) [nifi-email-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]
nifi-app.log:2016-10-19 18:56:09,057 ERROR [Timer-Driven Process Thread-4] o.a.n.p.email.ExtractEmailHeaders
nifi-app.log:javax.mail.MessagingException: Message failed RFC2822 validation
nifi-app.log:	at org.apache.nifi.processors.email.ExtractEmailHeaders$1.process(ExtractEmailHeaders.java:160) ~[nifi-email-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]
nifi-app.log:	at org.apache.nifi.processors.email.ExtractEmailHeaders.onTrigger(ExtractEmailHeaders.java:146) [nifi-email-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]

Example partial email with encoding

--001a11404b903cb34b053f3b322c
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable


   - Hortonworks DataFlow Webinar Series
   <http://hortonworks.com/blog/hortonworks-dataflow-webinar-series/>Octobe=
r
   18, 2016
   HDF makes streaming analytics faster and easier, by enabling accelerated
   data collection, curation, analysis and delivery in real-time, on-premis=
es
   or in the cloud through an integrated solution with Apache NiFi, Kaf

Re: ConsumePop3: Parsing Messages with Nifi 1.0.0

ozhurakousky — Thu, 20 Oct 2016 02:16:36 GMT

Timothy

It appears that the incoming email doesn't contain 'from' and/or 'sentDate'. Basically there is explicit validation for those fields inside the processor and if both missing the exception is triggered. The argument there is that it violates 'RFC2822' (Internet Message Format).

While it may be correct (I have to look at the doc itself) I am not sure that the goal for this processor is to enforce the mentioned spec.

Anyway, would you mind creating JIRA in https://issues.apache.org/jira/browse/NIFI and I can give it a closer look. Also, if you can find more details as to why the incoming message missing those field.

Cheers

Oleg

Re: ConsumePop3: Parsing Messages with Nifi 1.0.0

TimothySpann — Thu, 20 Oct 2016 02:33:43 GMT

https://community.hortonworks.com/questions/58848/rfc-2822-validation-error.html

http://apache-nifi.1125220.n5.nabble.com/Regarding-ConsumeIMAP-Processor-td12249.html

Re: ConsumePop3: Parsing Messages with Nifi 1.0.0

TimothySpann — Thu, 20 Oct 2016 02:34:33 GMT

I'll take a look at the exact messages sent, but it's what's pulled in by ConsumePOP3.

Re: ConsumePop3: Parsing Messages with Nifi 1.0.0

ozhurakousky — Thu, 20 Oct 2016 02:37:38 GMT

Yes, I understand and that's why I am having second thoughts about the check in the processor. In other words we may need to remove it

Re: ConsumePop3: Parsing Messages with Nifi 1.0.0

TimothySpann — Thu, 20 Oct 2016 02:48:18 GMT

1994148034455292.txt

Re: ConsumePop3: Parsing Messages with Nifi 1.0.0

TimothySpann — Tue, 25 Oct 2016 01:32:41 GMT

nifi-bootstrap.log:2016-10-24 18:30:33,712 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: connecting to host "pop.x.com", port 110, isSSL false nifi-bootstrap.log:2016-10-24 18:30:33,936 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: PIPELINING enabled nifi-bootstrap.log:2016-10-24 18:30:33,936 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: authentication command trace suppressed nifi-bootstrap.log:2016-10-24 18:30:34,090 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: authentication command succeeded nifi-bootstrap.log:2016-10-24 18:30:38,072 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 1 nifi-bootstrap.log:2016-10-24 18:30:38,156 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 2 nifi-bootstrap.log:2016-10-24 18:30:38,229 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 3 nifi-bootstrap.log:2016-10-24 18:30:38,304 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 4 nifi-bootstrap.log:2016-10-24 18:30:38,380 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 5 nifi-bootstrap.log:2016-10-24 18:30:38,531 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 6 nifi-bootstrap.log:2016-10-24 18:30:38,607 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 7 nifi-bootstrap.log:2016-10-24 18:30:38,690 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 8 nifi-bootstrap.log:2016-10-24 18:30:38,769 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 9 nifi-bootstrap.log:2016-10-24 18:30:38,846 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 10 nifi-bootstrap.log:2016-10-24 18:30:38,925 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 11 nifi-bootstrap.log:2016-10-24 18:30:39,014 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 12 nifi-bootstrap.log:2016-10-24 18:30:39,090 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 13 nifi-bootstrap.log:2016-10-24 18:30:39,167 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 14 nifi-bootstrap.log:2016-10-24 18:30:39,242 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 15 nifi-bootstrap.log:2016-10-24 18:30:39,329 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 16 nifi-bootstrap.log:2016-10-24 18:30:39,395 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 17 nifi-bootstrap.log:2016-10-24 18:30:39,473 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 18 nifi-bootstrap.log:2016-10-24 18:30:39,551 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 19 nifi-bootstrap.log:2016-10-24 18:30:39,627 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 20 nifi-bootstrap.log:2016-10-24 18:30:39,701 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 21 nifi-bootstrap.log:2016-10-24 18:30:39,781 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 22 nifi-bootstrap.log:2016-10-24 18:30:39,856 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 23 nifi-bootstrap.log:2016-10-24 18:30:40,121 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 24 nifi-bootstrap.log:2016-10-24 18:30:40,196 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 25 nifi-bootstrap.log:2016-10-24 18:30:40,437 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: mail.pop3.apop.enable: false nifi-bootstrap.log:2016-10-24 18:30:40,437 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: mail.pop3.disablecapa: false nifi-bootstrap.log:2016-10-24 18:30:40,437 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: connecting to host "pop.agilemobiledeveloper.com", port 110, isSSL false nifi-bootstrap.log:2016-10-24 18:30:40,663 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: PIPELINING enabled