Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar
Master Guru

ExtractText NiFi Custom Processor Powered by Apache Tika

Apache Tika is amazing, it is very easy to use it to analyze file and then to extract text with it. Apache Tika uses other powerful Apache projects like Apache PDFBox and Apache POI.

12171-tikaflow.png

Example Usage

  1. Feed in documents, I use my LinkProcessor which grabs links from a website and returns a JSON List.
  2. Split the resulting JSON list into individual JSON rows with SplitJSON.
  3. EvaluateJSONPath to extract just the URLs.
  4. InvokeHTTP to do a GET on that parsed URL.
  5. RouteOnAttribute to only process file types I am interested in like Microsoft Word.
  6. The new ExtractTextProcessor to extract the text of the document.
  7. 12173-tikaextractadd.png
  8. Then we save the text as a file in some data store, perhaps HDFS.

12174-tikaresultsprops.png

12172-tikaoutput.png

If you have a directory of files, you can just use GetFile to ingest them en masse.

LinkProcessor (https://github.com/tspannhw/linkextractorprocessor)

URL:  http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/index.html

This is an example of a URL that I want to grab all the documents from. You can point it at any URL that has links to documents (HTML, Word, Excel, PowerPoint, etc...).

RouteOnAttribute

I only want to process a few types of files, so I limit them here.

${filename:endsWith('.doc'):or(${filename:endsWith('.pdf')}):or(${filename:endsWith('.rtf')}):or(${filename:endsWith('.ppt')}):or( 
${filename:endsWith('.docx')}):or(${filename:endsWith('.pptx')}):or(${filename:endsWith('.html')}):or(${filename:endsWith('.htm')}):or(${filename:endsWith('.xls')}):or( 
${filename:endsWith('.xlsx')}):or(${filename:endsWith('.xml')}):or(${Content-Type:contains('text/html')}):or(${Content-Type:contains('application/pdf')}):or( 
${Content-Type:contains('application/msword')}):or(${Content-Type:contains('application/vnd')}):or(${Content-Type:contains('text/xml')})}

Release:

https://github.com/tspannhw/nifi-extracttext-processor/releases/tag/1.0

Reference:

13,748 Views
Comments
avatar
New Contributor

Hi,

I found your really interesting article about extracting text from any kind of files. I tried to test your processor but every time I have an error message: Error: ExtractTextProcessor[id=...] Apache Tika failed to parse input Unable to extract PDF content

The Apache NiFi is installed on a separate Ubuntu server 16.04 (only NiFi), not on Hortonworks sandbox. Do I need to install also Tika on the same server, because I imported only your processor in NiFi and run it? I tested also on the sandbox HDFS with Apache NiFi, the similar error message.

Sorry, I'm new in this field and any help any help would be appreciated.

Thank you

Gojko

avatar
Master Guru

Try the new one: https://community.hortonworks.com/articles/177370/extracting-html-from-pdf-excel-and-word-documents....

Make sure download the nar from github and put in lib directory then reboot. Make sure you are running NiFi with JDK 8.

avatar
New Contributor

Thanks, the version 1.7 (I saw already new one 1.17) works fantastic.

Great job.

avatar
Master Guru

copy to NiFi Lib Directory and restart

avatar
Explorer
I'm kinda new at Nifi, so I'm sure I'm doing something wrong.
 
Here's what I'm doing:

01.  I go to GitHub link     https://github.com/tspannhw/nifi-extracttext-processor/releases
02.  I download 'Source code.zip' to 'nifi-extracttext-processor-html' on my laptop
03.  I extract 'nifi-extracttext-processor-html'
04.  In Nifi, I upload the template '56409-tika.xml'
05.  When I try to add the 'tika' template to the Nifi Canvas, I'm still getting the 'org.apache.nifi.processors.kite.InferAvroSchema is not known to this NiFi instance.' error
 
 
Thanks again
avatar
Master Guru

that's an old example don't use it.   Inferavroschema is not needed.

 

just download the extract nar and put it in your nifi lib and restart.   then add extract processor to your nifi to load.

 

i don't have this example app anymore or this version so i can't resave it without the InferAvroSchema processor which is not needed or used here.

avatar
Explorer

Hello,

 

I see the 'nifi-extracttext-nar' folder, but I don't see '.nar' file in it.  Am I missing something?

 

Thanks

avatar
Explorer

Thanks for everything.  That worked great.

avatar
Master Guru

Awesome.   Good luck with NiFi.