Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar
Super Collaborator

In this article we will go over how we can use nifi to ingest PDFs and while we ingest we will use a custom groovy script with executescript processor to extract images from this PDF. The images will be tagged with PDF filename, page num and imagenum , so that it can be indexed in hbase/solr for quick searching and doing some machine learning or other analytics.

The Groovy code below depends on the following jars. Download and copy them to a folder on your computer. I my case they were under /var/pdimagelib/.

  • pdfbox-2.0.7.jar
  • fontbox-2.0.7.jar
  • jai_imageio.jar
  • commons-logging-1.1.1.jar

Copy the code below to a file on your computer, in my case the file was under /var/scripts/pdimage.groovy

import java.nio.charset.*;

import org.apache.commons.io.IOUtils;

import java.awt.image.BufferedImage;

import java.awt.image.RenderedImage;

import java.io.File;

import java.io.FileOutputStream;

import java.util.Iterator;

import java.util.List;

import java.util.ArrayList;

import java.util.Map;

import javax.imageio.ImageIO;

import org.apache.pdfbox.cos.COSName;

import org.apache.pdfbox.pdmodel.PDDocument;

import org.apache.pdfbox.pdmodel.PDPage;

import org.apache.pdfbox.pdmodel.PDPageTree;

import org.apache.pdfbox.pdmodel.PDResources;

import org.apache.pdfbox.pdmodel.graphics.PDXObject;

import org.apache.pdfbox.pdmodel.graphics.image.PDImageXObject; 

 

 def flowFile = session.get();

 if (flowFile == null) {

     return;

 }

def ffList = new ArrayList<FlowFile>()

try {

  session.read(flowFile,

     {inputStream ->

             PDDocument document = PDDocument.load(inputStream);

             int imageNum = 1;

             int pageNum = 1;

             document.getDocumentCatalog().getPages().each(){page ->

                 PDResources pdResources = page.getResources();

                 pdResources.getXObjectNames().each(){ cosName->

                 if (cosName != null) {

                   PDXObject image =pdResources.getXObject(cosName);

                   if(image instanceof PDImageXObject){

                    PDImageXObject pdImage = (PDImageXObject)image;

                   BufferedImage imageStream = pdImage.getImage();

                   imgFF = session.create(flowFile);

                   outputStream = session.write(imgFF,

                   { outputStream->

                           ImageIO.write((RenderedImage)imageStream, "png", outputStream);

                    } as OutputStreamCallback)

                    imgFF = session.putAttribute(imgFF, "imageNum",String.valueOf(imageNum));

                    imgFF = session.putAttribute(imgFF, "pageNum",String.valueOf(pageNum));

                    ffList.add(imgFF);

                    imageNum++;

                  }

                 }

                 }

               pageNum++;

               

             }

  } as InputStreamCallback)

 session.transfer(ffList, ExecuteScript.REL_SUCCESS)

 session.remove(flowFile);  

} catch (Exception e) {

           log.warn(e.getMessage());

           e.printStackTrace();

   session.remove(ffList);

          session.transfer(flowFile, ExecuteScript.REL_FAILURE);

     }

Below is a screenshot of executescript after it has been setup correctly.

39539-screen-shot-2017-09-26-at-124404-am.png

To ingest the PDF, i used a simple GetFile, though this approach should work for pdfs ingested with any other nifi processor. Below is a snapshot of the nifi flow. When a PDF is ingested, executescript will leverage groovy and pdfbox to extract images. The images will be tagged with the pdf filename,pagenum and imagenum. This can now be sent to hbase or any indexing solution for search or analytics.

39540-screen-shot-2017-09-26-at-124334-am.png

Hope you find the article useful. Please comment withy your thoughts/questions.

6,090 Views
Comments
avatar
Master Guru

sounds like a good custom processor

avatar
Super Collaborator

was going through some ORC based processing from pdf. looks it works better if each page is split into a monochrome image. Examples online show Ghostscript as an option. I was able to leverage this processor to extract images and with a property change it to grayscale if needed. I could now send this to OCR processor for extraction. @Jeremy Dyer

Version history
Last update:
‎08-17-2019 10:55 AM
Updated by:
Contributors