1973
Posts
1225
Kudos Received
124
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1931 | 04-03-2024 06:39 AM | |
| 3027 | 01-12-2024 08:19 AM | |
| 1659 | 12-07-2023 01:49 PM | |
| 2430 | 08-02-2023 07:30 AM | |
| 3381 | 03-29-2023 01:22 PM |
01-05-2017
03:47 AM
Did you upgrade to Nifi 1.1 Remove old SOAP processor and add latest edition references above? Stop NiFi and make sure no Java processes were running? (Might even want to reboot to clear JVM) Then add new NAR, restart NiFi, create a new flow with new SOAP processor. There are a lot of issues with complex SOAP security and encryption. Can you access this SOAP service with SOAPUI? Regular Java code? If it is complex, you could write your own NIFI custom processor that wraps your specific Java call.
... View more
01-04-2017
07:43 PM
Weird NIFI 1.1 warnings 2017-01-04 19:40:34,445 WARN [NiFi Web Server-1012] org.eclipse.jetty.http.HttpParser Illegal character 0x16 in state=START for buffer HeapByteBuffer@1dd9f5b4[p=1,l=227,c=8192,r=226]={\x16<<<\x03\x01\x00\xDe\x01\x00\x00\xDa\x03\x03\x16\xF8\xBd[\x9eC\xE3...\x00\x08\xFa\xFa\x00\x1d\x00\x17\x00\x18ZZ\x00\x01\x00>>> Chrome/55.0.2883...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2017-01-04 19:40:34,445 WARN [NiFi Web Server-1054] org.eclipse.jetty.http.HttpParser Illegal character 0x16 in state=START for buffer HeapByteBuffer@1a1c27d[p=1,l=227,c=8192,r=226]={\x16<<<\x03\x01\x00\xDe\x01\x00\x00\xDa\x03\x03\xA29\xC3\xD6\xA7{\xD8...\x00\x08::\x00\x1d\x00\x17\x00\x18\xDa\xDa\x00\x01\x00>>>5 Safari/537.36\r\n...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2017-01-04 19:40:34,453 WARN [NiFi Web Server-1054] org.eclipse.jetty.http.HttpParser bad HTTP parsed: 400 Illegal character 0x16 for HttpChannelOverHttp@29677b4c{r=0,c=false,a=IDLE,uri=null}
2017-01-04 19:40:34,453 WARN [NiFi Web Server-1012] org.eclipse.jetty.http.HttpParser bad HTTP parsed: 400 Illegal character 0x16 for HttpChannelOverHttp@2b00525e{r=0,c=false,a=IDLE,uri=null}
2017-01-04 19:42:51,278 ERROR [Provenance Maintenance Thread-6] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Provenance Maintenance Thread-6,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-01-04 19:42:51,278 ERROR [Flow Service Tasks Thread-1] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Flow Service Tasks Thread-1,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-01-04 19:42:51,278 ERROR [FileSystemRepository Workers Thread-1] o.a.n.c.repository.FileSystemRepository Failed to handle destructable claims due to java.lang.OutOfMemoryError: Java heap space
2017-01-04 19:42:51,279 ERROR [Provenance Maintenance Thread-6] org.apache.nifi.NiFi
java.lang.OutOfMemoryError: Java heap space
2017-01-04 19:42:51,279 ERROR [Flow Service Tasks Thread-1] org.apache.nifi.NiFi
java.lang.OutOfMemoryError: Java heap space
2017-01-04 19:42:51,279 WARN [NiFi Web Server-1548] o.e.jetty.util.thread.QueuedThreadPool
java.lang.OutOfMemoryError: Java heap space
2017-01-04 19:42:51,280 WARN [NiFi Web Server-1548] o.e.jetty.util.thread.QueuedThreadPool Unexpected thread death: org.eclipse.jetty.util.thread.QueuedThreadPool$2@4c5a6f9f in NiFi Web Server{STARTED,8<=10<=200,i=1,q=0}
2017-01-04 19:42:51,283 WARN [NiFi Web Server-1550] org.eclipse.jetty.servlet.ServletHandler Error for /
java.lang.OutOfMemoryError: Java heap space
... View more
Labels:
- Labels:
-
Apache NiFi
01-04-2017
05:01 PM
6 Kudos
My first caveat would be that in my tests, the pre-trained models is missing a lot of names. If this is for a production work load, I would recommend training your own models using your own data. Maybe use all of your corporate directory, client list, Salesforce data, LinkedIn and social media. I would recommend full name, first names and any nicknames that are commonly used. The current version is 1.7.0 and there are pre-trained 1.5.0 models that work. They have a number of pre-trained models in a few human languages. I chose English (http://opennlp.sourceforge.net/models-1.5/en-ner-person.bin). Walk Through: Create TokenNameFinderModel from pre-built person model. Tokenize the input sentence. Find the identified people. Convert to JSON array. You can easily plug this into a custom NiFi processor, microservice, command line tool or routine in a larger Apache Storm or Apache Spark pipeline. Code (JavaBean) public class PersonName {
private String name = "";
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
} Code (getPeople) import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
import com.google.gson.Gson;
import opennlp.tools.namefind.NameFinderME;
import opennlp.tools.namefind.TokenNameFinderModel;
import opennlp.tools.tokenize.SimpleTokenizer;
import opennlp.tools.tokenize.Tokenizer;
import opennlp.tools.tokenize.TokenizerME;
import opennlp.tools.tokenize.TokenizerModel;
import opennlp.tools.util.InvalidFormatException;
import opennlp.tools.util.Span;
public String getPeople(String sentence) {
String outputJSON = "";
TokenNameFinderModel model = null;
try {
model = new TokenNameFinderModel(
new File("en-ner-person.bin"));
} catch (InvalidFormatException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
NameFinderME finder = new NameFinderME(model);
Tokenizer tokenizer = SimpleTokenizer.INSTANCE;
String[] tokens = tokenizer.tokenize(sentence);
Span[] nameSpans = finder.find(tokens);
List<PersonName> people = new ArrayList<PersonName>();
String[] spanns = Span.spansToStrings(nameSpans, tokens);
for (int i = 0; i < spanns.length; i++) {
people.add(new PersonName(spanns[i]));
}
outputJSON = new Gson().toJson(people);
finder.clearAdaptiveData();
return "{\"names\":" + outputJSON + "}";
}
I used Eclipse for building and testing and you can build it with mvn package. Maven <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.dataflowdeveloper</groupId>
<artifactId>categorizer</artifactId>
<packaging>jar</packaging>
<version>1.0</version>
<name>categorizer</name>
<url>http://maven.apache.org</url>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.7</version>
</dependency>
<dependency>
<groupId>org.apache.opennlp</groupId>
<artifactId>opennlp-tools</artifactId>
<version>1.7.0</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.0</version>
</dependency>
</dependencies>
</project>
Run Input: Tim Spann is going to the store. Peter Smith is using Hortonworks Hive.
Output: {"names":[{"name":"Tim Spann"},{"name":"Peter Smith"}]} Reference: http://opennlp.apache.org/ http://opennlp.apache.org/documentation/1.7.0/manual/opennlp.html#tools.namefind https://www.packtpub.com/books/content/finding-people-and-things http://opennlp.sourceforge.net/models-1.5/
... View more
01-04-2017
04:28 PM
that's painful. I will send this discussion to the NIFI committers
... View more
01-04-2017
02:15 PM
1 Kudo
see here: https://community.hortonworks.com/questions/60081/using-nifi-soap-processor.html https://github.com/apsaltis/nifi-soap has been updated 7 hours ago with this comment: "Updating to use NiFi 1.1.0" Now it runs without "nifi-app.log:java.lang.NoSuchMethodError: org.apache.nifi.processors.soap.GetSOAP.getLogger()Lorg/apache/nifi/logging/ProcessorLog;" error
... View more
01-01-2017
03:29 PM
You may need to open a JIRA with spark.apache.org or parquet. Seems an issue in one of them.
... View more
01-01-2017
03:22 PM
you can move the bad files out of the directory
... View more
12-30-2016
06:12 PM
https://community.hortonworks.com/questions/62213/nifi-putsql-row-length-exception-for-phoenix-upser.html
... View more