Member since
07-04-2018
17
Posts
1
Kudos Received
0
Solutions
04-01-2019
05:57 PM
I have created a flow to read json messages from kafka, and covert them to avro so that I can use ConvertAvroToParquet. My issue is that I can only store one avro message in a parquet file. Is there away to store multiple records within a parquet file using ConvertAvroToParquet?
... View more
Labels:
- Labels:
-
Apache NiFi
03-22-2019
07:22 PM
@Matt Clarke, do you mean stopping the nifi process on all nodes, and can a rolling restart of the process also work?
... View more
03-22-2019
05:47 PM
One of the nodes in our nifi cluster experienced some issues and had to be shutdown and deleted. In Cluster, under the hamburger menu, the node is showing as offloading for almost a week even though the node is deleted. Is there any way to forcefully remove the node from the cluster so that I can make changes to the flow?
... View more
Labels:
- Labels:
-
Apache NiFi
07-09-2018
03:12 AM
The contents of the flow file is 200mb in size, what I am looking to do is retrieve some relevant information for this data in a postgresql so that I can parse the data correctly down the flow, so its necessary that I keep the contents. I have tried to save the contents in an attribute, but it slows the flow down and even causes nodes to disconnect. Haven't looked into LookupAttribute yet, but I tried to modify ExecuteSQL processor so that the query result gets stored into an attribute. I am having issues with that though, as discussed here https://community.hortonworks.com/questions/201941/created-a-custom-nifi-processor-after-placing-nar.html?childToView=201998#comment-201998.
... View more
07-06-2018
05:01 PM
@Matt ClarkeSo I deleted the contents of the work dir and set nifi.flowcontroller.autoResumeState=false. Still same error. I think its an issue with my pom file or dependency related at least.
... View more
07-06-2018
04:02 PM
Earlier, Just to make sure it wasn't the code, I copied the source from ExecuteSQL 1.6 and renamed it. After deployment I still had the same errors. My suspicion is maybe the way I built the nar with its dependencies. I'm hoping that there is an error with my pom file, or like you said, deleting my work dir.
... View more
07-06-2018
03:43 PM
Yes the nar is in a seperate lib dir, and its still causing the issue, will try deleting the work dir see if that works. Any other ideas?
... View more
07-05-2018
09:16 PM
2018-07-05 17:01:05,806 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.standard.ConvertJSONToSQL could not be instantiated
java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.standard.ConvertJSONToSQL could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:148)
at org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:123)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:771)
at org.apache.nifi.NiFi.<init>(NiFi.java:157)
at org.apache.nifi.NiFi.<init>(NiFi.java:71)
at org.apache.nifi.NiFi.main(NiFi.java:292)
Caused by: java.lang.NoClassDefFoundError: org/apache/nifi/dbcp/DBCPService
at org.apache.nifi.processors.standard.ConvertJSONToSQL.<clinit>(ConvertJSONToSQL.java:127)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at java.lang.Class.newInstance(Class.java:442)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
... 8 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.apache.nifi.dbcp.DBCPService
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 15 common frames omitted
2018-07-05 17:01:05,808 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2018-07-05 17:01:05,809 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise). <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-nar-bundles</artifactId>
<version>1.6.0</version>
</parent>
<groupId>fidnifi</groupId>
<artifactId>ExecuteSqlToAttribute</artifactId>
<version>1.6</version>
<packaging>nar</packaging>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<nifi.version>1.6.0</nifi.version>
</properties>
<repositories>
<repository>
<id>jcenter</id>
<url>http://jcenter.bintray.com</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-api</artifactId>
<version>${nifi.version}</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-utils</artifactId>
<version>${nifi.version}</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-processor-utils</artifactId>
<version>${nifi.version}</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-io</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>com.jayway.jsonpath</groupId>
<artifactId>json-path</artifactId>
<version>1.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-mock</artifactId>
<version>${nifi.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-standard-processors</artifactId>
<version>${nifi.version}</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-dbcp-service-api</artifactId>
<version>1.6.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>commons-dbcp</groupId>
<artifactId>commons-dbcp</artifactId>
<version>1.4</version>
</dependency>
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp</artifactId>
<version>3.8.1</version>
</dependency>
<dependency>
<groupId>com.burgstaller</groupId>
<artifactId>okhttp-digest</artifactId>
<version>1.13</version>
<type>jar</type>
<scope>system</scope>
<systemPath>${project.basedir}/src/main/resources/okhttp-digest-1.13.jar</systemPath>
</dependency>
<dependency>
<groupId>com.martiansoftware</groupId>
<artifactId>macnificent</artifactId>
<version>0.2.0</version>
<scope>system</scope>
<systemPath>${project.basedir}/src/main/resources/macnificent-0.2.0.jar</systemPath>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-nar-maven-plugin</artifactId>
<version>1.2.0</version>
<extensions>true</extensions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.0</version>
</plugin>
</plugins>
</build>
</project> If I add the missing class as a jar in /lib folder error goes away, but then get another missing class error on other nifi processors. Any ideas?
... View more
Labels:
- Labels:
-
Apache NiFi
07-05-2018
06:44 PM
Sorry I wasn't clear, I need to retain the contents if the incoming flowfile in ExecuteSQL, that's why I want the results of the query to output in a flowfile attribute otherwise all the data that I need within the original flowfile will be deleted. Thanks.
... View more
07-04-2018
10:18 PM
1 Kudo
I am open to any other nifi processor that that can achieve this as long as it can perform queries in postgresql and result routed to flow file attribute or even modifying the the source code of executesql to achieve this. Thanks!
... View more
Labels:
- Labels:
-
Apache NiFi