Member since
10-06-2017
12
Posts
0
Kudos Received
0
Solutions
01-27-2021
01:54 AM
@sow I am also having the same issue, did you get any resolution for this issue?
... View more
11-23-2017
07:49 AM
@Matt Clarke Thanks Matt, thanks for your reply. I have solved the problem, I was doing it wrong. I placed the input port in a process group, that is why its was not creating any input ports for the source RPG. Now I placed the input port in the main ui page and it created the remote input port for RPG in source.Thanks... Now I have couple more questions: If we want to transmit data from multiple sources then how to distinguish which flow is coming from which source in target? Is there any way to create multiple input ports in target and customize the flow to send the data to different flows or it will just send all the data from various sources to a single input port and if this is the process then how to distinguish which flow is coming from which source and filter the flow to different flows. And is there any way to create target job in a process group? Thank You Again. 🙂
... View more
11-22-2017
03:40 PM
Hi
I am trying to send data from one nifi to another nifi (which are working in different nodes) using remote process group. So i created one remote process group and mention the url of other nifi host. Now whenever i am trying to connect a getfile or generate flowfile with this RPG its giving me error that it does not have input ports for RPG. I have configured the nifi.properties file as mentioned in here https://nifi.apache.org/minifi/getting-started.html and mention the site to site port number but it still not working.
I am attaching some screenshots regarding my problem. Hoping for a solution. nifi1.png nifi2.png nifi3.png
Thanks in Advance.
Pratik
... View more
Labels:
- Labels:
-
Apache NiFi
11-14-2017
07:25 AM
Hi I am using python script and groovy script to do same task. I was trying to check the performance of this two scripts. But i noticed python script is working slow and groovy is working really fast. In this below script i am just replacing the space with "|" in both of the scripts. Python Script: import string
import java.io
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
class PyStreamCallback(StreamCallback):
def __init__(self):
pass
def process(self, inputStream, outputStream):
text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
text=text.replace(",","|",1).replace(" ","|",3).replace("|"," ",1).replace("] ","]|")
outputStream.write(bytearray(text.encode('utf-8')))
flowFile = session.get()
if(flowFile != None):
try:
flowFile = session.write(flowFile, PyStreamCallback())
session.transfer(flowFile, REL_SUCCESS)
except:
log.error('Something went wrong', e)
session.transfer(flowFile, REL_FAILURE) Groovy Script: import org.apache.nifi.processor.io.StreamCallback
import java.nio.charset.StandardCharsets
def flowFile = session.get()
if(!flowFile) return
flowFile = session.write(flowFile, {inputStream, outputStream ->
inputStream.eachLine { line ->
String[] names = line.split("\\[")
names[0]=(names[0].replace(",", " ")).replace(" ","|")
def a=(names[0]+"["+names[1])
outputStream.write("${a}\n".toString().getBytes(StandardCharsets.UTF_8))
}
} as StreamCallback)
session.transfer(flowFile, REL_SUCCESS)
... View more
Labels:
- Labels:
-
Apache NiFi
10-23-2017
02:29 PM
Yes i have connected hive using DBCPConnectionPool, But is there any way to replace those jars with a lower version one?
... View more
10-23-2017
02:27 PM
Its working using DBCPConnectionPool.
... View more
10-23-2017
02:09 PM
Thanks for the Help. Its working.
... View more
10-11-2017
07:30 AM
Hi I am using NiFi 1.3.0. But facing an issue, it has httpclient-4.5.2.jar in nifi-hive-nar-1.3.0.nar but my hiveserver2 doesnt support that higher version. I need httpclient 4.2.5.jar to connect with hive. So is there any way to replace those jar with lower version? Thanks in advance Pratik
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Hive
-
Apache NiFi
10-06-2017
06:28 PM
Hi,
I am trying to connect Hive from Apache NiFi 1.3.0 but its giving me error SelectHiveQL[id=f0a5404d-015e-1000-9cda-00f1242476aa] SelectHiveQL[id=f0a5404d-015e-1000-9cda-00f1242476aa] failed to process session due to java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient: java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient
SelectHiveQL[id=f0a5404d-015e-1000-9cda-00f1242476aa] org.apache.nifi.processors.hive.SelectHiveQL$Lambda$182/1006763062@50dd1ed1 failed to process due to java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient; rolling back session: java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient Its clearly indicating that its not getting HttpClient.jar. Now i have added httpclient-4.5.2.jar in NiFi lib folder and tried to run NiFi again but NiFi failed to start. If i add any httpclient.jar in Nifi lib folder, nifi is failing to run. Its giving me this error this time in NiFi-app.log 2017-10-06 05:00:00,835 ERROR [Timer-Driven Process Thread-8] org.apache.hive.jdbc.HiveConnection Error
opening session org.apache.thrift.TApplicationException: Required field 'client_protocol' is unset! I think its a error related to version mismatch. Now i tried with httpclient-4.3.3.jar and httpclient-4.2.3.jar, Now for httpclient-4.3.3.jar its giving the same client protocol is unset error.
And For httpclient-4.2.3.jar its giving me this error 2017-10-06 06:00:19,820 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.standard.GetHTTP could not be instantiated
java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.standard.GetHTTP could not be instantiated Now My NiFi Settings: In nifi.properties: nifi.kerberos.krb5.file=/path/krb5.conf In NiFi Lib folder: I have added couple of libs hive-jdbc-1.1.0-cdh5.7.0.jar hive-service-1.1.0-cdh5.7.0.jar libthrift-0.9.2.jar Without this 3 libs nifi was giving other types of errors. In HiveConnectionPool Setting Database Connection Url: jdbc:hive2://MyServer:10000/;principal=hive/MyServer@MyDomain Hive Conf Resources: /path/core-site.xml,/path/hive-site.xml DB User: Password:
Max Wait Time: 500 millis Max Total Connections: 8 Validation query: Kerberos principle: hive/MyServer@MyDomain Kerberos keytab: /path/keytabs/hive.service.keytab for all of these above i am commonly getting a warning SelectHiveQL[id=f0a5404d-015e-1000-9cda-00f1242476aa] Processor Administratively Yielded for 1 sec due to processing failure and an error HiveConnectionPool[id=f0a678e0-015e-1000-1235-94651d8210e9] Kerberos Authentication for Hive failed: org.apache.nifi.util.hive.AuthenticationFailedException: Kerberos Authentication for Hive failed After reading a similar thread https://community.hortonworks.com/questions/62014/nifi-hive-connection-pool-error.html I have added this property to Hive-site.xml . And tried to connect to hive using the "binary" transport mode. <property>
<name>hive.server2.transport.mode</name>
<value>binary</value>
</property> But It still giving me same error SelectHiveQL[id=f0a5404d-015e-1000-9cda-00f1242476aa] SelectHiveQL[id=f0a5404d-015e-1000-9cda-00f1242476aa] failed to process session due to java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient: java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient What should i do now? Any suggestions or Solutions Thanks in advance, Pratik
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi