Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Can we put data into HDFS from the nifi which in not part of any cluster

Solved Go to solution

Can we put data into HDFS from the nifi which in not part of any cluster

New Contributor

Can we put data into HDFS from the nifi which in not part of any cluster


I have HDP-Ambari with 4 node cluster, and I have installed nifi as standalone server in the master node machine


Can get the data from my local machine and store them in HDFS ?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Can we put data into HDFS from the nifi which in not part of any cluster

Mentor

@Naveenraj Devadoss

You need to copy the core-site.xml and hdfs-site.xml from your HDP cluster to the machine where NiFi is running. Then configure PutHDFS so that the configuration resources are "/path/to/core-site.xml,/path/to/hdfs-site.xml". That is all that is required from the NiFi perspective, those files contain all of the information it needs to connect to the Hadoop cluster.

You'll also need to ensure that the machine where NiFi is running has network access to all of the machines in your Hadoop cluster. You can look through those config files and find any hostnames and IP addresses and make sure they can be accessed from the machine where NiFi is running.

HTH

5 REPLIES 5

Re: Can we put data into HDFS from the nifi which in not part of any cluster

Mentor

@Naveenraj Devadoss

You need to copy the core-site.xml and hdfs-site.xml from your HDP cluster to the machine where NiFi is running. Then configure PutHDFS so that the configuration resources are "/path/to/core-site.xml,/path/to/hdfs-site.xml". That is all that is required from the NiFi perspective, those files contain all of the information it needs to connect to the Hadoop cluster.

You'll also need to ensure that the machine where NiFi is running has network access to all of the machines in your Hadoop cluster. You can look through those config files and find any hostnames and IP addresses and make sure they can be accessed from the machine where NiFi is running.

HTH

Re: Can we put data into HDFS from the nifi which in not part of any cluster

New Contributor

Hi Geoffrey,


Thanks , I have placed the same as you suggested , (PFB snapshot)

But still the data is not moving from local to HDFS


107965-1555398193047.png

Error Log :

2019-04-16 02:43:14,417 ERROR [Timer-Driven Process Thread-7] org.apache.nifi.util.ReflectionUtils Failed while invoking annotated method 'public final void org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.abstractOnStopped()' with arguments '[]'.

java.lang.reflect.InvocationTargetException: null

at sun.reflect.GeneratedMethodAccessor379.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)

at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)

at org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotations(ReflectionUtils.java:268)

at org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotation(ReflectionUtils.java:90)

at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$4(StandardProcessorNode.java:1547)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.NullPointerException: null

at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.abstractOnStopped(AbstractHadoopProcessor.java:286)

... 14 common frames omitted

Re: Can we put data into HDFS from the nifi which in not part of any cluster

Mentor

@Naveenraj Devadoss

Did you remember this part?

"You'll also need to ensure that the machine where NiFi is running has network access to all of the machines in your Hadoop cluster."

Please revert


Re: Can we put data into HDFS from the nifi which in not part of any cluster

New Contributor

Working fine now , thanks

Re: Can we put data into HDFS from the nifi which in not part of any cluster

Mentor

@Naveenraj Devadoss

What was the solution? Did you update the network config in /etc/host?