Member since
03-06-2017
45
Posts
4
Kudos Received
0
Solutions
05-09-2018
07:06 PM
@Matt Thanks for reply. My point behind "Nifi Slow Response" is UI gives slow response in some conditions i.e. 1. When I open configuration model window of any processor. 2. When I click on apply button after configuration. 3. When I am going to open controller services to configure one of my controller. 4. When I reload my Nifi instance. In all the above cases sometimes it works fine and respond me in a second or two but sometime it take 20 sec to 1 min or more then 1 minute. Now I want to know that I have enough memory still I am facing this type of response. And I use my NIFI with KYLO so I wont able to disable my processor, Is there any other way to solve this problem? Max Timer Driven Thread Count is set to 20 Max Cron Driven Thread Count is set to 50. Thanks in advance
... View more
05-09-2018
07:06 PM
@Matt Thanks for reply. My point behind "Nifi Slow Response" is UI gives slow response in some conditions i.e. 1. When I open configuration model window of any processor. 2. When I click on apply button after configuration. 3. When I am going to open controller services to configure one of my controller. In all the above cases sometimes it works fine and respond me in a second or two but sometime it take 20 sec to 1 min or more then 1 minute. Now I want to know that I have enough memory still I am facing this type of response. And I use my NIFI with KYLO so I wont able to disable my processor, Is there any other way to solve this problem? Max Timer Driven Thread Count is set to 20 Max Cron Driven Thread Count is set to 50. Thanks in advance
... View more
05-09-2018
03:36 PM
Thanks Shu for quick response. According to you, I can disable all invalid processor but I won't able to disable all stopped processor as 90% stopped processor in my nifi are useful and I can run it on time to time. And there may be possibilities of increase in no. of processor. When I check memory usage using "system-diagnostics" rest api then it give me below output: {"systemDiagnostics":{"aggregateSnapshot":{"totalNonHeap":"831.29 MB","totalNonHeapBytes":871669760,"usedNonHeap":"796.05 MB","usedNonHeapBytes":834719376,"freeNonHeap":"35.24 MB","freeNonHeapBytes":36950384,"maxNonHeap":"-1 bytes","maxNonHeapBytes":-1,"totalHeap":"19.55 GB","totalHeapBytes":20992491520,"usedHeap":"8.25 GB","usedHeapBytes":8862073056,"freeHeap":"11.3 GB","freeHeapBytes":12130418464,"maxHeap":"19.55 GB","maxHeapBytes":20992491520,"heapUtilization":"42.0%","availableProcessors":8,"processorLoadAverage":10.91,"totalThreads":428,"daemonThreads":201,"uptime":"00:23:16.322","flowFileRepositoryStorageUsage":{"freeSpace":"314.39 GB","totalSpace":"1.43 TB","usedSpace":"1.12 TB","freeSpaceBytes":337573191680,"totalSpaceBytes":1572096000000,"usedSpaceBytes":1234522808320,"utilization":"79.0%"},"contentRepositoryStorageUsage":[{"identifier":"default","freeSpace":"314.39 GB","totalSpace":"1.43 TB","usedSpace":"1.12 TB","freeSpaceBytes":337573191680,"totalSpaceBytes":1572096000000,"usedSpaceBytes":1234522808320,"utilization":"79.0%"}],"provenanceRepositoryStorageUsage":[{"identifier":"default","freeSpace":"314.39 GB","totalSpace":"1.43 TB","usedSpace":"1.12 TB","freeSpaceBytes":337573191680,"totalSpaceBytes":1572096000000,"usedSpaceBytes":1234522808320,"utilization":"79.0%"}],"garbageCollection":[{"name":"G1 Young Generation","collectionCount":1471,"collectionTime":"00:05:30.811","collectionMillis":330811},{"name":"G1 Old Generation","collectionCount":0,"collectionTime":"00:00:00.000","collectionMillis":0}],"statsLastRefreshed":"20:06:22 IST","versionInfo":{"niFiVersion":"1.4.0","javaVendor":"Oracle Corporation","javaVersion":"1.8.0_77","osName":"Linux","osVersion":"3.10.0-514.el7.x86_64","osArchitecture":"amd64","buildTag":"nifi-1.4.0-RC2","buildTimestamp":"12/04/2017 19:55:41 IST"}}}} As you can notice that, I have provided max 20GB memory. So, I think this is enough memory to run nifi smoothly with 15k+ no. of processor. As there is no limit of no. of processor dragged on canvas then what can I don to run nifi smoothly with 15k+ or 20k+ processors. Please let me know as soon as possible. Thanks in advance.
... View more
05-09-2018
07:41 AM
Hello, I am using Nifi with maximum ram of 26 GB and when i run my flows, it uses 33 GB. I have used an API mentioned below, to track the status of all my flows, http://localhost/nifi-api/flow/status Here is my response from the API {"controllerStatus":{"activeThreadCount":3,"queued":"52,687 / 27.18 MB","flowFilesQueued":52687,"bytesQueued":28495782,"runningCount":127,"stoppedCount":9918,"invalidCount":3239,"disabledCount":153,"activeRemotePortCount":0,"inactiveRemotePortCount":0}} After some time it started responding very slow unless i restart it again.I am facing this problem again and again, and i am not able to run my flows properly.
... View more
Labels:
- Labels:
-
Apache NiFi
05-04-2018
06:32 AM
I have setup NCM with 3 nodes. Somehow one node disconnected or failed. Again I started that node But It shown error message that 'Failed to connect node to cluster because local flow is different than cluster flow'. For this I have deleted flow.xml.gz file from the failure node and start it. Then it's work fine. But I don't want to delete flow.xml.gz file. Is there any way that NIFI automatically delete flow.xml.gz file from failure node when I will start failure node in the cluster.
... View more
Labels:
- Labels:
-
Apache NiFi
04-24-2018
12:58 PM
Hello, Thanks for replying soon, It works fine. But I am getting one more problem with input port and remote process group.
I have GetFile processor & RemoteProcessGroup inside Group1 & when I am going to connect both using input ports then I am getting drop down of all inpurt ports which are pl aced in root canvas. But I want to connect with "Group1_Port1" input port that are also placed inside my "Group1" process group. But I am not getting this input port in drop down when I connect GetFile processor with remote process group. Is there any way to connect GetFile processor with RemoteProcessGroup with port that are placed inside Group1 not on root canvas. I have also captured screen shot of my flows, Please check & reply as soon as possible. Thank
... View more
04-24-2018
11:01 AM
1 Kudo
I have setup a NCM with 2 nodes and create a flow using GetFile & PutFile processor inside a process group and configure GetFile that it runs only on primary node. Now for distributing whatever flow file I gets from getfile processor I use remote process group inside my process group using a input port. But when I try to connect GetFile processor to remote process group it gives message that "Does not have any input port".
So, I place my input port on root and then connect again getfile processor to remote process group then it connected successfully. But what when I want to use my input port inside my process group, is there any way to connect getfile processor to remote process group using input port which I use inside my process group. For placing input port on root, I am unable to give upstream connection to PutFile processor. Because my PutFile processor reside inside process group and input port insde root. What can I do for this scenario, Please reply.
... View more
- Tags:
- rpg
03-03-2018
02:39 PM
Hello, I am using Nifi GetHbase Processor, and i want to get the number of records fetched by the GetHbase Processor or any how no of flowfiles it has generated. Is there any way to do this?Please help
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache NiFi
02-28-2018
02:43 PM
Thanks @Matt Burgess.What i am trying to do is i have a custom processor , that inputs the user custom java code and executes it .If the code contains some external jars, then it should be present either in the nifi lib folder or in the NAR i have.But sometimes the custom code uses some external jars which are not present in both places, so i want some way to compile the java code using external jars that are present on a particular path other than nifi lib using some api. and
... View more
02-28-2018
12:05 PM
Hello, I want to compile my java code using some external jars that are not included in the nifi lib folder.Is there any way to define a external path in the nifi, from where along with the nifi lib folder, my java code compiler will look into for the code compilation process?
... View more
Labels:
- Labels:
-
Apache NiFi
01-04-2018
11:07 AM
Hello, I am going to make a template in which I use GetSftp processor, but I want to give path to fetch files dynamically which I get from database. For this I am using nifi expression language by which I set my dynamic value which comes from sql processor in attribute of a flow file. But now problem here is that GetSftp is a starting processor so it won't able to take input stream so I wont able to pass dynamic values to sftp. Please let me know some solution Thanks,
... View more
- Tags:
- processor
Labels:
- Labels:
-
Apache NiFi
09-18-2017
09:21 AM
wait-notify-process-flow.xml Hello...above is my flow which i am currently running .I want to trigger my second flow after a certain condition is met in my first flow, which will start putting all the flowfiles to the wait processor and untill the second processor is completed and when it will complete, i want to resume my first flow.....But i am facing following issues ..... 1. I want to resume my first flow when my whole second flow is completed but notify processor receives a single flow file, it triggers the corresponding wait processor which is not suitable for me. 2. When my whole flow is completed like triggering second flow after flowfiles of first flow are waiting and then resumed execution after completion of my second flow , if i want to run the whole process again the wait processor of the first flow is not holding all the flowfiles in the queue like it did before, but processing it continuously , although my second flow has not even started , as it was supposed to be running after the completion of my second flow. Please find below the diagram of the flow. wait-notify-flow.png Below here the first flow wait-notify-first-flow-pic.png Below is the second flow wait-notify-second-flow-pic.png Here the second flow is in the left most part which is triggered by the wait processor in the left and the notify processor in the bottom will trigger the right most wait processor of the first flow. Here are the configurations of the processors distributed-map-cacheclientservice-config.png
... View more
Labels:
- Labels:
-
Apache NiFi
07-14-2017
04:53 PM
Hello, I am using Nifi 1.3.0 with ExecuteSparkJob Processor and i am getting the following error in processor -- error.png and in the nifi-app.log --> 2017-07-14 21:53:44,231 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=dd071c5c-c28e-3cc0-f9a5-ee5f965deaad] Diagnostics: File does not exist: hdfs://inno1.start.com:8020/user/root/.sparkStaging/application_1500027428726_0007/__spark_conf__8060965695322916343.zip Here is the configuration of the processor sparkjob.png. Please help me with the solution.
... View more
Labels:
- Labels:
-
Apache NiFi
-
Apache Spark
07-11-2017
08:01 AM
Sorry....I attached the wrong code...please find it here nodereadfrommongo.txt
... View more
07-10-2017
03:44 PM
I have created a custom processor and after defining some properties it is showing " not a supported property" error as shown in the picture .Can anybody help me with this?
... View more
Labels:
- Labels:
-
Apache NiFi
06-14-2017
10:48 AM
Hello, I am getting a scenario in my nifi flow in which I will have to keep some data globally in runtime for all processor that contained in my flow, so I can able to get that global data in any processor. Thankyou
... View more
- Tags:
- processor
Labels:
- Labels:
-
Apache NiFi
06-08-2017
02:23 PM
image0.pngimage1.pngHello, I am getting error in my custom processor, second time when I am going to start it is not starting and showing error not eligible component is selected, but I have selected processor and then trying to start. For your reference I have attached two images, in Image1 all processor are stopped and in Image2 I have selected my customprocessor and trying to start but getting message not eligible components are selected. Please let me know where I am doing mistake. Thankyou
... View more
- Tags:
- processor
05-29-2017
03:15 PM
Thanks Timothy I have find out the way to read template's xml file in onTrigger method of my custom processor, but I am getting some issues. I am using getTemplates() method of StandardNiFiServiceFacade class that implements NiFiServiceFacade interface, and I make object of that interface like this -- private NiFiServiceFacade serviceFacade = new StandardNiFiServiceFacade(); in my custom processor's code. But when I am going to up my nifi server, I got error on this line. Below are the error I am getting: java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.hadoop.SparkConnector could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232) ~[na:1.8.0_111]
at java.util.ServiceLoader.access$100(ServiceLoader.java:185) ~[na:1.8.0_111]
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) ~[na:1.8.0_111]
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[na:1.8.0_111]
at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[na:1.8.0_111]
at org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:116) ~[nifi-nar-utils-1.1.2.jar:1.1.2]
at org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:97) ~[nifi-nar-utils-1.1.2.jar:1.1.2]
at org.apache.nifi.NiFi.<init>(NiFi.java:139) ~[nifi-runtime-1.1.2.jar:1.1.2]
at org.apache.nifi.NiFi.main(NiFi.java:262) ~[nifi-runtime-1.1.2.jar:1.1.2] Caused by: java.lang.NoClassDefFoundError: org/apache/nifi/web/revision/RevisionClaim
at org.apache.nifi.processors.hadoop.SparkConnector.<init>(SparkConnector.java:53) ~[nifi-hdfs-processors-1.1.2.jar:1.1.2]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_111]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_111]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_111]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_111]
at java.lang.Class.newInstance(Class.java:442) ~[na:1.8.0_111]
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ~[na:1.8.0_111]
... 6 common frames omitted Caused by: java.lang.ClassNotFoundException: org.apache.nifi.web.revision.RevisionClaim
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_111]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_111]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_111]
... 13 common frames omitted
I wont able to understand what exactly is the problem, as I have already added all required jar files in nar. Still I am getting this error. Please suggest me what I am doing wrong here or tell me some alternate way if you know that how I get all template object in TemplateEntity class in my onTrigger method of customProcessor. Thanks in advance
... View more
05-24-2017
08:17 AM
1 Kudo
Hello, I am going to make a custom processor which needs NIFI template xml file's data(existing saved template) in run time, is this possible in nifi to read existing nifi template file's data in run time. Thanks
... View more
- Tags:
- xml
Labels:
- Labels:
-
Apache NiFi
05-11-2017
03:54 PM
@Thanks Wynner I have successfully setup cluster using 3 nodes. Now I have one query regarding cluster architecture. In my flow I use getSFTP processor to get file and for distributing on cluster I am using Remote Process Group. Now I according to my flow only one primary node can get file and keep on their local content repository untill RPG distribute files on different nodes. so is there any way to make this content repository common between all three nodes because right now if in any case my primary node have low memory then what happen with files.
... View more
04-14-2017
02:52 PM
Hello I am trying to setup the Nifi Cluster....is there any way to setup and run clusters without using zookeeper and with which version of nifi [currently using(1.1.2)]?
... View more
Labels:
- Labels:
-
Apache NiFi
04-06-2017
04:20 PM
Thanks @Bryan Bende , it worked for me. Now i am getting the following error after the line session.commit() in the following code ontriggermethod.txt ------- DBColumnMapper[id=43376208-015b-1000-8176-8a62b92462aa] DBColumnMapper[id=43376208-015b-1000-8176-8a62b92462aa] failed to process due to org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=357d0770-3aea-4def-b607-e93f574593ea,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1491495241588-2, container=default, section=2], offset=10858, length=496],offset=0,name=187456790845166,size=496] transfer relationship not specified; rolling back session: org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=357d0770-3aea-4def-b607-e93f574593ea,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1491495241588-2, container=default, section=2], offset=10858, length=496],offset=0,name=187456790845166,size=496] transfer relationship not specified.... Thanks in Advance.
... View more
04-05-2017
11:05 AM
Hello...I am trying to convert my CSV file to get an AvroSchema by using my own custom processor which will get the file from the disk and converts it into flowfile and then passes it to the method named "inferAvroSchemaFromCSV" of the processor inferAvroSchema.After that i am getting my required schema successfully, which i am trying to transfer it into the success relationship so as to get on the disk, but i am getting following error - "DBColumnMapper[id=3836e9a0-015b-1000-9d9b-3d14d4066516] Failed to infer Avro schema for StandardFlowFileRecord[uuid=785128f9-acea-4d1c-a3dc-ec1313c900a0,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1491380551546-1, container=default, section=1], offset=15934, length=797],offset=0,name=vikas,size=797] due to org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=785128f9-acea-4d1c-a3dc-ec1313c900a0,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1491380551546-1, container=default, section=1], offset=15934, length=797],offset=0,name=vikas,size=797] is not the most recent version of this FlowFile within this session (StandardProcessSession[id=12]): org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=785128f9-acea-4d1c-a3dc-ec1313c900a0,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1491380551546-1, container=default, section=1], offset=15934, length=797],offset=0,name=vikas,size=797] is not the most recent version of this FlowFile within this session (StandardProcessSession[id=12])
14:20:53 ISTERROR3836e9a0-015b-1000-9d9b-3d14d4066516" Following is my workflow template.png and my configuration of custom processor dbcolumnmapper-configuration.png Here is the Java Code-- @Override
public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
URI uri=null;
String url=context.getProperty(INPUT_SCHEMA).getValue();
StringBuilder out = new StringBuilder();
try {
uri = new URI(url);
} catch (URISyntaxException e) {
e.printStackTrace();
}
Path schemaPath = new Path(uri);
FileSystem fs;
InputStream in=null;
try {
fs = schemaPath.getFileSystem(DefaultConfiguration.get());
in= fs.open(schemaPath);
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
String line;
while ((line = reader.readLine()) != null) {
out.append(line);
}
System.out.println("csv file data"+out.toString()); //Prints the string content read from input stream
reader.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
FlowFile flowFile=null;
try{
//Writing csv data inside flow file for infer avro schema converter
final byte[] dataByte=String.valueOf(out).getBytes(StandardCharsets.UTF_8);
flowFile = session.create();
flowFile = session.write(flowFile, new OutputStreamCallback() {
@Override
public void process(final OutputStream output) throws IOException {
output.write(dataByte);
}
});
//flowFile = session.putAttribute(flowFile, CoreAttributes.MIME_TYPE.key(), "text/plain");
//flowFile = session.putAttribute(flowFile, CoreAttributes.FILENAME.key(), "vikas");
final AtomicReference<String> avroSchema = new AtomicReference<>();
avroSchema.set(inferAvroSchemaFromCSV(flowFile, context, session));
FlowFile avroSchemaFF = null;
avroSchemaFF = session.write(session.create(), new OutputStreamCallback() {
@Override
public void process(OutputStream out) throws IOException {
out.write(avroSchema.get().getBytes());
}
});
avroSchemaFF = session.putAttribute(avroSchemaFF, CoreAttributes.MIME_TYPE.key(), AVRO_MIME_TYPE);
avroSchemaFF = session.putAttribute(avroSchemaFF, CoreAttributes.FILENAME.key(), "newFlowFile");
session.transfer(avroSchemaFF, REL_ORIGINAL);
session.transfer(avroSchemaFF, SUCCESS);
//session.transfer(avroSchemaFF, FAILURE);
session.commit();
}catch (Exception ex) {
getLogger().error("Failed to infer Avro schema for {} due to {}", new Object[] {flowFile, ex});
}
}
... View more
Labels:
- Labels:
-
Apache NiFi
03-18-2017
11:29 AM
I have a data-file which have say 10 records with primary keys 1-10 and i have some data in my database,say 5 records with primary keys 1-5. I want to insert that file data into my database which will update my 5 records which is having same primary key and insert the remaining 5 records, a total of 10 records in database.Here is my workflow insertupdateworkflow.xml , insertupdateworkflow.png,in this , i am first trying to update the records and redirecting the failed records to another jsontosql processor with insert statement so as to insert the remaining five records in the database.But the putsql is giving error "Index array out of bounds" and not inserting the data but is updating the existing data correctly.Please help me out with this.
... View more
Labels:
- Labels:
-
Apache NiFi
03-15-2017
08:31 AM
I have data in my xls files and i want to process it in nifi but i am unable to find any processor which would help me with this. Is there any solution for it?
... View more
Labels:
- Labels:
-
Apache NiFi
03-15-2017
08:11 AM
Hi....I have data in various kinds of files and i want to insert them into database but i don't want it to convert them first into JSON and then to SQL as it is very time consuming.Is there any way so that i can put my data into database from different kinds of files like CSV,Avro etc directly?
... View more
Labels:
- Labels:
-
Apache NiFi
03-09-2017
12:06 PM
Hi.. I am using a flow which is like GetFile--->SplitText---->ExtractText---->RouteOnAttribute----->PutFile. In this I have defined certain conditions in extract text as mentioned in the file extracttext.png. It is working fine but i want that after filtering the text from extracttext processor i want to make separate directories depends on the condition like i want for condition1 i want to put it in Folder1, for condition2 i want it on Folder2 and Folder3 for unmatched files.For that i used RouteOnAttribute with following properties as shown in file routeonattribute.png. But it is not working as i am not getting any file in the folders.Can you please tell me what am i doing wrong in it. Here is my flow after routeOnAttribute routingflowbyrouteonattribute.png.
... View more
03-07-2017
02:30 PM
Thanks alot @Matt Burgess .It finally worked for me. But can we directly convert the column names coming from the CSV or any other files including JSON using replace text processor only ? so it would not take so much processing time, like X-A,Y-B,Z-C,W-D and insert directly into the database.
... View more
03-07-2017
10:35 AM
Thanks @Matt Burgess for the help, I did what you mention but i am getting null after the transformation.Can you please tell me what am i doing wrong.jolttransformationjson.png
... View more
03-06-2017
08:27 AM
I also have the same query as want the excel data to be inserted into my database but unable to find one.
... View more