About srijitachaturve

srijitachaturve · ‎05-30-2019

Hi @Andrew Lim , Thanks for a detailed explaination. Following your article i am trying to convert a csv to json using convertrecord processor and then load the merged json (output of convertrecord) to redshift using copy from a file.my merged json is stored in s3.I am getting error that csv is not in json format, could you please suggest how to load these records all at once to redshift?

srijitachaturve · ‎04-10-2019

This doesn't works for me, i places a flow.xml.gz from dev to prd cluster,cleared all repsotiories of prod but still i see state in processors.Could you please suggest other way to clear state for all processors at one go ? i tried deleting state folder contents under /nifi/conf but that too dint help,it gave me some error.

srijitachaturve · ‎04-08-2019

@Matt Burgess your comment on max fragment size feature if querydb processor. is this applicable to max rows fetch size too, i have set this in my flow and facing data loss issues. few records are missed by the processor working on 3 node cluster.

srijitachaturve · ‎04-08-2019

Hi @mattburgess i am using the same processor for fetching incremental data from relational tables.i have given max rows fetch size as 500 and max value column as a timestamp. Is fetching data in batch can lead to data loss, as i have seen few records of some timestamp are not being fetched when doing incremental run but are fetched when i clear state and run full load? want to understand working of max rows feature. read your comment regarding max fragment setting on this blog https://community.hortonworks.com/questions/178505/querydatabasetable-processor-shutting-down.html , is the same applicable for max row fetch size too?Please suggest

srijitachaturve · ‎04-03-2019

@Matt Clarke @Matt Burgess Hi All, I am using querydatabase processor in incremental and primary node execution mode to fetch records from source.But the processor is fetching duplicate records sometimes, is it because the state of the last fetched records is not propogated between the same processor in cluster by zookeepr and the other querydb processor fetches the same records again?(i have read that processors running on prmary node share states) so is this because of lag in reading state ferom zoookeeper.please suggest!! or should i not use querydb in cluster mode and go with generatetablefetch as suggested in this post https://community.hortonworks.com/questions/203372/querydatabase-to-run-in-distributed-manner.html

srijitachaturve · ‎04-03-2019

Thanks Matt for your view on this,the ask is to generate a batchid which should be a sequence number, so whenever querydb processor fetches records from source db (sqlserver) a batchid should be added to the flowfile so that all records have a same batchid when loaded to target table,this will help in auditing of records.but here in cluster mode it seems difficult to achieve this using updateattribute processor.i liked your idea of appending node hostname with the sequence but if i could generate atomic values across all nodes it would be much better.

srijitachaturve · ‎04-03-2019

Thanks David, Idea looks good ,I will try this.

srijitachaturve · ‎03-22-2019

@shu,@Mattclarke,@markpayne How do i generate the sequence number to be used as a stored value as you suggested.As per my knowledge there is only one processor in nifi to generate sequence number and that is update attribute which in cluster mode will again produce different values across all nodes.

srijitachaturve · ‎03-20-2019

Hi All, @mattclarke,@mattburgess,@markpayne I want to generate sequence number in my nifi cluster (3 nodes), I was using update attribute processor with store state locally option , but this is not serving my purpose as each node is generating its own value incrementally and this is creating duplicate values while loading data to target table.I would be grateful if i can get alternate solution to achieve this batchid generation in cluster mode. Thanks in advance!!

srijitachaturve · ‎03-20-2019

@Matt Clarke,@matt burgess Exactly the second point is happening, each node is generating its own value incrementing from last value it has stored in its local state. So which processor or method should i use to generate an incremental batchid (batch1,batch2...so on) since update attribute is messing values when running on cluster. or is there any property by which updateattribute processors on all nodes can pickup each others's last state variable?..please suggest

srijitachaturve · ‎03-20-2019

@Shu will this work even if we have some state alraedy stored withing the processor? for eg: i have a timestamp (2019-03-17 02:00:00:0) stored in the state of my processor now i want the processor to start fetching data after 2019-03-20, will this property help in such scenario?

srijitachaturve · ‎03-20-2019

@mattburgess,@markpayne Hi All, I am using stateful variables to generate an incremental batchid value using update attribute processor,this runs in cluster and have set the processor to run on all nodes. But the batchid values generated are not in an incremental fashion,the processor is missing some values sometime or generating a duplicate value.Is it happening due to restart of the cluster and wiping off of stateful variables data? could you please suggest how can i persist stateful variables data? attached update attribute configuration for reference. Please s uggest!!

srijitachaturve · ‎01-03-2019

I uploaded 100 templates by copying xmls/templates in a folder named .conf/templates as defined in nifi.properties file and when restarted nifi it appeared in nifi UI.but now when i deleted templates from the .conf/templates its still visible in UI,I want to delete all those UI templates all at once as deleting one by one from UI is very tedious. Any suggestion to do this? @matt burgess,@mark payne @Matt Burgess

srijitachaturve · ‎09-14-2018

Thanks a lot Matt for such a detailed and outstanding explanation.Does this mean that 4 concurrent tasks and run duration of 50 ms will make the processor to occupy cpu time for 200 ms that is tasks multiplied by run duration? Thanks, Sri

srijitachaturve · ‎09-12-2018

Hi Team, @Matt Clarke,@matt burgess I am using PUTSQL processor to execute copy command in redshift database.This processor usually processes more than 20 flowfiles which are merged files containing json objects.I have configured it this way:(please see attached image). I want to know have i configured run-duration and run schedule correctly? My nifi canvas has more than 2000 processor and i am facing ui slowness issues.Please suggest how to decide these configurations in order to get high throughput and less resource consumption. Thanks in advance!! Sri

srijitachaturve · ‎09-12-2018

you can go through this link: https://docs.hortonworks.com/HDPDocuments/HDF1/HDF-1.2/bk_UserGuide/content/scheduling-tab.html

srijitachaturve · ‎09-10-2018

I too faced the same issue of not receiving emails,although configuration was set as mentioned in the above article,then i added this property in botstrap-notification.xml and it worked for me. since my smtp didnt require any username and password authentication, i deleted those properties and added this one, <property name="SMTP Auth">false</property> (note this needs to be added additionally its not already there in default xml, need to false it as its default value is true (reference:https://github.com/apache/nifi/blob/master/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/notification/email/EmailNotificationService.java) please vote up if my answer helped.

srijitachaturve · ‎09-05-2018

Thanks for your reply @Matt Burgess,understood the concept. Actually i had a puthbase processor which had 15 flowfiles in failure and since the processor keeps retrying huge number of bulletins were generated and i guess this was the reason the json from sitetositebulletinreportingtask was not generated.later on when i minimized the number of errors by stopping the processor i was able to see json. Is my assumption correct? if this was the reason the of delay response from reportingtask, any solution you could suggest? as in production we may get a large number of bulletins. or is something wrong at with my nifi instance? here is my sitetositebulletinreportingtaskreportingtask configuration:

srijitachaturve · ‎09-04-2018

Hi Team, @matt clark,@matt burgess I want to use site2sitebulletin reporting task for collecting all bulletins on nifi instance and then send them to some monitoring flow for logging or sending emails to admin. But i found that only global bulletins (one shown in right corner of application) are captured using site2sitebulletin reporting task not bulletins which are seen on processors. for eg: when i get an error on getsftp processor it was captured by the reporting task but errors of table not found in puthbase processor were not captured. How to capture all bulletins at once ? I want to do this for implementing error handling and notifying mechanism,i checked this way https://community.hortonworks.com/questions/77336/nifi-best-practices-for-error-handling.html but as my flow alraedy has too many processors i am looking for a way to not use retry check loop in order to minimize count of processors. Can all bulletins be captured using sitetositebulletin reportingtask or should i implement retrycheckloop to capture processor specific failure events? Please suggest!! Thanks in advance.

srijitachaturve · ‎08-29-2018

Thanks a lot for your answer,helped me. also I had to enable allow less secure apps setting in my gmail.

srijitachaturve · ‎08-28-2018

Hi Everyone, @balalaika,@wynner Any solution for the above mentioned error? I too tried with ports 25,465 & 587 as mentioned in below comments, i am able to telnet smtp.gmail.com, but still getting connection refused error.Any help is highly appreciated. Thanks in advance!!

srijitachaturve · ‎04-18-2018

Thanks Matt for your valuable answer, i will look for all these points in my system.

srijitachaturve · ‎04-17-2018

Hi Matt, I followed all the steps mentioned in this article to improve performance of my nifi -1.1.1 standalone instance. I have a lot of processors about 2385 stopped, 2524 disabled & 115 Running.Please have a look at below link for more details. https://community.hortonworks.com/questions/185773/nifi-ui-working-very-slow-how-to-increase-performa.html?childToView=184781#answer-184781 I made changes to bootstrap.conf file,added 7,8,9 parameters for Java 8 and all other changes mentioned in your article, but this didn't helped to increase nifi processing speed and when i started nifi flowfiles were stucked and nothing flowed through. Then when i commented 7,8,9 parameters , processing started but still its very slow. Any suggestions why it didn't helped?

srijitachaturve · ‎04-12-2018

Also while creating a template from an existing flow which contains controller services assigned to processors, why are controller services replicated in the new template created and why new disabled services are assigned to the processors.This is very troublesome as again we need to go to each processor and delete the extra/disabled ones and reassign the previously assigned enabled one. Any Suggestion how to get rid of extra controller services being created this way?

srijitachaturve · ‎04-12-2018

Hi All, @Matt clarke,@Matt Burgess,@Mark Payne I am using Nifi -1.1.1, and have process-group which contains 385 processors in disabled state. when i am trying to create a template for that process group, a large number of processors are getting truncated or left and just a few like 14 are picked in the new template created, also the disabled processors have now become in stopped ones and a few became un-configured. Also when i import the newly created template the flow gets distorted and processors overlap on each other. Is this a bug or Nifi's behavior for disabled processor? Please find screenshots attached. (Screenshot1 shows how 345 processors reduced to 14 & 3 while creating the template of existing flow; Screenshot_original_flow shows how flow looked before; Screenshot_distorted shows: how flows got overlapped after importing the template) Please help!!!

srijitachaturve · ‎04-11-2018

Thanks for the solution, but since i am not familiar with rest api, solution by Matt looks easy to me. Will surely try yours one too.

srijitachaturve · ‎04-10-2018

Hi Matt, Thanks a lot for a wonderful article, i was looking for this since a long time. I have a question, How can we disable or enable multiple processors or a complete process group all at once? As i have many process groups on my canvas which contains too many processors and enabling or disabling them one at a time would be very cumbersome. Please suggest!! Thanks, Sri

srijitachaturve · ‎04-09-2018

Thanks a lot Matt for your valuable inputs on this. I was looking for the solution on this since a long time. I will definitely disable the unused processors and check the performance. Thanks again 🙂

srijitachaturve · ‎04-09-2018

Hi All, We have Nifi-1.1.1 installed on hdp 2.6.3, standalone instance. The UI has become very slow, it takes more than a minute to move between process-groups, also many a times while entering into a process-group the cursor returns back to the Nifi-Flow root window. Also CPU Usage goes to 500 % and memory usage to 10 percent when only 115 processors are running and rest (2k) processors are stopped. Could you please suggest some settings to improve the performance of UI : I have applied all settings in nifi.properties & bootstrap.conf files according to this article by @Matt Clarke: https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html Server Disc Space Details: total used free shared buffers cached Mem: 120G 73G 46G 40G 488M 43G -/+ buffers/cache: 29G 90G Swap: 511M 0B 511M CPU Usage Details: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 118806 root 20 0 19.7g 10g 103m S 455.0 9.0 559:31.81 java Canvas Details: (Is processor crowding can be issue for slowness?) 20385 Processors Stopped 2524 Disabled 115 Running @Matt Burgess,@Mark Payne,@Matt Clarke Please suggest solutions to improve nifi performance. Thanks in advance!!!!

srijitachaturve · ‎01-02-2018

Thanks @Manchun Kumar , but i am unable to find post for this in the link you provided. could you please share the particualr link for ssl/TLS in Nifi-1.1.1 or above

Online	Offline
Last Visited	‎08-08-2019 05:19 AM

Date Registered	‎07-12-2017 06:19 PM
Last Visited	‎08-08-2019 05:19 AM
Total Messages Posted	53
Total Kudos Received	3

Re: Convert CSV to JSON, Avro, XML using ConvertRe...

Re: Nifi clear state

Re: QueryDatabaseTable processor shutting down

Re: Incremental Fetch in NiFi with QueryDatabaseTa...

querydatabase processor fetches duplicate records ...

Re: how to generate sequence number in nifi cluste...

Re: how to generate sequence number in nifi cluste...

Re: how to generate sequence number in nifi cluste...

how to generate sequence number in nifi cluster mo...

Re: Are values of stateful variables in nifi clust...

Re: How to set "Initial Max Value" for QueryDataba...

Are values of stateful variables in nifi cluster g...

How to delete Templates visible in Nifi UI from ba...

Re: What should be Ideal Run-duration and Run sche...

What should be Ideal Run-duration and Run schedule...

Re: Run Duration

Re: HDF-2.0 NiFi Notification Services

Re: want to use site2sitebulletin reporting task f...

want to use site2sitebulletin reporting task for c...

Re: Nifi PutEmail failure to send flowfile to gmai...

Re: Nifi PutEmail failure to send flowfile to gmai...

Re: HDF/NIFI Best practices for setting up a high ...

Re: HDF/NIFI Best practices for setting up a high ...

Re: Nifi not picking all the processors while crea...

Nifi not picking all the processors while creating...

Re: HDF/NiFi Improving the performance of your UI

Re: HDF/NiFi Improving the performance of your UI

Re: Nifi UI Working very Slow, How to increase per...

Nifi UI Working very Slow, How to increase perform...

Re: Does Nifi use One-way or two-way SSL authentic...