About Anishkumarv

Anishkumarv · ‎12-01-2019

Hi Rehman, May i know how much blocks you have in your cluster ? , it's look like issue with your heap size for namenode service. Please refer the below namenode heap calculation and configure based on that. One namenode object uses about 150 bytes to store metadata information. Assume a 128 MB block size - you should increase the block size if you have lot of data (PB scale or even 500+ TB in some cases). Assume a file size 150 MB. The file will be split in two blocks. First block with 128 MB and second block with 22MB. For this file following information will be stored by Namenode. 1 file inode and 2 blocks. That is 3 namenode objects. They will take about 450 bytes on namenode. For example, at 1MB block size, in this case we will have 150 file blocks. We will have one inode and 150 blocks information in namenode. This means 151 namenode objects for same data. 151 x 150 bytes = 22650 bytes. Even worse would be to have 150 files with 1MB each. That would require 150 inodes and 150 blocks = 300 x 150 bytes = 45000 bytes. See how this all changes. That's why we don't recommend small files for Hadoop. Now assuming 128 MB file blocks, on average 1GB of memory is required for 1 million blocks. Now let's do this calculation at PB scale. Assume 6000 TB of data. That's a lot. Imagine 30 TB capacity for each node. This will require 200 nodes. At 128 MB block size, and replication factor of 3. Cluster capacity in MB = 30 x 1000 (convert to GB) x 1000 (convert to MB) x 200 nodes = 6 000000000 MB (6000 TB) How many blocks can we store in this cluster? 6 000 000 000 MB/128 MB = 46875000 (that's 47 million blocks) Assume 1 GB of memory required per million blocks, you need a mere 46875000 blocks / 1000000 blocks per GB = 46 GB of memory. Namenodes with 64-128 GB memory are quite common.

Anishkumarv · ‎04-06-2019

Can you share further more details to help further. Cloudera bundle version zookeeper version. Complete logs to cross check the error messages. With Regards Anishkumar Valsalam

Anishkumarv · ‎04-06-2018

@Matt Clarke Can you explain.. how we can ensure the remaining threads was using by other components Ex :controller services / Reporting tasks. If you explain that really helpful to close this thread.

Anishkumarv · ‎04-05-2018

@Matt Clarke How we can ensure that ? the remaining threads was using by other Ex :controller services / Reporting tasks.

Anishkumarv · ‎04-05-2018

@Matt Clarke Recently we didn't do any update and we have tried with restart also but still we are seeing differences in the thread values in status bar. What actually we configured. What actually running. How this drastic differences ? pls help to close this.

Anishkumarv · ‎04-05-2018

We are running only two processor group and the overall usage you can see in the 2 PG ( 8 + 4). Hdf - 3.0.1.0-43 Maximum Timer Driven Thread Count = 100 Max Event Driven Thread Count = 5 we have two nodes but it not configured as cluster.

Anishkumarv · ‎04-05-2018

We are running only two processor group and the overall usage you can see in the 2 PG ( 8 + 4). Hdf - 3.0.1.0-43 Maximum Timer Driven Thread Count = 100 Max Event Driven Thread Count = 5 we have two nodes but it not configured as cluster.

Anishkumarv · ‎04-04-2018

@Bryan Bende experts inputs required.

Anishkumarv · ‎04-04-2018

@Matt Clarke Hi Bro need your inputs.

Anishkumarv · ‎04-04-2018

Hi Team, How to calculate Nifi threads. ? because in the screenshot it showing totally 68 threads its using but the actual used is only 12 so how to find the remaining threads details ?? Kinldy help on this.

Anishkumarv · ‎03-27-2018

@Matt Clarke Hi Matt, We are having an internal argument on whether its a good idea to keep VMS for production NIFI need your inputs on this What are the pros and cons of each hardware configuration?

Anishkumarv · ‎03-27-2018

Hi Team, Why Virtual Machine is not advisable for Nifi Production environment ?

Anishkumarv · ‎03-27-2018

Hi Team, Why Virtual Machine is not advisable for Nifi Production environment ?

Anishkumarv · ‎02-22-2018

@Matt Clarke Need your inputs bro.

Anishkumarv · ‎02-22-2018

Hi Team, We are planning to move HDF nodes from some non hdf nifi instances befor moving we just want to calculate the current utilzation of utilization of Nifi (non HDF). for better planning. How to get that stats for example for last 1day nifi how much mem/cores used?

Anishkumarv · ‎11-25-2017

@Bryan Bende @Matt Clarke Kindly help on the above issue.

Anishkumarv · ‎11-24-2017

Hi Team, Kindly share us the simple steps to install “Hadoop clients” on the HDF Cluster. Meanwhile, there are many applications which requires Hadoop clients that are deployed in HDF cluster and hence we are synchronizing those clients manually. The main drawback is that there may be missing of jars as part of manual synchronization. Hence, request you to share the steps for synchronization With Regards Anishkumar Valsalam

Anishkumarv · ‎11-09-2017

Hi team, Please advice on the above issue it will helpful for others too.

Anishkumarv · ‎11-08-2017

@Matt Clarke hi matt can you guide me to resolve this issue.

Anishkumarv · ‎11-08-2017

Hi Team, Is there any option to allocate resourcces for nifi processor group? see in the below image we have allocated "Maximum Timer Driven Thread Count " 30cores all the 30 cores took the below processor itself becuase of that other PG not processing. is there any option available allocate resource for PG or set priority ? in nifi. Please guide me to close this issue.

Anishkumarv · ‎11-01-2017

Hi Abdelkrim, Thanks its working now The same we can do in standalone as well right? in rpg also it taking from one node? how we can process from multiple nodes and balance the load. Why we need to put input port in outside of processor group? root path.

Anishkumarv · ‎11-01-2017

@Abdelkrim Hadjidj Thanks if the new file comes it processing and it fetch to hdfs but i am getting the below error also in Fetchsftp. 2017-11-02 01:56:41,942 ERROR [Timer-Driven Process Thread-5] o.a.nifi.processors.standard.FetchSFTP FetchSFTP[id=788c7b35-015f-1000-0000-00000f3979fd] Failed to fetch content for StandardFlowFileRecord[uuid=af918b98-5632-4877-9876-221477204e73,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1509553663114-11, container=default, section=11], offset=677691, length=0],offset=0,name=a.txt,size=0] from filename /home/1548691/*.txt on remote host 10.20.174.137:22 due to java.io.IOException: Failed to obtain file content for /home/1548691/*.txt; routing to failure: java.io.IOException: Failed to obtain file content for /home/1548691/*.txt 2017-11-02 01:56:41,943 ERROR [Timer-Driven Process Thread-5] o.a.nifi.processors.standard.FetchSFTP java.io.IOException: Failed to obtain file content for /home/1548691/*.txt at org.apache.nifi.processors.standard.util.SFTPTransfer.getInputStream(SFTPTransfer.java:300) ~[nifi-standard-processors-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2] at org.apache.nifi.processors.standard.FetchFileTransfer.onTrigger(FetchFileTransfer.java:236) ~[nifi-standard-processors-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_60] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_60] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_60] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_60] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_60] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] Caused by: com.jcraft.jsch.SftpException: /home/1548691/*.txt is not unique: [/home/1548691/b.txt, /home/1548691/HKLPATHAS03.txt, /home/1548691/a.txt] at com.jcraft.jsch.ChannelSftp.isUnique(ChannelSftp.java:2965) ~[jsch-0.1.54.jar:na] at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:1314) ~[jsch-0.1.54.jar:na] at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:1290) ~[jsch-0.1.54.jar:na] at org.apache.nifi.processors.standard.util.SFTPTransfer.getInputStream(SFTPTransfer.java:292) ~[nifi-standard-processors-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2] ... 13 common frames omitted @ how to process all set of files instead of .txt and it picking from only one node? how to pick from all 3 nodes ?

Anishkumarv · ‎11-01-2017

yes i have data in that. [root@HKLPATHAS03 1548691]# ll total 4 -rw-r--r-- 1 1548691 root 0 Nov 2 00:38 HKLPATHAS03.txt drwxr-xr-x 9 1548691 domain users 4096 Nov 2 00:27 nifi-toolkit-1.1.2 see it not taking that file to hdfs. [root@HKLPATHAS03 1548691]# sudo -u hdfs hadoop fs -ls /user/1548691/HKLPATHAS03.txt ls: `/user/1548691/HKLPATHAS03.txt': No such file or directory [root@HKLPATHAS03 1548691]#

Anishkumarv · ‎11-01-2017

Hi Team, I am trying to configure the RPG in my cluster environemt (3nodes) how to distribute the data between the 3 nodes. This the below processor i am trying with RPG but it not processing at all attached the details FYR. Kindly help me to close this thread.

Anishkumarv · ‎11-01-2017

@Abdelkrim Hadjidj Thanks i am able to create policy now , I was checked into default flow policy so its was not enabled. In Input port policy it is enabled And In your flow i can see this is between cluster to cluster. https://community.hortonworks.com/articles/88473/site-to-site-communication-between-secured-https-a.html How we can test the same between 3 instances ( 3 node cluster) ? I am looking for the sample flows to test RPG in the cluster. Thanks Matt and Abdelkrim.

Anishkumarv · ‎10-31-2017

Hi @Matt Clarke @Abdelkrim Hadjidj Thanks for the detailed note but stll the below 2 options are not enabled eventhough we careated polocies (Retrieve Site-To-Site Details) for nodes. -> receive data via site-to-site --> send data via site-to-site And please guide now in standalone cluster we are facing some load(mem / cpu) , In cluster do we face this kind of issue? or it will distribute the data and process so that the above mem/cpur load issue we can avoid. Is my understanding correct?

Anishkumarv · ‎10-21-2017

@Bryan Bende @Pierre Villard Need your advice on this issue.

Anishkumarv · ‎10-21-2017

@Matt Clarke need your help on this.

Anishkumarv · ‎10-21-2017

Hi Team, After upgrade to HDF 3.0.1 nifi not starting and getting below error while starting.kinldy helpt to fix this error. /loginIdentityProviders> 2017/10/22 01:42:07 ERROR [main] org.apache.nifi.properties.ConfigEncryptionTool: Encountered an error javax.crypto.BadPaddingException: pad block corrupted at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher$BufferedGenericBlockCipher.doFinal(Unknown Source) at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineDoFinal(Unknown Source) at javax.crypto.Cipher.doFinal(Cipher.java:2165) at javax.crypto.Cipher$doFinal$2.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) at org.apache.nifi.properties.ConfigEncryptionTool.decryptFlowElement(ConfigEncryptionTool.groovy:541) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:384) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:69) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:190) at org.apache.nifi.properties.ConfigEncryptionTool$_migrateFlowXmlContent_closure4.doCall(ConfigEncryptionTool.groovy:636) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019) at groovy.lang.Closure.call(Closure.java:426) at groovy.lang.Closure.call(Closure.java:442) at org.codehaus.groovy.runtime.StringGroovyMethods.getReplacement(StringGroovyMethods.java:1543) at org.codehaus.groovy.runtime.StringGroovyMethods.replaceAll(StringGroovyMethods.java:2580) at org.codehaus.groovy.runtime.StringGroovyMethods.replaceAll(StringGroovyMethods.java:2506) at org.codehaus.groovy.runtime.dgm$1127.invoke(Unknown Source) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133) at org.apache.nifi.properties.ConfigEncryptionTool.migrateFlowXmlContent(ConfigEncryptionTool.groovy:635) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210) at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.apache.nifi.properties.ConfigEncryptionTool.main(ConfigEncryptionTool.groovy:1184) pad block corrupted

Anishkumarv · ‎10-13-2017

@Matt Clarke Thanks Bro you are my saviour 🙂

Online	Offline
Last Visited	‎05-08-2020 03:33 AM

Date Registered	‎04-06-2019 02:41 AM
Last Visited	‎05-08-2020 03:33 AM
Total Messages Posted	125
Total Kudos Received	2

Re: Nifi Without Embeded Zookeeper

Re: NameNode service did not started successfully

Re: Zookeeper update

Re: How to calculate Nifi threads.

Re: How to calculate Nifi threads.

Re: How to calculate Nifi threads.

Re: How to calculate Nifi threads.

Re: How to calculate Nifi threads.

Re: How to calculate Nifi threads.

Re: How to calculate Nifi threads.

How to calculate Nifi threads.

Re: VM for Nifi Prod

VM for Nifi Prod

VM for Nifi prod

Re: Nifi - capacity planning - moitoring

Nifi - capacity planning - moitoring

Re: Hadoop Clients - HDF Cluster

Hadoop Clients - HDF Cluster

Re: Resource Allocation - Nifi Processor Group

Re: Resource Allocation - Nifi Processor Group

Resource Allocation - Nifi Processor Group

Re: RPG - Cluster not processing

Re: RPG - Cluster not processing

Re: RPG - Cluster not processing

RPG - Cluster not processing

Re: Nifi - RPG - does not have any input port erro...

Re: Nifi - RPG - does not have any input port erro...

Re: Nifi after upgrade not starting - getting errr...

Re: Nifi after upgrade not starting - getting errr...

Nifi after upgrade not starting - getting errror "...

Re: HDF Nifi not starting