HDFS balancer with small files

dzbeda · ‎10-16-2021

Hi, I have an HDFS cluster with 3 nodes. The cluster holds lots of small files (KB) and I have reached Millions of blocks per node. I have added 4 more new servers to the cluster and started the balancer process but it looks that it does not do much. - The goal is to reduce the Million of blocks per server In order to balance the small-size files should i change the value of the following parameter to support moving files from 1KB size? Ddfs.balancer.getBlocks.min-block-size=1048

dzbeda · ‎04-17-2021

Hi, I would like to monitor NIFI with Prometheus -Mainly to understand if Queue is full and the performance of specific processors. Is there an exporter for that? Grafan dashboards? Thanks

dzbeda · ‎03-24-2021

is it possible to get records from Splunk based on a specific index? Where can I define the different indexes I want to quey? Thanks

dzbeda · ‎12-08-2020

Using GetFile I’m collecting files that include JSON records. Each record includes a parameter (let's name it Sensor) that have the Value A or B. Based on that value I wish to route each record to a different output port – which module I should use? Do I need to split the records or I can stay at the level of the file? An example of a record that can be found in the file {"EventTime":"2020-12-07 04:49:00", "Sensor":"A", "Keywords":-9223372036854775808, "EventType":"INFO", "SeverityValue":2, "Severity":"INFO",}

dzbeda · ‎12-06-2020

Hi, I'm using NIFI EvaluateXPath module. I am trying to extract the value from the first Channel parameter (Microsoft-Windows-Sysmon/Operational) and add it as an attribute - The channel1 attribute is empty when setting the value to "/Event/System/Channel" I'm trying to understand what should be the Value in the Channel1 property XML <?xml version="1.0" encoding="UTF-8"?> <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Microsoft-Windows-Sysmon" Guid="{5770385f-c22a-43e0-bf4c-06f5698ffbd9}"/> <EventID>10</EventID> <Version>3</Version> <Level>4</Level> <Task>10</Task> <Opcode>0</Opcode> <Keywords>0x8000000000000000</Keywords> <TimeCreated SystemTime="2020-12-03T14:23:35.660463800Z"/> <EventRecordID>94211</EventRecordID> <Correlation/> <Execution ProcessID="10052" ThreadID="9516"/> <Channel>Microsoft-Windows-Sysmon/Operational</Channel> <Computer>workstation.test.com</Computer> <Security UserID="S-1-5-18"/> </System> <EventData> <Data Name="RuleName"/> <Data Name="UtcTime">2020-12-03 14:23:35.659</Data> <Data Name="SourceProcessGUID">{921b204f-2632-5fc2-0000-0010a0d20100}</Data> <Data Name="SourceProcessId">3428</Data> <Data Name="SourceThreadId">4072</Data> <Data Name="SourceImage">C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\MsMpEng.exe</Data> <Data Name="TargetProcessGUID">{921b204f-2636-5fc2-0000-001085f80200}</Data> <Data Name="TargetProcessId">4212</Data> <Data Name="TargetImage">C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\NisSrv.exe</Data> <Data Name="GrantedAccess">0x1400</Data> <Data Name="CallTrace">C:\Windows\SYSTEM32\ntdll.dll+9c584|C:\Windows\System32\KERNELBASE.dll+2730e|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+1c606f|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+103572|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+1093e6|C:\Windows\System32\ucrtbase.dll+20e72|C:\Windows\System32\KERNEL32.DLL+17bd4|C:\Windows\SYSTEM32\ntdll.dll+6ced1</Data> </EventData> <RenderingInfo Culture="en-US"> <Message>Process accessed: RuleName: UtcTime: 2020-12-03 14:23:35.659 SourceProcessGUID: {921b204f-2632-5fc2-0000-0010a0d20100} SourceProcessId: 3428 SourceThreadId: 4072 SourceImage: C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\MsMpEng.exe TargetProcessGUID: {921b204f-2636-5fc2-0000-001085f80200} TargetProcessId: 4212 TargetImage: C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\NisSrv.exe GrantedAccess: 0x1400 CallTrace: C:\Windows\SYSTEM32\ntdll.dll+9c584|C:\Windows\System32\KERNELBASE.dll+2730e|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+1c606f|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+103572|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+1093e6|C:\Windows\System32\ucrtbase.dll+20e72|C:\Windows\System32\KERNEL32.DLL+17bd4|C:\Windows\SYSTEM32\ntdll.dll+6ced1</Message> <Level>Information</Level> <Task>Process accessed (rule: ProcessAccess)</Task> <Opcode>Info</Opcode> <Channel/> <Provider/> <Keywords/> </RenderingInfo> </Event> Configuration Empty Attribute

dzbeda · ‎12-01-2020

I'm trying to create a site to site connection between 2 Nifi servers. Nifi1 (Hostname wef2 - running on windows) need to deliver traffic to nifi2 (IP 10.168.233.168 - running on Linux) - I defined the connection and all is green but on Nifi1 I'm getting the following error messages What am I missing? *port ID 0176100c-8d25-196b-1f72-6befa5cab12a is the input port in NIFI2 2020-12-01 01:54:13,508 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 2 records in 0 milliseconds 2020-12-01 01:54:20,677 INFO [Http Site-to-Site PeerSelector] o.apache.nifi.remote.client.PeerSelector Successfully refreshed peer status cache; remote group consists of 2 peers 2020-12-01 01:54:23,203 WARN [Timer-Driven Process Thread-3] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://localhost:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'. 2020-12-01 01:54:23,208 WARN [Timer-Driven Process Thread-3] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://wef2:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'. 2020-12-01 01:54:23,208 INFO [Timer-Driven Process Thread-3] o.a.nifi.remote.client.http.HttpClient Couldn't find a valid peer to communicate with. 2020-12-01 01:54:33,214 WARN [Timer-Driven Process Thread-2] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://wef2:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'. 2020-12-01 01:54:33,219 WARN [Timer-Driven Process Thread-2] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://localhost:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'. 2020-12-01 01:54:33,219 INFO [Timer-Driven Process Thread-2] o.a.nifi.remote.client.http.HttpClient Couldn't find a valid peer to communicate with. 2020-12-01 01:54:33,510 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository 2020-12-01 01:54:33,510 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 2 records in 0 milliseconds** Nifi 1 NIFI 2 Remote processor Remote process configuration

dzbeda · ‎11-21-2020

I have 2 Sites (FE & BE )that on each I need to install the NIFI Cluster. FE site is required to transfer data to the BE site using site-to-site protocol. I do know that from time to I have connectivity issues between the sites. since I don't want to lose data, is it possible to configure the FE NIFI cluster to Keep data for 2-3 days in case of a network disconnection between the sites? I do know it will require more disk space If it is possible, which repository I need to extend the Content or the Flow file? and how can it be done? Thank you.

dzbeda · ‎11-17-2020

Hi, I have a domain environment including windows servers. In order to collect all the lof i have implemented WEF and all logs are been pushed to a collector. Now, I wish to inject the logs from the collector to a NIFI server - What is the best way to do it? what kind of listener i should use? Pull or Push? Thanks

dzbeda · ‎06-24-2020

Problem solved. Someone changed system1 UID

dzbeda · ‎06-23-2020

When trying to run a simple job on a Kerborized env. that includes 3 worker nodes. , I’m getting permission error when trying to create a folder under /data1/yarn/nm/usercache/system1/ Job: spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster /opt/cloudera/parcels/CDH/jars/spark-examples*.jar 1000 Errors

dzbeda · ‎03-01-2020

When running the Cloudera wizard with cluster environment more than 10 servers, the final step failed (service startup) due to low memory of Service monitor and Host monitor. In order to overcome this problem, I wish to update the Host Monitor and Service monitor None-java memory via Cloudera API before running the wizard. I was looking for the parameter in the API using swagger but I didn't manage to locate it. Please advice

Online	Offline
Last Visited	‎02-14-2022 04:36 AM

Date Registered	‎03-01-2020 06:38 AM
Last Visited	‎02-14-2022 04:36 AM
Total Messages Posted	14

HDFS balancer with small files

Monitoring NIFI with Prometheus - Is there an avai...

NIFI Get Splunk

NIFI - How to route a record\event based on conten...

Nifi - Extract value from XML content and add it a...

NIFI Site to site -what am I missing ?

Can NIFI storage be extended?

How to collect windows event log using Nifi

Re: Unable to execute spark \ MapReduce job

Unable to execute spark \ MapReduce job

HOw to update Host Monitor and Service monitor Non...