About dzbeda

dzbeda · ‎10-16-2021

Hi, I have an HDFS cluster with 3 nodes. The cluster holds lots of small files (KB) and I have reached Millions of blocks per node. I have added 4 more new servers to the cluster and started the balancer process but it looks that it does not do much. - The goal is to reduce the Million of blocks per server In order to balance the small-size files should i change the value of the following parameter to support moving files from 1KB size? Ddfs.balancer.getBlocks.min-block-size=1048

dzbeda · ‎04-17-2021

Hi, I would like to monitor NIFI with Prometheus -Mainly to understand if Queue is full and the performance of specific processors. Is there an exporter for that? Grafan dashboards? Thanks

dzbeda · ‎03-24-2021

is it possible to get records from Splunk based on a specific index? Where can I define the different indexes I want to quey? Thanks

dzbeda · ‎12-08-2020

Using GetFile I’m collecting files that include JSON records. Each record includes a parameter (let's name it Sensor) that have the Value A or B. Based on that value I wish to route each record to a different output port – which module I should use? Do I need to split the records or I can stay at the level of the file? An example of a record that can be found in the file {"EventTime":"2020-12-07 04:49:00", "Sensor":"A", "Keywords":-9223372036854775808, "EventType":"INFO", "SeverityValue":2, "Severity":"INFO",}

dzbeda · ‎12-06-2020

Hi, I'm using NIFI EvaluateXPath module. I am trying to extract the value from the first Channel parameter (Microsoft-Windows-Sysmon/Operational) and add it as an attribute - The channel1 attribute is empty when setting the value to "/Event/System/Channel" I'm trying to understand what should be the Value in the Channel1 property XML <?xml version="1.0" encoding="UTF-8"?> <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Microsoft-Windows-Sysmon" Guid="{5770385f-c22a-43e0-bf4c-06f5698ffbd9}"/> <EventID>10</EventID> <Version>3</Version> <Level>4</Level> <Task>10</Task> <Opcode>0</Opcode> <Keywords>0x8000000000000000</Keywords> <TimeCreated SystemTime="2020-12-03T14:23:35.660463800Z"/> <EventRecordID>94211</EventRecordID> <Correlation/> <Execution ProcessID="10052" ThreadID="9516"/> <Channel>Microsoft-Windows-Sysmon/Operational</Channel> <Computer>workstation.test.com</Computer> <Security UserID="S-1-5-18"/> </System> <EventData> <Data Name="RuleName"/> <Data Name="UtcTime">2020-12-03 14:23:35.659</Data> <Data Name="SourceProcessGUID">{921b204f-2632-5fc2-0000-0010a0d20100}</Data> <Data Name="SourceProcessId">3428</Data> <Data Name="SourceThreadId">4072</Data> <Data Name="SourceImage">C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\MsMpEng.exe</Data> <Data Name="TargetProcessGUID">{921b204f-2636-5fc2-0000-001085f80200}</Data> <Data Name="TargetProcessId">4212</Data> <Data Name="TargetImage">C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\NisSrv.exe</Data> <Data Name="GrantedAccess">0x1400</Data> <Data Name="CallTrace">C:\Windows\SYSTEM32\ntdll.dll+9c584|C:\Windows\System32\KERNELBASE.dll+2730e|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+1c606f|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+103572|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+1093e6|C:\Windows\System32\ucrtbase.dll+20e72|C:\Windows\System32\KERNEL32.DLL+17bd4|C:\Windows\SYSTEM32\ntdll.dll+6ced1</Data> </EventData> <RenderingInfo Culture="en-US"> <Message>Process accessed: RuleName: UtcTime: 2020-12-03 14:23:35.659 SourceProcessGUID: {921b204f-2632-5fc2-0000-0010a0d20100} SourceProcessId: 3428 SourceThreadId: 4072 SourceImage: C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\MsMpEng.exe TargetProcessGUID: {921b204f-2636-5fc2-0000-001085f80200} TargetProcessId: 4212 TargetImage: C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\NisSrv.exe GrantedAccess: 0x1400 CallTrace: C:\Windows\SYSTEM32\ntdll.dll+9c584|C:\Windows\System32\KERNELBASE.dll+2730e|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+1c606f|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+103572|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+1093e6|C:\Windows\System32\ucrtbase.dll+20e72|C:\Windows\System32\KERNEL32.DLL+17bd4|C:\Windows\SYSTEM32\ntdll.dll+6ced1</Message> <Level>Information</Level> <Task>Process accessed (rule: ProcessAccess)</Task> <Opcode>Info</Opcode> <Channel/> <Provider/> <Keywords/> </RenderingInfo> </Event> Configuration Empty Attribute

dzbeda · ‎12-01-2020

I'm trying to create a site to site connection between 2 Nifi servers. Nifi1 (Hostname wef2 - running on windows) need to deliver traffic to nifi2 (IP 10.168.233.168 - running on Linux) - I defined the connection and all is green but on Nifi1 I'm getting the following error messages What am I missing? *port ID 0176100c-8d25-196b-1f72-6befa5cab12a is the input port in NIFI2 2020-12-01 01:54:13,508 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 2 records in 0 milliseconds 2020-12-01 01:54:20,677 INFO [Http Site-to-Site PeerSelector] o.apache.nifi.remote.client.PeerSelector Successfully refreshed peer status cache; remote group consists of 2 peers 2020-12-01 01:54:23,203 WARN [Timer-Driven Process Thread-3] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://localhost:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'. 2020-12-01 01:54:23,208 WARN [Timer-Driven Process Thread-3] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://wef2:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'. 2020-12-01 01:54:23,208 INFO [Timer-Driven Process Thread-3] o.a.nifi.remote.client.http.HttpClient Couldn't find a valid peer to communicate with. 2020-12-01 01:54:33,214 WARN [Timer-Driven Process Thread-2] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://wef2:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'. 2020-12-01 01:54:33,219 WARN [Timer-Driven Process Thread-2] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://localhost:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'. 2020-12-01 01:54:33,219 INFO [Timer-Driven Process Thread-2] o.a.nifi.remote.client.http.HttpClient Couldn't find a valid peer to communicate with. 2020-12-01 01:54:33,510 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository 2020-12-01 01:54:33,510 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 2 records in 0 milliseconds** Nifi 1 NIFI 2 Remote processor Remote process configuration

dzbeda · ‎11-21-2020

I have 2 Sites (FE & BE )that on each I need to install the NIFI Cluster. FE site is required to transfer data to the BE site using site-to-site protocol. I do know that from time to I have connectivity issues between the sites. since I don't want to lose data, is it possible to configure the FE NIFI cluster to Keep data for 2-3 days in case of a network disconnection between the sites? I do know it will require more disk space If it is possible, which repository I need to extend the Content or the Flow file? and how can it be done? Thank you.

dzbeda · ‎11-17-2020

Hi, I have a domain environment including windows servers. In order to collect all the lof i have implemented WEF and all logs are been pushed to a collector. Now, I wish to inject the logs from the collector to a NIFI server - What is the best way to do it? what kind of listener i should use? Pull or Push? Thanks

Online	Offline
Last Visited	‎03-07-2023 03:43 PM

Member Since	‎03-01-2020 06:38 AM
Last Visited	‎03-07-2023 03:43 PM
Posts	15

Cloudera Community

HDFS balancer with small files

Monitoring NIFI with Prometheus - Is there an avai...

NIFI Get Splunk

NIFI - How to route a record\event based on conten...

Nifi - Extract value from XML content and add it a...

NIFI Site to site -what am I missing ?

Can NIFI storage be extended?

How to collect windows event log using Nifi