About dzbeda

dzbeda · ‎12-08-2020

Using GetFile I’m collecting files that include JSON records. Each record includes a parameter (let's name it Sensor) that have the Value A or B. Based on that value I wish to route each record to a different output port – which module I should use? Do I need to split the records or I can stay at the level of the file? An example of a record that can be found in the file {"EventTime":"2020-12-07 04:49:00", "Sensor":"A", "Keywords":-9223372036854775808, "EventType":"INFO", "SeverityValue":2, "Severity":"INFO",}

dzbeda · ‎12-07-2020

Hi I have NIFI-1 that is installed on a windows box. NIFI-1 should take JSON files from the windows box (each file includes multi-windows events & i have used getfile in order to get teh files) and pass them to NIFI-2 for further processing On NIFI -2 I need to take each event and add an attribute that is based on one of the content parameters Should I send the JSON files from NIFI-1 to NIFI-2 using site to site or I should extract the attribute on NIFI-1 and send the events instead of the file I’m guessing that Network wise it is better to move file instead of events Storage-wise it will require more IOPS on NIFI-2 Which module I should use that extracts the events from the file? Can the same module extract the content and define the attribute or I should use an additional module such as EvaluateJsonPath processor?

dzbeda · ‎12-06-2020

Hi, I'm using NIFI EvaluateXPath module. I am trying to extract the value from the first Channel parameter (Microsoft-Windows-Sysmon/Operational) and add it as an attribute - The channel1 attribute is empty when setting the value to "/Event/System/Channel" I'm trying to understand what should be the Value in the Channel1 property XML <?xml version="1.0" encoding="UTF-8"?> <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Microsoft-Windows-Sysmon" Guid="{5770385f-c22a-43e0-bf4c-06f5698ffbd9}"/> <EventID>10</EventID> <Version>3</Version> <Level>4</Level> <Task>10</Task> <Opcode>0</Opcode> <Keywords>0x8000000000000000</Keywords> <TimeCreated SystemTime="2020-12-03T14:23:35.660463800Z"/> <EventRecordID>94211</EventRecordID> <Correlation/> <Execution ProcessID="10052" ThreadID="9516"/> <Channel>Microsoft-Windows-Sysmon/Operational</Channel> <Computer>workstation.test.com</Computer> <Security UserID="S-1-5-18"/> </System> <EventData> <Data Name="RuleName"/> <Data Name="UtcTime">2020-12-03 14:23:35.659</Data> <Data Name="SourceProcessGUID">{921b204f-2632-5fc2-0000-0010a0d20100}</Data> <Data Name="SourceProcessId">3428</Data> <Data Name="SourceThreadId">4072</Data> <Data Name="SourceImage">C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\MsMpEng.exe</Data> <Data Name="TargetProcessGUID">{921b204f-2636-5fc2-0000-001085f80200}</Data> <Data Name="TargetProcessId">4212</Data> <Data Name="TargetImage">C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\NisSrv.exe</Data> <Data Name="GrantedAccess">0x1400</Data> <Data Name="CallTrace">C:\Windows\SYSTEM32\ntdll.dll+9c584|C:\Windows\System32\KERNELBASE.dll+2730e|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+1c606f|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+103572|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+1093e6|C:\Windows\System32\ucrtbase.dll+20e72|C:\Windows\System32\KERNEL32.DLL+17bd4|C:\Windows\SYSTEM32\ntdll.dll+6ced1</Data> </EventData> <RenderingInfo Culture="en-US"> <Message>Process accessed: RuleName: UtcTime: 2020-12-03 14:23:35.659 SourceProcessGUID: {921b204f-2632-5fc2-0000-0010a0d20100} SourceProcessId: 3428 SourceThreadId: 4072 SourceImage: C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\MsMpEng.exe TargetProcessGUID: {921b204f-2636-5fc2-0000-001085f80200} TargetProcessId: 4212 TargetImage: C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\NisSrv.exe GrantedAccess: 0x1400 CallTrace: C:\Windows\SYSTEM32\ntdll.dll+9c584|C:\Windows\System32\KERNELBASE.dll+2730e|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+1c606f|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+103572|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+1093e6|C:\Windows\System32\ucrtbase.dll+20e72|C:\Windows\System32\KERNEL32.DLL+17bd4|C:\Windows\SYSTEM32\ntdll.dll+6ced1</Message> <Level>Information</Level> <Task>Process accessed (rule: ProcessAccess)</Task> <Opcode>Info</Opcode> <Channel/> <Provider/> <Keywords/> </RenderingInfo> </Event> Configuration Empty Attribute

dzbeda · ‎12-01-2020

I'm trying to create a site to site connection between 2 Nifi servers. Nifi1 (Hostname wef2 - running on windows) need to deliver traffic to nifi2 (IP 10.168.233.168 - running on Linux) - I defined the connection and all is green but on Nifi1 I'm getting the following error messages What am I missing? *port ID 0176100c-8d25-196b-1f72-6befa5cab12a is the input port in NIFI2 2020-12-01 01:54:13,508 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 2 records in 0 milliseconds 2020-12-01 01:54:20,677 INFO [Http Site-to-Site PeerSelector] o.apache.nifi.remote.client.PeerSelector Successfully refreshed peer status cache; remote group consists of 2 peers 2020-12-01 01:54:23,203 WARN [Timer-Driven Process Thread-3] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://localhost:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'. 2020-12-01 01:54:23,208 WARN [Timer-Driven Process Thread-3] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://wef2:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'. 2020-12-01 01:54:23,208 INFO [Timer-Driven Process Thread-3] o.a.nifi.remote.client.http.HttpClient Couldn't find a valid peer to communicate with. 2020-12-01 01:54:33,214 WARN [Timer-Driven Process Thread-2] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://wef2:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'. 2020-12-01 01:54:33,219 WARN [Timer-Driven Process Thread-2] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://localhost:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'. 2020-12-01 01:54:33,219 INFO [Timer-Driven Process Thread-2] o.a.nifi.remote.client.http.HttpClient Couldn't find a valid peer to communicate with. 2020-12-01 01:54:33,510 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository 2020-12-01 01:54:33,510 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 2 records in 0 milliseconds** Nifi 1 NIFI 2 Remote processor Remote process configuration

dzbeda · ‎11-21-2020

I have 2 Sites (FE & BE )that on each I need to install the NIFI Cluster. FE site is required to transfer data to the BE site using site-to-site protocol. I do know that from time to I have connectivity issues between the sites. since I don't want to lose data, is it possible to configure the FE NIFI cluster to Keep data for 2-3 days in case of a network disconnection between the sites? I do know it will require more disk space If it is possible, which repository I need to extend the Content or the Flow file? and how can it be done? Thank you.

dzbeda · ‎11-17-2020

Hi, I have a domain environment including windows servers. In order to collect all the lof i have implemented WEF and all logs are been pushed to a collector. Now, I wish to inject the logs from the collector to a NIFI server - What is the best way to do it? what kind of listener i should use? Pull or Push? Thanks

dzbeda · ‎06-24-2020

Problem solved. Someone changed system1 UID

dzbeda · ‎06-23-2020

When trying to run a simple job on a Kerborized env. that includes 3 worker nodes. , I’m getting permission error when trying to create a folder under /data1/yarn/nm/usercache/system1/ Job: spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster /opt/cloudera/parcels/CDH/jars/spark-examples*.jar 1000 Errors

dzbeda · ‎04-12-2020

I'm running Cloudera 6.3.3 on Redhat 7.6 OS. After full installation, NFS Gateway started successfully. Following Cloudera recommendation, I have disabled IPv6 and NFS Gateway failed to load IPv6 Disable command : net.ipv6.conf.all.disable_ipv6 = 1 in /etc/sysctl.conf net.ipv6.conf.default.disable_ipv6 = 1 in /etc/sysctl.conf Please note that I have failed to start rpcbind service Error log :

dzbeda · ‎03-01-2020

When running the Cloudera wizard with cluster environment more than 10 servers, the final step failed (service startup) due to low memory of Service monitor and Host monitor. In order to overcome this problem, I wish to update the Host Monitor and Service monitor None-java memory via Cloudera API before running the wizard. I was looking for the parameter in the API using swagger but I didn't manage to locate it. Please advice

Online	Offline
Last Visited	‎12-09-2020 11:18 PM

Date Registered	‎03-01-2020 06:38 AM
Last Visited	‎12-09-2020 11:18 PM
Total Messages Posted	10

NIFI - How to route a record\event based on conten...

NIFI - Which module i should use in order to extra...

Nifi - Extract value from XML content and add it a...

NIFI Site to site -what am I missing ?

Can NIFI storage be extended?

How to collect windows event log using Nifi

Re: Unable to execute spark \ MapReduce job

Unable to execute spark \ MapReduce job

NFS Gateway failed to start after IPv6 was disable...

HOw to update Host Monitor and Service monitor Non...