Member since
03-01-2020
10
Posts
0
Kudos Received
0
Solutions
12-08-2020
12:34 AM
Using GetFile I’m collecting files that include JSON records. Each record includes a parameter (let's name it Sensor) that have the Value A or B. Based on that value I wish to route each record to a different output port – which module I should use? Do I need to split the records or I can stay at the level of the file? An example of a record that can be found in the file {"EventTime":"2020-12-07 04:49:00", "Sensor":"A", "Keywords":-9223372036854775808, "EventType":"INFO", "SeverityValue":2, "Severity":"INFO",}
... View more
- Tags:
- NiFi
- route record
Labels:
12-07-2020
11:28 PM
Hi I have NIFI-1 that is installed on a windows box. NIFI-1 should take JSON files from the windows box (each file includes multi-windows events & i have used getfile in order to get teh files) and pass them to NIFI-2 for further processing On NIFI -2 I need to take each event and add an attribute that is based on one of the content parameters Should I send the JSON files from NIFI-1 to NIFI-2 using site to site or I should extract the attribute on NIFI-1 and send the events instead of the file I’m guessing that Network wise it is better to move file instead of events Storage-wise it will require more IOPS on NIFI-2 Which module I should use that extracts the events from the file? Can the same module extract the content and define the attribute or I should use an additional module such as EvaluateJsonPath processor?
... View more
Labels:
12-06-2020
10:36 PM
Hi, I'm using NIFI EvaluateXPath module. I am trying to extract the value from the first Channel parameter (Microsoft-Windows-Sysmon/Operational) and add it as an attribute - The channel1 attribute is empty when setting the value to "/Event/System/Channel" I'm trying to understand what should be the Value in the Channel1 property XML <?xml version="1.0" encoding="UTF-8"?>
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-Sysmon"
Guid="{5770385f-c22a-43e0-bf4c-06f5698ffbd9}"/>
<EventID>10</EventID>
<Version>3</Version>
<Level>4</Level>
<Task>10</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2020-12-03T14:23:35.660463800Z"/>
<EventRecordID>94211</EventRecordID>
<Correlation/>
<Execution ProcessID="10052" ThreadID="9516"/>
<Channel>Microsoft-Windows-Sysmon/Operational</Channel>
<Computer>workstation.test.com</Computer>
<Security UserID="S-1-5-18"/>
</System>
<EventData>
<Data Name="RuleName"/>
<Data Name="UtcTime">2020-12-03 14:23:35.659</Data>
<Data Name="SourceProcessGUID">{921b204f-2632-5fc2-0000-0010a0d20100}</Data>
<Data Name="SourceProcessId">3428</Data>
<Data Name="SourceThreadId">4072</Data>
<Data Name="SourceImage">C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\MsMpEng.exe</Data>
<Data Name="TargetProcessGUID">{921b204f-2636-5fc2-0000-001085f80200}</Data>
<Data Name="TargetProcessId">4212</Data>
<Data Name="TargetImage">C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\NisSrv.exe</Data>
<Data Name="GrantedAccess">0x1400</Data>
<Data Name="CallTrace">C:\Windows\SYSTEM32\ntdll.dll+9c584|C:\Windows\System32\KERNELBASE.dll+2730e|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+1c606f|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+103572|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+1093e6|C:\Windows\System32\ucrtbase.dll+20e72|C:\Windows\System32\KERNEL32.DLL+17bd4|C:\Windows\SYSTEM32\ntdll.dll+6ced1</Data>
</EventData>
<RenderingInfo Culture="en-US">
<Message>Process accessed:
RuleName:
UtcTime: 2020-12-03 14:23:35.659
SourceProcessGUID: {921b204f-2632-5fc2-0000-0010a0d20100}
SourceProcessId: 3428
SourceThreadId: 4072
SourceImage: C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\MsMpEng.exe
TargetProcessGUID: {921b204f-2636-5fc2-0000-001085f80200}
TargetProcessId: 4212
TargetImage: C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\NisSrv.exe
GrantedAccess: 0x1400
CallTrace: C:\Windows\SYSTEM32\ntdll.dll+9c584|C:\Windows\System32\KERNELBASE.dll+2730e|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+1c606f|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+103572|C:\ProgramData\Microsoft\Windows Defender\platform\4.18.2010.7-0\mpsvc.dll+1093e6|C:\Windows\System32\ucrtbase.dll+20e72|C:\Windows\System32\KERNEL32.DLL+17bd4|C:\Windows\SYSTEM32\ntdll.dll+6ced1</Message>
<Level>Information</Level>
<Task>Process accessed (rule: ProcessAccess)</Task>
<Opcode>Info</Opcode>
<Channel/>
<Provider/>
<Keywords/>
</RenderingInfo>
</Event> Configuration Empty Attribute
... View more
- Tags:
- EvaluateXPath
- NiFi
Labels:
12-01-2020
04:22 AM
I'm trying to create a site to site connection between 2 Nifi servers. Nifi1 (Hostname wef2 - running on windows) need to deliver traffic to nifi2 (IP 10.168.233.168 - running on Linux) - I defined the connection and all is green but on Nifi1 I'm getting the following error messages What am I missing? *port ID 0176100c-8d25-196b-1f72-6befa5cab12a is the input port in NIFI2 2020-12-01 01:54:13,508 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 2 records in 0 milliseconds
2020-12-01 01:54:20,677 INFO [Http Site-to-Site PeerSelector] o.apache.nifi.remote.client.PeerSelector Successfully refreshed peer status cache; remote group consists of 2 peers
2020-12-01 01:54:23,203 WARN [Timer-Driven Process Thread-3] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://localhost:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'.
2020-12-01 01:54:23,208 WARN [Timer-Driven Process Thread-3] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://wef2:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'.
2020-12-01 01:54:23,208 INFO [Timer-Driven Process Thread-3] o.a.nifi.remote.client.http.HttpClient Couldn't find a valid peer to communicate with.
2020-12-01 01:54:33,214 WARN [Timer-Driven Process Thread-2] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://wef2:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'.
2020-12-01 01:54:33,219 WARN [Timer-Driven Process Thread-2] o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://localhost:8080/nifi-api] due to java.io.IOException: Unexpected response code: 404 errCode:Abort errMessage:Unable to find port with id '0176100c-8d25-196b-1f72-6befa5cab12a'.
2020-12-01 01:54:33,219 INFO [Timer-Driven Process Thread-2] o.a.nifi.remote.client.http.HttpClient Couldn't find a valid peer to communicate with.
2020-12-01 01:54:33,510 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2020-12-01 01:54:33,510 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 2 records in 0 milliseconds** Nifi 1 NIFI 2 Remote processor Remote process configuration
... View more
- Tags:
- NiFi
Labels:
11-21-2020
12:15 PM
I have 2 Sites (FE & BE )that on each I need to install the NIFI Cluster. FE site is required to transfer data to the BE site using site-to-site protocol. I do know that from time to I have connectivity issues between the sites. since I don't want to lose data, is it possible to configure the FE NIFI cluster to Keep data for 2-3 days in case of a network disconnection between the sites? I do know it will require more disk space If it is possible, which repository I need to extend the Content or the Flow file? and how can it be done? Thank you.
... View more
Labels:
11-17-2020
02:21 AM
Hi, I have a domain environment including windows servers. In order to collect all the lof i have implemented WEF and all logs are been pushed to a collector. Now, I wish to inject the logs from the collector to a NIFI server - What is the best way to do it? what kind of listener i should use? Pull or Push? Thanks
... View more
Labels:
06-23-2020
11:56 PM
When trying to run a simple job on a Kerborized env. that includes 3 worker nodes. , I’m getting permission error when trying to create a folder under /data1/yarn/nm/usercache/system1/ Job: spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster /opt/cloudera/parcels/CDH/jars/spark-examples*.jar 1000 Errors
... View more
- Tags:
- Map Reduce
- Spark
- YARN
Labels:
04-12-2020
01:27 AM
I'm running Cloudera 6.3.3 on Redhat 7.6 OS. After full installation, NFS Gateway started successfully. Following Cloudera recommendation, I have disabled IPv6 and NFS Gateway failed to load IPv6 Disable command : net.ipv6.conf.all.disable_ipv6 = 1 in /etc/sysctl.conf net.ipv6.conf.default.disable_ipv6 = 1 in /etc/sysctl.conf Please note that I have failed to start rpcbind service Error log :
... View more
Labels:
03-01-2020
06:46 AM
When running the Cloudera wizard with cluster environment more than 10 servers, the final step failed (service startup) due to low memory of Service monitor and Host monitor. In order to overcome this problem, I wish to update the Host Monitor and Service monitor None-java memory via Cloudera API before running the wizard. I was looking for the parameter in the API using swagger but I didn't manage to locate it.
Please advice
... View more
Labels: