Member since
03-01-2020
10
Posts
0
Kudos Received
0
Solutions
01-15-2021
07:11 AM
MiNiFi offers CPP version that is well suited for Windows event log ingestion.
... View more
12-09-2020
10:16 AM
Look at everything that has *Record in the name for anything like CSV, JSON, Parquet, AVRO, Logs. https://www.datainmotion.dev/2020/07/ingesting-all-weather-data-with-apache.html
... View more
12-07-2020
11:28 PM
Hi I have NIFI-1 that is installed on a windows box. NIFI-1 should take JSON files from the windows box (each file includes multi-windows events & i have used getfile in order to get teh files) and pass them to NIFI-2 for further processing On NIFI -2 I need to take each event and add an attribute that is based on one of the content parameters Should I send the JSON files from NIFI-1 to NIFI-2 using site to site or I should extract the attribute on NIFI-1 and send the events instead of the file I’m guessing that Network wise it is better to move file instead of events Storage-wise it will require more IOPS on NIFI-2 Which module I should use that extracts the events from the file? Can the same module extract the content and define the attribute or I should use an additional module such as EvaluateJsonPath processor?
... View more
Labels:
12-07-2020
11:28 AM
I recommend not using nifi and working with the console. Using nifi is not recommended because additional logs are generated. It is recommended to divide and compress the files that you want to move from the console into appropriate sizes and send them. For HDFS, you can use the distcp command. https://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html
... View more
12-06-2020
10:50 PM
1 Kudo
@dzbeda Try it with: /*:Event/*:System/*:Channel
... View more
12-01-2020
06:53 AM
1 Kudo
@dzbeda Seeing "localhost" in your shared log output leads to what may be the issue. When you configure the URL in the Remote Process Group (RPG), it tells that NiFI RPG to communicate with that URL to fetch the Site-To-Site (S2S) details. Included in those returned details are things like: - number of nodes in target NiFi cluster (if standalone returns only one host) - The hostnames of those node(s) (in this case it looks like maybe localhost is being returned - Configured RAW port if configured - Whether HTTP transport protocol is enabled - etc... So when you RPG actually tries to send FlowFiles over S2S it is trying to send to localhost which results in itself rather than the actual target linux NiFi it fetched the S2S details from. When some properties are left unconfigured, NiFi returns whatever the OS resolves. I am going to guess your linux server is returning localhost rather than the actual hostname. You will want to verify your S2S configuration setup in the target NiFi (linux server): http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#site_to_site_properties Try setting the " nifi.remote.input.host" to see if that helps you. Hope this helps, Matt
... View more
04-12-2020
01:27 AM
I'm running Cloudera 6.3.3 on Redhat 7.6 OS. After full installation, NFS Gateway started successfully. Following Cloudera recommendation, I have disabled IPv6 and NFS Gateway failed to load IPv6 Disable command : net.ipv6.conf.all.disable_ipv6 = 1 in /etc/sysctl.conf net.ipv6.conf.default.disable_ipv6 = 1 in /etc/sysctl.conf Please note that I have failed to start rpcbind service Error log :
... View more
Labels:
03-01-2020
07:32 AM
1 Kudo
You can use the config API to update a service configuration - Document is in link https://cloudera.github.io/cm_api/apidocs/v11/path__clusters_-clusterName-_services_-serviceName-_roleConfigGroups_-roleConfigGroupName-_config.html I am quoting an example for service monitor below You can find the list of all roleconfiggroup by running command
curl -u 'username:password' http://xxx:7180/api/v14/cm/service/roleConfigGroups/
Next I am updating the heap of service monitor to 650MB
curl -u 'username:password' -X PUT -H "Content-Type:application/json" -H "Accept:application/json" -d '{"items":[{ "name": "firehose_heapsize","value": "681574400"}]}' http://xxxx:7180/api/v14/cm/service/roleConfigGroups/mgmt-SERVICEMONITOR-BASE/config
Then restarted it
curl -X POST -u 'username:password' http://xxxx:7180/api/v14/cm/service/commands/restart
... View more