Member since
01-15-2016
34
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1561 | 08-16-2017 09:48 AM |
08-24-2017
07:57 AM
Hi @bkosaraju, sorry, was my fault! Solution runs fine !!! My input data was wrong! Thanks, great solution. Timo
... View more
08-23-2017
01:56 PM
PutSolrStreamContent config
... View more
08-23-2017
01:50 PM
Hi @bkosaraju, i have some problems configuring filed mapping for each of the json fields. This is the json file structure : { "event_timestamp":"${time:substring(0,20):toDate('yyyy MMM dd HH:mm:ss'):format('yyyy-MM-dd HH:mm:ss')}", "host":"${message:substring(21):substringBefore('')}", "message":"${message:substring(21):substringAfter(' '):replace('"', '\\\\"')}" } this is my PutSolrContentStream config: Indexing failed ...
... View more
08-22-2017
12:29 PM
Hi @bkosaraju, thats what what I#m trying! i got the string from ListenSyslog and convert the string to json. example: {"event_timestamp":"2017-08-22 11:15:06","host":"meyer.devcon.com","message":"sshd[9411]: debug1: PAM: setting PAM_RHOST to monitoring-portal.devcon1.com"} The ingest runs fine, but a can't do sorting. We got this error: can not sort on multivalued field: event_timestamp. that's wy im asking for a solution to create a schema (HDP Search / Solr) with indexed single-valued fields (event_timestamp, host). regards Timo
... View more
08-22-2017
09:17 AM
Hi @bkosaraju, we need a realtime ingest. The syslog messages comes from hundreds of host ingested by Nifi ListenSyslog over PutSolrContentStream. regards Timo
... View more
08-22-2017
08:55 AM
Hi @Geoffrey Shelton Okot, my intension is to search all message sorted by timestamp or host. so, how can i create a schema (HDP Search / Solr) with indexed single-valued fields (event_timestamp, host). Sorting of messages is not necessary. Best regards Timo
... View more
08-22-2017
08:28 AM
Hi @Sonu Sahi Unfortunately the tutorial does not help me in this case. Using the Sample schemaless configuration uses creates multiValue filelds. In addition we ingest the document by nifi with PutSolrContentStream The sorting of the fields does not work like this. So we got this error: can not sort on multivalued field: event_timestamp regards Timo
... View more
08-21-2017
09:16 AM
Hi, we store all of out system syslog files in our Data Lake. For realtime Monitoring we have to store the syslogs with SOLR. The syslog structure is easy: example: timestamp, hostname, message time | host | message ---------------------+------------------------+---------------------------------------------- 2017-08-08 19:39:48 | xxxx.xxx.com | systemd[4554]: Stopped target Default. 2017-08-09 09:54:57 | yxz.gsj.com| systemd[28096]: Starting Shutdown. 2017-08-09 14:48:39 | yxz.gsj.com | systemd[22015]: Stopping Timers. 2017-08-09 12:12:37 | yxz.gsj.com| systemd[1]: Started User Manager for UID 0. 2017-08-10 00:00:37 | xxxx.xxx.com | systemd[15736]: Stopping Basic System. How can we build a schema in SOLR with thew possibility to search in all fields and do sorting the output by time, host or message? Thanks for you help. Timo
... View more
Labels:
- Labels:
-
Apache Solr
08-16-2017
09:48 AM
Got the solution: 1. Replace String with actual year search value: <(.*?)> Replacement Value: ${now():format('yyyy ')} 2. Extract Timestamp and Message time = (^.{20}) message = (^.*$) 3. Replace String with insert into clause: search value: (?s)(^.*$) Replacement Value: insert into syslog values ('${time:substring(0,20):toDate('yyyy MMM dd HH:mm:ss'):format('yyyy-MM-dd HH:mm:ss')}','${message:substring(21):substringBefore(' ')}','${message:substring(21):substringAfter(' ')}');
... View more
08-14-2017
02:00 PM
Hi, i have the following input single row FlowFiles: Aug 9 10:34:12 sysxyz-p2 sudo: pam_unix(sudo:session): session opened for user root by (uid=0) Aug 10 10:34:12 sysxxx.xxx.xx sudo: pam_unix(sudo:session): session opened for user root by (uid=0) Aug 11 10:34:12 sysxxx.xxx.xx sudo: pam_unix(sudo:session): session closed for user root Aug 12 10:34:12 sysxxx.xxx.xyz sudo: pam_unix(sudo:session): session closed for user root Aug 13 10:34:01 sysxxx.xxx.xx CRON[26312]: pam_unix(cron:session): session opened for user monadmin by (uid=0) 1. Split the Flow into timestamp, host, message fields: timestamp host message Aug 9 10:34:12 sysxyz-p2 sudo: pam_unix(sudo:session): session opened for user root by (uid=0) Aug 10 10:34:12 sysxxx.xxx.xx sudo: pam_unix(sudo:session): session opened for user root by (uid=0) Aug 11 10:34:12 sysxxx.xxx.xx sudo: pam_unix(sudo:session): session closed for user root Aug 12 10:34:12 sysxxx.xxx.xyz sudo: pam_unix(sudo:session): session closed for user root Aug 13 10:34:01 sysxxx.xxx.xx CRON[26312]: pam_unix(cron:session): session opened for user monadmin by (uid=0) 2. Convert timestamp Format: Aug 9 10:34:12 - > 'YYYY.MM.DD HH:MI:SS' 3. Concat the extracted files to insert into ... string: example: insert into table ('2017-08-09 10:34:12', 'sysxyz-p2', 'sudo: pam_unix(sudo:session): session opened for user root by (uid=0)'; Thanks for you help! Timo
... View more
Labels:
- Labels:
-
Apache NiFi
08-11-2017
01:44 PM
Great, works !!! Thanks!
... View more
08-11-2017
11:52 AM
Hi Bryan, how can i merge the FlowData Files to a multiline File? Using the MergeCentent processor i got a concatenated line. input: 1 FlowData: <30>Aug 11 06:27:26 xxx.xxx.com systemd[28892]: Stopping Timers. 2 FlowData:<30>Aug 11 06:27:15 xxx.xxx.com systemd[24517]: Stopping Paths etc... MergeContent output: <30>Aug 11 06:27:26 xxx.xxx.com systemd[28892]: Stopping Timers.<30>Aug 11 06:27:15 xxx.xxx.com systemd[24517]:Stopping Paths I need the following multiline structure: <30>Aug 11 06:27:26 xxx.xxx.com systemd[28892]: Stopping Timers. <30>Aug 11 06:27:15 xxx.xxx.com systemd[24517]:Stopping Paths any solution? regards Timo
... View more
08-09-2017
04:02 PM
Thanks, this was really easy. I just generating "insert into table" statements including the Flow File with the ReplaceText processor and connect it with the PutSQL processor. regards Timo
... View more
08-08-2017
06:04 PM
Hi, 1. step we ingest SYSLOG messages with the ListenSyslog and PutHDFS in our DataLake. 2. step ingest with PXF external table technology into HAWQ. CREATE EXTERNAL TABLE ext_syslog_hist (message TEXT) LOCATION ('pxf://xxx.xxx.com:8020/apps/nifi/syslog/history/*?PROFILE=HdfsTextSimple') FORMAT 'TEXT' (delimiter=E'\t'); Every minute there are thousands of small (100 bytes) files created in the HDFS. So I'm looking for a way to ingest the incoming ListenSyslog directly into a HAWQ table. Regards Timo
... View more
Labels:
- Labels:
-
Apache NiFi
07-13-2017
01:50 PM
Labels:
- Labels:
-
Apache NiFi
06-21-2017
09:34 AM
Hi,
we get every day a dynamic list if host with the InvokeHTTP processor from our source system. On this base we have to request the source system again with InvokeHTTP host by host to get detailed data. 1. InvokeHTTP Request: https://xyz.abc.com/prod/check_mk/view.py?view_name=hostgroup&hostgroup=vmware-host&limit=hard&output_format=json&_username=bi&_secret=XXXXXXXXX
Output :
[{"host_state":"UP","host“:“xxx.yyy.com“,“host_icons":"menu","num_services_ok":"28","num_services_warn":"0","num_services_unknown":"0","num_services_crit":"0","num_services_pending":"0","ctime":"Wed Jun 21 06:00:02 CEST 2017"},{"host_state":"UP","host“:“yyy.yyy.com“,“host_icons":"menu","num_services_ok":"34","num_services_warn":"0","num_services_unknown":"0","num_services_crit":"0","num_services_pending":"0","ctime":"Wed Jun 21 06:00:02 CEST 2017"},{"host_state":"UP","host“:“zzz.yyy.com“,“host_icons":"menu","num_services_ok":"34","num_services_warn":"0","num_services_unknown":"0","num_services_crit":"0","num_services_pending":"0","ctime":"Wed Jun 21 06:00:02 CEST 2017“}] From this output we have to grep the host values (“xxx.yyy.com“, ....) and build an request for every host: 2. InvokeHTTP Requests https://xyz.abc.com/prod/check_mk/webapi.py?action=get_graph&_username=ansible&_secret=XXXXXXXXX&request={"specification":["template",{"service_description":"Filesystem /","site":"prod","graph_index":0,"host_name":"xxx.yyy.com"}],"data_range":{"time_range":[1491174000,1491260340]}} https://xyz.abc.com/prod/check_mk/webapi.py?action=get_graph&_username=ansible&_secret=XXXXXXXXX&request={"specification":["template",{"service_description":"Filesystem /","site":"prod","graph_index":0,"host_name":"yyy.yyy.com"}],"data_range":{"time_range":[1491174000,1491260340]}} https://xyz.abc.com/prod/check_mk/webapi.py?action=get_graph&_username=ansible&_secret=XXXXXXXXX&request={"specification":["template",{"service_description":"Filesystem /","site":"prod","graph_index":0,"host_name":"zzz.yyy.com"}],"data_range":{"time_range":[1491174000,1491260340]}}
...
How can we challenge this?
Thanks
Timo
... View more
Labels:
- Labels:
-
Apache NiFi
06-20-2017
12:24 PM
Thanks Matt! Works perfekt ...
... View more
06-20-2017
08:06 AM
Hi Vineet,
i don't have the ability.
... View more
06-19-2017
01:57 PM
i have the following son structure and want to replace the special character string "[{" and "]}" with "{" and "}" : {"result": {"step": 1800, "start_time": 1491174000, "end_time": 1491260400, "curves": [{"color": "#a05830", "rrddata": [0.603695, 1.06903, 0.94504, 0.68786, 31.3228, 0.316447, 0.808407, 0.247655, 0.174552, 0.123072, 0.62, 0.0689, 0.30758, 0.0869783, 0.14478, 0.305993, 0.808873, 0.193055, 0.113133, 0.46116, 8.047, 1.88388, 2.62721, 0.770247, 8.06144, 2.25591, 22.3061, 57.5539, 0.270233, 1.50602, 0.819887, 5.90425, 0.43361, 0.526907, 2.46678, 0.759873, 0.451133, 0.25843, 0.224033, 0.661373, 1.1279, 0.348587, 0.277142, 0.06647, 0.16693, 0.06225, 0.0588483, 0.08057], "line_type": "area", "title": "Disk utilization"}]}, "result_code": 0}
... View more
Labels:
- Labels:
-
Apache NiFi
06-16-2017
02:35 PM
Hi Vineet,
We use HDB 2.2 and have tried without success the hints of this link.
We tried this without success: CREATE EXTERNAL TABLE kum.ext_checkmk_memory
(
result_code integer,
"result.step" integer,
"result.start_time" integer,
"result.end_time" integer,
"result.curves[0].color" text,
"result.curves[0].rrddata[0]" integer,
"result.curves[0].rrddata[1]" integer,
"result.curves[0].rrddata[2]" integer,
"result.curves[0].rrddata[3]" integer,
"result.curves[0].line_type" text,
"result.curves[0].title" text,
"result.curves[1].color" text,
"result.curves[1].rrddata[0]" integer,
"result.curves[1].rrddata[1]" integer,
"result.curves[1].rrddata[2]" integer,
"result.curves[1].rrddata[3]" integer,
"result.curves[1].line_type" text,
"result.curves[1].title" text,
"result.curves[2].color" text,
"result.curves[2].rrddata[0]" integer,
"result.curves[2].rrddata[1]" integer,
"result.curves[2].rrddata[2]" integer,
"result.curves[2].rrddata[3]" integer,
"result.curves[2].line_type" text,
"result.curves[2].title" text,
"result.curves[3].color" text,
"result.curves[3].rrddata[0]" integer,
"result.curves[3].rrddata[1]" integer,
"result.curves[3].rrddata[2]" integer,
"result.curves[3].rrddata[3]" integer,
"result.curves[3].line_type" text,
"result.curves[3].title" text
)
LOCATION (
'pxf://hdw1:8020/user/nifi/check_mk/memory/*?PROFILE=JSON'
)
FORMAT 'CUSTOM' (formatter='pxfwritable_import' )
the problem are the objects in the curves array ... kum=# select * from kum.ext_checkmk_memory; result_code | result.step | result.start_time | result.end_time | result.curves[0].color | result.curves[0].rrddata[0] | result.curves[0].rrddata[1] | result.curves[0].rrddata[2] | result.curves[0].rrddata[3] | result.curves[0].line_type | result.curves[0].title | result .curves[1].color | result.curves[1].rrddata[0] | result.curves[1].rrddata[1] | result.curves[1].rrddata[2] | result.curves[1].rrddata[3] | result.curves[1].line_type | result.curves[1].title | result.curves[2].color | result.curves[2].rrddata[0] | result.curves[2].rrddata [1] | result.curves[2].rrddata[2] | result.curves[2].rrddata[3] | result.curves[2].line_type | result.curves[2].title | result.curves[3].color | result.curves[3].rrddata[0] | result.curves[3].rrddata[1] | result.curves[3].rrddata[2] | result.curves[3].rrddata[3] | result. curves[3].line_type | result.curves[3].title -------------+-------------+-------------------+-----------------+------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+----------------------------+------------------------+------- -----------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+----------------------------+------------------------+------------------------+-----------------------------+------------------------- ----+-----------------------------+-----------------------------+----------------------------+------------------------+------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+-------- --------------------+------------------------ 0 | 1800 | 1491174000 | 1491260400 | | | | | | | | | | | | | | | | | | | | | | | | | | | |
... View more
06-16-2017
09:04 AM
Hi, we want to access complex json data via a hawq external table. JSON structure: { "result":
{
"step": 1800,
"start_time": 1491174000,
"end_time": 1491260400,
"curves":
[
{
"color": "#f0f0f0",
"rrddata":
[
33672800000,
33672800000,
33672800000,
33672800000
],
"line_type": "area",
"title": "RAM installed"
},
{
"color": "#e0e0e0",
"rrddata":
[
4302300000,
4302300000,
4302300000,
4302300000
],
"line_type": "stack",
"title": "Swap installed"
},
{
"color": "#80ff40",
"rrddata":
[
16683600000,
16690000000,
16690900000,
16696200000
],
"line_type": "area",
"title": "RAM used"
},
{
"color": "#408f20",
"rrddata":
[
1088450000,
1088480000,
1087960000,
1088710000
],
"line_type": "stack",
"title": "Swap used"
} ]
},
"result_code": 0
}
... View more
06-09-2017
08:34 AM
We access Json HDFS files as an external Table in HAWQ (external table:ext_chkmk_hosts).
The "insert into table check_host ... select ..." loads the new data into a HAWQ table (check_hosts) in the same database.
We want to schedule all the jobs with Nifi.
1. Load Data from System (InvokeHTTP)
2. Convert the JSON output (JoltTransformJSON)
3. add key:value (timestamp) (JoltTransformJSON)
4. store the data in HDFS (PutHDFS)
5. Access all HDFS JSON file in HAWQ (external table)
6. insert new data into HAWQ Table (insert into check_hosts ... select ... from ext_checkmk_hosts.
... View more
06-08-2017
03:17 PM
Hi, how can we implement the following Postgres SQL Statement in Nifi?: insert into kum.checkmk_hosts ( ctime, host_state, host, host_icons, num_services_ok, num_services_warn, num_services_unknown, num_services_crit, num_services_pending ) select ctime::timestamp, host_state, host, host_icons, num_services_ok, num_services_warn, num_services_unknown, num_services_crit, num_services_pending from kum.ext_checkmk_hosts where date(ctime) >= current_date;
... View more
Labels:
- Labels:
-
Apache NiFi
06-07-2017
03:15 PM
Hi Matt, after upgrading to nifi 1.2 all works fine !!! Great work!
Thanks
... View more
06-07-2017
02:33 PM
I think my problem my version is : nifi-1.1.2
... View more
06-07-2017
01:57 PM
Hi Matt, now i got only this output : [{ "ctime":"${now()}", "host":"DUSTSADMIN.ads.xyz.de", "host_icons":"menu", "host_state":"UP", "num_services_crit":"0", "num_services_ok":"28", "num_services_pending":"0", "num_services_unknown":"0", "num_services_warn":"0" }, "ctime":"${now()}", "host":"DUSTSVMDC01.ads.xyz.de", "host_icons":"menu", "host_state":"UP", "num_services_crit":"0", "num_services_ok":"34", "num_services_pending":"0", "num_services_unknown":"0", "num_services_warn":"0" },{"ctime":"${now()}", "host":"DUSTSVMDC02.ads.xyz.de", "host_icons":"menu", "host_state":"UP", "num_services_crit":"0", "num_services_ok":"34", "num_services_pending":"0", "num_services_unknown":"0", "num_services_warn":"0" }] maybe there ist something wrong with ""? regards Timo
... View more
06-07-2017
01:55 PM
Hi Matt,
now i got only this output :
[{ "ctime":"${now()}", "host":"DUSTSADMIN.ads.xyz.de", "host_icons":"menu", "host_state":"UP", "num_services_crit":"0", "num_services_ok":"28", "num_services_pending":"0", "num_services_unknown":"0", "num_services_warn":"0" }, "ctime":"${now()}", "host":"DUSTSVMDC01.ads.xyz.de", "host_icons":"menu", "host_state":"UP", "num_services_crit":"0", "num_services_ok":"34", "num_services_pending":"0", "num_services_unknown":"0", "num_services_warn":"0" },{"ctime":"${now()}", "host":"DUSTSVMDC02.ads.xyz.de", "host_icons":"menu", "host_state":"UP", "num_services_crit":"0", "num_services_ok":"34", "num_services_pending":"0", "num_services_unknown":"0", "num_services_warn":"0" }] maybe there ist something wrong with ""?
regards
Timo
... View more
06-07-2017
11:43 AM
Hi,
we need add a default key:value with actual timestamp to the following json input structure: [ {
"host" : "DUSTSADMIN.ads.xyz.de",
"host_icons" : "menu",
"host_state" : "UP",
"num_services_crit" : "0",
"num_services_ok" : "28",
"num_services_pending" : "0",
"num_services_unknown" : "0",
"num_services_warn" : "0"
}, {
"host" : "DUSTSVMDC01.ads.xyz.de",
"host_icons" : "menu",
"host_state" : "UP",
"num_services_crit" : "0",
"num_services_ok" : "34",
"num_services_pending" : "0",
"num_services_unknown" : "0",
"num_services_warn" : "0"
}, {
"host" : "DUSTSVMDC02.ads.xyz.de",
"host_icons" : "menu",
"host_state" : "UP",
"num_services_crit" : "0",
"num_services_ok" : "34",
"num_services_pending" : "0",
"num_services_unknown" : "0",
"num_services_warn" : "0"
} ]
needed output: [ { "ctime" : "Current_timestamp",
"host" : "DUSTSADMIN.ads.xyz.de",
"host_icons" : "menu",
"host_state" : "UP",
"num_services_crit" : "0",
"num_services_ok" : "28",
"num_services_pending" : "0",
"num_services_unknown" : "0",
"num_services_warn" : "0"
}, {"ctime" : "Current_timestamp",
"host" : "DUSTSVMDC01.ads.xyz.de",
"host_icons" : "menu",
"host_state" : "UP",
"num_services_crit" : "0",
"num_services_ok" : "34",
"num_services_pending" : "0",
"num_services_unknown" : "0",
"num_services_warn" : "0"
}, {"ctime" : "Current_timestamp",
"host" : "DUSTSVMDC02.ads.xyz.de",
"host_icons" : "menu",
"host_state" : "UP",
"num_services_crit" : "0",
"num_services_ok" : "34",
"num_services_pending" : "0",
"num_services_unknown" : "0",
"num_services_warn" : "0"
} ]
... View more
Labels:
- Labels:
-
Apache NiFi
06-07-2017
11:04 AM
Hi,
we need to add a default key:value with actual timestamp to the following json input structure: [ {
"host" : "DUSTSADMIN.ads.xyz.de",
"host_icons" : "menu",
"host_state" : "UP",
"num_services_crit" : "0",
"num_services_ok" : "28",
"num_services_pending" : "0",
"num_services_unknown" : "0",
"num_services_warn" : "0"
}, {
"host" : "DUSTSVMDC01.ads.xyz.de",
"host_icons" : "menu",
"host_state" : "UP",
"num_services_crit" : "0",
"num_services_ok" : "34",
"num_services_pending" : "0",
"num_services_unknown" : "0",
"num_services_warn" : "0"
}, {
"host" : "DUSTSVMDC02.ads.xyz.de",
"host_icons" : "menu",
"host_state" : "UP",
"num_services_crit" : "0",
"num_services_ok" : "34",
"num_services_pending" : "0",
"num_services_unknown" : "0",
"num_services_warn" : "0"
} ]
needed output: [ { "ctime" : "Current_timestamp",
"host" : "DUSTSADMIN.ads.xyz.de",
"host_icons" : "menu",
"host_state" : "UP",
"num_services_crit" : "0",
"num_services_ok" : "28",
"num_services_pending" : "0",
"num_services_unknown" : "0",
"num_services_warn" : "0"
}, {"ctime" : "Current_timestamp",
"host" : "DUSTSVMDC01.ads.xyz.de",
"host_icons" : "menu",
"host_state" : "UP",
"num_services_crit" : "0",
"num_services_ok" : "34",
"num_services_pending" : "0",
"num_services_unknown" : "0",
"num_services_warn" : "0"
}, {"ctime" : "Current_timestamp",
"host" : "DUSTSVMDC02.ads.xyz.de",
"host_icons" : "menu",
"host_state" : "UP",
"num_services_crit" : "0",
"num_services_ok" : "34",
"num_services_pending" : "0",
"num_services_unknown" : "0",
"num_services_warn" : "0"
} ]
... View more
Labels:
- Labels:
-
Apache NiFi