About timo_burmeister

timo_burmeister · ‎08-24-2017

Hi @bkosaraju, sorry, was my fault! Solution runs fine !!! My input data was wrong! Thanks, great solution. Timo

timo_burmeister · ‎08-23-2017

PutSolrStreamContent config

timo_burmeister · ‎08-23-2017

Hi @bkosaraju, i have some problems configuring filed mapping for each of the json fields. This is the json file structure : { "event_timestamp":"${time:substring(0,20):toDate('yyyy MMM dd HH:mm:ss'):format('yyyy-MM-dd HH:mm:ss')}", "host":"${message:substring(21):substringBefore('')}", "message":"${message:substring(21):substringAfter(' '):replace('"', '\\\\"')}" } this is my PutSolrContentStream config: Indexing failed ...

timo_burmeister · ‎08-22-2017

Hi @bkosaraju, thats what what I#m trying! i got the string from ListenSyslog and convert the string to json. example: {"event_timestamp":"2017-08-22 11:15:06","host":"meyer.devcon.com","message":"sshd[9411]: debug1: PAM: setting PAM_RHOST to monitoring-portal.devcon1.com"} The ingest runs fine, but a can't do sorting. We got this error: can not sort on multivalued field: event_timestamp. that's wy im asking for a solution to create a schema (HDP Search / Solr) with indexed single-valued fields (event_timestamp, host). regards Timo

timo_burmeister · ‎08-22-2017

Hi @bkosaraju, we need a realtime ingest. The syslog messages comes from hundreds of host ingested by Nifi ListenSyslog over PutSolrContentStream. regards Timo

timo_burmeister · ‎08-22-2017

Hi @Geoffrey Shelton Okot, my intension is to search all message sorted by timestamp or host. so, how can i create a schema (HDP Search / Solr) with indexed single-valued fields (event_timestamp, host). Sorting of messages is not necessary. Best regards Timo

timo_burmeister · ‎08-22-2017

Hi @Sonu Sahi Unfortunately the tutorial does not help me in this case. Using the Sample schemaless configuration uses creates multiValue filelds. In addition we ingest the document by nifi with PutSolrContentStream The sorting of the fields does not work like this. So we got this error: can not sort on multivalued field: event_timestamp regards Timo

timo_burmeister · ‎08-21-2017

Hi, we store all of out system syslog files in our Data Lake. For realtime Monitoring we have to store the syslogs with SOLR. The syslog structure is easy: example: timestamp, hostname, message time | host | message ---------------------+------------------------+---------------------------------------------- 2017-08-08 19:39:48 | xxxx.xxx.com | systemd[4554]: Stopped target Default. 2017-08-09 09:54:57 | yxz.gsj.com| systemd[28096]: Starting Shutdown. 2017-08-09 14:48:39 | yxz.gsj.com | systemd[22015]: Stopping Timers. 2017-08-09 12:12:37 | yxz.gsj.com| systemd[1]: Started User Manager for UID 0. 2017-08-10 00:00:37 | xxxx.xxx.com | systemd[15736]: Stopping Basic System. How can we build a schema in SOLR with thew possibility to search in all fields and do sorting the output by time, host or message? Thanks for you help. Timo

timo_burmeister · ‎08-16-2017

Got the solution: 1. Replace String with actual year search value: <(.*?)> Replacement Value: ${now():format('yyyy ')} 2. Extract Timestamp and Message time = (^.{20}) message = (^.*$) 3. Replace String with insert into clause: search value: (?s)(^.*$) Replacement Value: insert into syslog values ('${time:substring(0,20):toDate('yyyy MMM dd HH:mm:ss'):format('yyyy-MM-dd HH:mm:ss')}','${message:substring(21):substringBefore(' ')}','${message:substring(21):substringAfter(' ')}');

timo_burmeister · ‎08-14-2017

Hi, i have the following input single row FlowFiles: Aug 9 10:34:12 sysxyz-p2 sudo: pam_unix(sudo:session): session opened for user root by (uid=0) Aug 10 10:34:12 sysxxx.xxx.xx sudo: pam_unix(sudo:session): session opened for user root by (uid=0) Aug 11 10:34:12 sysxxx.xxx.xx sudo: pam_unix(sudo:session): session closed for user root Aug 12 10:34:12 sysxxx.xxx.xyz sudo: pam_unix(sudo:session): session closed for user root Aug 13 10:34:01 sysxxx.xxx.xx CRON[26312]: pam_unix(cron:session): session opened for user monadmin by (uid=0) 1. Split the Flow into timestamp, host, message fields: timestamp host message Aug 9 10:34:12 sysxyz-p2 sudo: pam_unix(sudo:session): session opened for user root by (uid=0) Aug 10 10:34:12 sysxxx.xxx.xx sudo: pam_unix(sudo:session): session opened for user root by (uid=0) Aug 11 10:34:12 sysxxx.xxx.xx sudo: pam_unix(sudo:session): session closed for user root Aug 12 10:34:12 sysxxx.xxx.xyz sudo: pam_unix(sudo:session): session closed for user root Aug 13 10:34:01 sysxxx.xxx.xx CRON[26312]: pam_unix(cron:session): session opened for user monadmin by (uid=0) 2. Convert timestamp Format: Aug 9 10:34:12 - > 'YYYY.MM.DD HH:MI:SS' 3. Concat the extracted files to insert into ... string: example: insert into table ('2017-08-09 10:34:12', 'sysxyz-p2', 'sudo: pam_unix(sudo:session): session opened for user root by (uid=0)'; Thanks for you help! Timo

timo_burmeister · ‎08-11-2017

Great, works !!! Thanks!

timo_burmeister · ‎08-11-2017

Hi Bryan, how can i merge the FlowData Files to a multiline File? Using the MergeCentent processor i got a concatenated line. input: 1 FlowData: <30>Aug 11 06:27:26 xxx.xxx.com systemd[28892]: Stopping Timers. 2 FlowData:<30>Aug 11 06:27:15 xxx.xxx.com systemd[24517]: Stopping Paths etc... MergeContent output: <30>Aug 11 06:27:26 xxx.xxx.com systemd[28892]: Stopping Timers.<30>Aug 11 06:27:15 xxx.xxx.com systemd[24517]:Stopping Paths I need the following multiline structure: <30>Aug 11 06:27:26 xxx.xxx.com systemd[28892]: Stopping Timers. <30>Aug 11 06:27:15 xxx.xxx.com systemd[24517]:Stopping Paths any solution? regards Timo

timo_burmeister · ‎08-09-2017

Thanks, this was really easy. I just generating "insert into table" statements including the Flow File with the ReplaceText processor and connect it with the PutSQL processor. regards Timo

timo_burmeister · ‎08-08-2017

Hi, 1. step we ingest SYSLOG messages with the ListenSyslog and PutHDFS in our DataLake. 2. step ingest with PXF external table technology into HAWQ. CREATE EXTERNAL TABLE ext_syslog_hist (message TEXT) LOCATION ('pxf://xxx.xxx.com:8020/apps/nifi/syslog/history/*?PROFILE=HdfsTextSimple') FORMAT 'TEXT' (delimiter=E'\t'); Every minute there are thousands of small (100 bytes) files created in the HDFS. So I'm looking for a way to ingest the incoming ListenSyslog directly into a HAWQ table. Regards Timo

timo_burmeister · ‎07-13-2017

timo_burmeister · ‎06-21-2017

Hi, we get every day a dynamic list if host with the InvokeHTTP processor from our source system. On this base we have to request the source system again with InvokeHTTP host by host to get detailed data. 1. InvokeHTTP Request: https://xyz.abc.com/prod/check_mk/view.py?view_name=hostgroup&hostgroup=vmware-host&limit=hard&output_format=json&_username=bi&_secret=XXXXXXXXX Output : [{"host_state":"UP","host“:“xxx.yyy.com“,“host_icons":"menu","num_services_ok":"28","num_services_warn":"0","num_services_unknown":"0","num_services_crit":"0","num_services_pending":"0","ctime":"Wed Jun 21 06:00:02 CEST 2017"},{"host_state":"UP","host“:“yyy.yyy.com“,“host_icons":"menu","num_services_ok":"34","num_services_warn":"0","num_services_unknown":"0","num_services_crit":"0","num_services_pending":"0","ctime":"Wed Jun 21 06:00:02 CEST 2017"},{"host_state":"UP","host“:“zzz.yyy.com“,“host_icons":"menu","num_services_ok":"34","num_services_warn":"0","num_services_unknown":"0","num_services_crit":"0","num_services_pending":"0","ctime":"Wed Jun 21 06:00:02 CEST 2017“}] From this output we have to grep the host values (“xxx.yyy.com“, ....) and build an request for every host: 2. InvokeHTTP Requests https://xyz.abc.com/prod/check_mk/webapi.py?action=get_graph&_username=ansible&_secret=XXXXXXXXX&request={"specification":["template",{"service_description":"Filesystem /","site":"prod","graph_index":0,"host_name":"xxx.yyy.com"}],"data_range":{"time_range":[1491174000,1491260340]}} https://xyz.abc.com/prod/check_mk/webapi.py?action=get_graph&_username=ansible&_secret=XXXXXXXXX&request={"specification":["template",{"service_description":"Filesystem /","site":"prod","graph_index":0,"host_name":"yyy.yyy.com"}],"data_range":{"time_range":[1491174000,1491260340]}} https://xyz.abc.com/prod/check_mk/webapi.py?action=get_graph&_username=ansible&_secret=XXXXXXXXX&request={"specification":["template",{"service_description":"Filesystem /","site":"prod","graph_index":0,"host_name":"zzz.yyy.com"}],"data_range":{"time_range":[1491174000,1491260340]}} ... How can we challenge this? Thanks Timo

timo_burmeister · ‎06-20-2017

Thanks Matt! Works perfekt ...

timo_burmeister · ‎06-20-2017

Hi Vineet, i don't have the ability.

timo_burmeister · ‎06-19-2017

i have the following son structure and want to replace the special character string "[{" and "]}" with "{" and "}" : {"result": {"step": 1800, "start_time": 1491174000, "end_time": 1491260400, "curves": [{"color": "#a05830", "rrddata": [0.603695, 1.06903, 0.94504, 0.68786, 31.3228, 0.316447, 0.808407, 0.247655, 0.174552, 0.123072, 0.62, 0.0689, 0.30758, 0.0869783, 0.14478, 0.305993, 0.808873, 0.193055, 0.113133, 0.46116, 8.047, 1.88388, 2.62721, 0.770247, 8.06144, 2.25591, 22.3061, 57.5539, 0.270233, 1.50602, 0.819887, 5.90425, 0.43361, 0.526907, 2.46678, 0.759873, 0.451133, 0.25843, 0.224033, 0.661373, 1.1279, 0.348587, 0.277142, 0.06647, 0.16693, 0.06225, 0.0588483, 0.08057], "line_type": "area", "title": "Disk utilization"}]}, "result_code": 0}

timo_burmeister · ‎06-16-2017

Hi Vineet, We use HDB 2.2 and have tried without success the hints of this link. We tried this without success: CREATE EXTERNAL TABLE kum.ext_checkmk_memory ( result_code integer, "result.step" integer, "result.start_time" integer, "result.end_time" integer, "result.curves[0].color" text, "result.curves[0].rrddata[0]" integer, "result.curves[0].rrddata[1]" integer, "result.curves[0].rrddata[2]" integer, "result.curves[0].rrddata[3]" integer, "result.curves[0].line_type" text, "result.curves[0].title" text, "result.curves[1].color" text, "result.curves[1].rrddata[0]" integer, "result.curves[1].rrddata[1]" integer, "result.curves[1].rrddata[2]" integer, "result.curves[1].rrddata[3]" integer, "result.curves[1].line_type" text, "result.curves[1].title" text, "result.curves[2].color" text, "result.curves[2].rrddata[0]" integer, "result.curves[2].rrddata[1]" integer, "result.curves[2].rrddata[2]" integer, "result.curves[2].rrddata[3]" integer, "result.curves[2].line_type" text, "result.curves[2].title" text, "result.curves[3].color" text, "result.curves[3].rrddata[0]" integer, "result.curves[3].rrddata[1]" integer, "result.curves[3].rrddata[2]" integer, "result.curves[3].rrddata[3]" integer, "result.curves[3].line_type" text, "result.curves[3].title" text ) LOCATION ( 'pxf://hdw1:8020/user/nifi/check_mk/memory/*?PROFILE=JSON' ) FORMAT 'CUSTOM' (formatter='pxfwritable_import' ) the problem are the objects in the curves array ... kum=# select * from kum.ext_checkmk_memory; result_code | result.step | result.start_time | result.end_time | result.curves[0].color | result.curves[0].rrddata[0] | result.curves[0].rrddata[1] | result.curves[0].rrddata[2] | result.curves[0].rrddata[3] | result.curves[0].line_type | result.curves[0].title | result .curves[1].color | result.curves[1].rrddata[0] | result.curves[1].rrddata[1] | result.curves[1].rrddata[2] | result.curves[1].rrddata[3] | result.curves[1].line_type | result.curves[1].title | result.curves[2].color | result.curves[2].rrddata[0] | result.curves[2].rrddata [1] | result.curves[2].rrddata[2] | result.curves[2].rrddata[3] | result.curves[2].line_type | result.curves[2].title | result.curves[3].color | result.curves[3].rrddata[0] | result.curves[3].rrddata[1] | result.curves[3].rrddata[2] | result.curves[3].rrddata[3] | result. curves[3].line_type | result.curves[3].title -------------+-------------+-------------------+-----------------+------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+----------------------------+------------------------+------- -----------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+----------------------------+------------------------+------------------------+-----------------------------+------------------------- ----+-----------------------------+-----------------------------+----------------------------+------------------------+------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+-------- --------------------+------------------------ 0 | 1800 | 1491174000 | 1491260400 | | | | | | | | | | | | | | | | | | | | | | | | | | | |

timo_burmeister · ‎06-16-2017

Hi, we want to access complex json data via a hawq external table. JSON structure: { "result": { "step": 1800, "start_time": 1491174000, "end_time": 1491260400, "curves": [ { "color": "#f0f0f0", "rrddata": [ 33672800000, 33672800000, 33672800000, 33672800000 ], "line_type": "area", "title": "RAM installed" }, { "color": "#e0e0e0", "rrddata": [ 4302300000, 4302300000, 4302300000, 4302300000 ], "line_type": "stack", "title": "Swap installed" }, { "color": "#80ff40", "rrddata": [ 16683600000, 16690000000, 16690900000, 16696200000 ], "line_type": "area", "title": "RAM used" }, { "color": "#408f20", "rrddata": [ 1088450000, 1088480000, 1087960000, 1088710000 ], "line_type": "stack", "title": "Swap used" } ] }, "result_code": 0 }

timo_burmeister · ‎06-09-2017

We access Json HDFS files as an external Table in HAWQ (external table:ext_chkmk_hosts). The "insert into table check_host ... select ..." loads the new data into a HAWQ table (check_hosts) in the same database. We want to schedule all the jobs with Nifi. 1. Load Data from System (InvokeHTTP) 2. Convert the JSON output (JoltTransformJSON) 3. add key:value (timestamp) (JoltTransformJSON) 4. store the data in HDFS (PutHDFS) 5. Access all HDFS JSON file in HAWQ (external table) 6. insert new data into HAWQ Table (insert into check_hosts ... select ... from ext_checkmk_hosts.

timo_burmeister · ‎06-08-2017

Hi, how can we implement the following Postgres SQL Statement in Nifi?: insert into kum.checkmk_hosts ( ctime, host_state, host, host_icons, num_services_ok, num_services_warn, num_services_unknown, num_services_crit, num_services_pending ) select ctime::timestamp, host_state, host, host_icons, num_services_ok, num_services_warn, num_services_unknown, num_services_crit, num_services_pending from kum.ext_checkmk_hosts where date(ctime) >= current_date;

timo_burmeister · ‎06-07-2017

Hi Matt, after upgrading to nifi 1.2 all works fine !!! Great work! Thanks

timo_burmeister · ‎06-07-2017

I think my problem my version is : nifi-1.1.2

timo_burmeister · ‎06-07-2017

Hi Matt, now i got only this output : [{ "ctime":"${now()}", "host":"DUSTSADMIN.ads.xyz.de", "host_icons":"menu", "host_state":"UP", "num_services_crit":"0", "num_services_ok":"28", "num_services_pending":"0", "num_services_unknown":"0", "num_services_warn":"0" }, "ctime":"${now()}", "host":"DUSTSVMDC01.ads.xyz.de", "host_icons":"menu", "host_state":"UP", "num_services_crit":"0", "num_services_ok":"34", "num_services_pending":"0", "num_services_unknown":"0", "num_services_warn":"0" },{"ctime":"${now()}", "host":"DUSTSVMDC02.ads.xyz.de", "host_icons":"menu", "host_state":"UP", "num_services_crit":"0", "num_services_ok":"34", "num_services_pending":"0", "num_services_unknown":"0", "num_services_warn":"0" }] maybe there ist something wrong with ""? regards Timo

timo_burmeister · ‎06-07-2017

Hi Matt, now i got only this output : [{ "ctime":"${now()}", "host":"DUSTSADMIN.ads.xyz.de", "host_icons":"menu", "host_state":"UP", "num_services_crit":"0", "num_services_ok":"28", "num_services_pending":"0", "num_services_unknown":"0", "num_services_warn":"0" }, "ctime":"${now()}", "host":"DUSTSVMDC01.ads.xyz.de", "host_icons":"menu", "host_state":"UP", "num_services_crit":"0", "num_services_ok":"34", "num_services_pending":"0", "num_services_unknown":"0", "num_services_warn":"0" },{"ctime":"${now()}", "host":"DUSTSVMDC02.ads.xyz.de", "host_icons":"menu", "host_state":"UP", "num_services_crit":"0", "num_services_ok":"34", "num_services_pending":"0", "num_services_unknown":"0", "num_services_warn":"0" }] maybe there ist something wrong with ""? regards Timo

timo_burmeister · ‎06-07-2017

Hi, we need add a default key:value with actual timestamp to the following json input structure: [ { "host" : "DUSTSADMIN.ads.xyz.de", "host_icons" : "menu", "host_state" : "UP", "num_services_crit" : "0", "num_services_ok" : "28", "num_services_pending" : "0", "num_services_unknown" : "0", "num_services_warn" : "0" }, { "host" : "DUSTSVMDC01.ads.xyz.de", "host_icons" : "menu", "host_state" : "UP", "num_services_crit" : "0", "num_services_ok" : "34", "num_services_pending" : "0", "num_services_unknown" : "0", "num_services_warn" : "0" }, { "host" : "DUSTSVMDC02.ads.xyz.de", "host_icons" : "menu", "host_state" : "UP", "num_services_crit" : "0", "num_services_ok" : "34", "num_services_pending" : "0", "num_services_unknown" : "0", "num_services_warn" : "0" } ] needed output: [ { "ctime" : "Current_timestamp", "host" : "DUSTSADMIN.ads.xyz.de", "host_icons" : "menu", "host_state" : "UP", "num_services_crit" : "0", "num_services_ok" : "28", "num_services_pending" : "0", "num_services_unknown" : "0", "num_services_warn" : "0" }, {"ctime" : "Current_timestamp", "host" : "DUSTSVMDC01.ads.xyz.de", "host_icons" : "menu", "host_state" : "UP", "num_services_crit" : "0", "num_services_ok" : "34", "num_services_pending" : "0", "num_services_unknown" : "0", "num_services_warn" : "0" }, {"ctime" : "Current_timestamp", "host" : "DUSTSVMDC02.ads.xyz.de", "host_icons" : "menu", "host_state" : "UP", "num_services_crit" : "0", "num_services_ok" : "34", "num_services_pending" : "0", "num_services_unknown" : "0", "num_services_warn" : "0" } ]

timo_burmeister · ‎06-07-2017

Hi, we need to add a default key:value with actual timestamp to the following json input structure: [ { "host" : "DUSTSADMIN.ads.xyz.de", "host_icons" : "menu", "host_state" : "UP", "num_services_crit" : "0", "num_services_ok" : "28", "num_services_pending" : "0", "num_services_unknown" : "0", "num_services_warn" : "0" }, { "host" : "DUSTSVMDC01.ads.xyz.de", "host_icons" : "menu", "host_state" : "UP", "num_services_crit" : "0", "num_services_ok" : "34", "num_services_pending" : "0", "num_services_unknown" : "0", "num_services_warn" : "0" }, { "host" : "DUSTSVMDC02.ads.xyz.de", "host_icons" : "menu", "host_state" : "UP", "num_services_crit" : "0", "num_services_ok" : "34", "num_services_pending" : "0", "num_services_unknown" : "0", "num_services_warn" : "0" } ] needed output: [ { "ctime" : "Current_timestamp", "host" : "DUSTSADMIN.ads.xyz.de", "host_icons" : "menu", "host_state" : "UP", "num_services_crit" : "0", "num_services_ok" : "28", "num_services_pending" : "0", "num_services_unknown" : "0", "num_services_warn" : "0" }, {"ctime" : "Current_timestamp", "host" : "DUSTSVMDC01.ads.xyz.de", "host_icons" : "menu", "host_state" : "UP", "num_services_crit" : "0", "num_services_ok" : "34", "num_services_pending" : "0", "num_services_unknown" : "0", "num_services_warn" : "0" }, {"ctime" : "Current_timestamp", "host" : "DUSTSVMDC02.ads.xyz.de", "host_icons" : "menu", "host_state" : "UP", "num_services_crit" : "0", "num_services_ok" : "34", "num_services_pending" : "0", "num_services_unknown" : "0", "num_services_warn" : "0" } ]

timo_burmeister · ‎06-06-2017

Hi Matt, great solution!!! Thanks Timo

Online	Offline
Last Visited	‎12-13-2021 12:06 PM

Date Registered	‎01-15-2016 03:00 PM
Last Visited	‎12-13-2021 12:06 PM
Total Messages Posted	34

Re: Nifi: How to split , convert and concat dynami...

Re: How can we build a SOLR Schema für SYSLOGS wit...

Re: How can we build a SOLR Schema für SYSLOGS wit...

Re: How can we build a SOLR Schema für SYSLOGS wit...

Re: How can we build a SOLR Schema für SYSLOGS wit...

Re: How can we build a SOLR Schema für SYSLOGS wit...

Re: How can we build a SOLR Schema für SYSLOGS wit...

Re: How can we build a SOLR Schema für SYSLOGS wit...

How can we build a SOLR Schema für SYSLOGS with so...

Re: Nifi: How to split , convert and concat dynami...

Nifi: How to split , convert and concat dynamic St...

Re: Nifi ListenSyslog howto ingest into HAWQ

Re: Nifi ListenSyslog howto ingest into HAWQ

Re: Nifi ListenSyslog howto ingest into HAWQ

Nifi ListenSyslog howto ingest into HAWQ

How to get syslogs from different syslogd Servern...

Nifi InvokeHttp: How to work with a list of Remote...

Re: NiFi Replace Text: how t0 replace string " [{"...

Re: How to create HAWQ external table with complex...

NiFi Replace Text: how t0 replace string " [{" wit...

Re: How to create HAWQ external table with complex...

How to create HAWQ external table with complex jso...

Re: Nifi - Postgres SQL insert into table as selec...

Nifi - Postgres SQL insert into table as select ?

Re: Nifi - How to add key:value to json

Re: Nifi - How to add key:value to json

Re: Nifi - How to add key:value to json

Re: Nifi - How to add key:value to json

Nifi - How to add key:value to json

Nifi - How to add key:value to json

Re: How tho convert JSON format