Hi @bkosaraju, sorry, was my fault! Solution runs fine !!! My input data was wrong! Thanks, great solution. Timo ... View more

PutSolrStreamContent config ... View more

Hi @bkosaraju, i have some problems configuring filed mapping for each of the json fields. This is the json file structure : { "event_timestamp":"${time:substring(0,20):toDate('yyyy MMM dd HH:mm:ss'):format('yyyy-MM-dd HH:mm:ss')}", "host":"${message:substring(21):substringBefore('')}", "message":"${message:substring(21):substringAfter(' '):replace('"', '\\\\"')}" } this is my PutSolrContentStream config: Indexing failed ... ... View more

Hi @bkosaraju, thats what what I#m trying! i got the string from ListenSyslog and convert the string to json. example: {"event_timestamp":"2017-08-22 11:15:06","host":"meyer.devcon.com","message":"sshd[9411]: debug1: PAM: setting PAM_RHOST to monitoring-portal.devcon1.com"} The ingest runs fine, but a can't do sorting. We got this error: can not sort on multivalued field: event_timestamp. that's wy im asking for a solution to create a schema (HDP Search / Solr) with indexed single-valued fields (event_timestamp, host). regards Timo ... View more

Hi @bkosaraju, we need a realtime ingest. The syslog messages comes from hundreds of host ingested by Nifi ListenSyslog over PutSolrContentStream. regards Timo ... View more

Hi @Geoffrey Shelton Okot, my intension is to search all message sorted by timestamp or host. so, how can i create a schema (HDP Search / Solr) with indexed single-valued fields (event_timestamp, host). Sorting of messages is not necessary. Best regards Timo ... View more

Hi @Sonu Sahi Unfortunately the tutorial does not help me in this case. Using the Sample schemaless configuration uses creates multiValue filelds. In addition we ingest the document by nifi with PutSolrContentStream The sorting of the fields does not work like this. So we got this error: can not sort on multivalued field: event_timestamp regards Timo ... View more

Hi Vineet, i don't have the ability. ... View more

Hi Vineet, We use HDB 2.2 and have tried without success the hints of this link. We tried this without success: CREATE EXTERNAL TABLE kum.ext_checkmk_memory ( result_code integer, "result.step" integer, "result.start_time" integer, "result.end_time" integer, "result.curves[0].color" text, "result.curves[0].rrddata[0]" integer, "result.curves[0].rrddata[1]" integer, "result.curves[0].rrddata[2]" integer, "result.curves[0].rrddata[3]" integer, "result.curves[0].line_type" text, "result.curves[0].title" text, "result.curves[1].color" text, "result.curves[1].rrddata[0]" integer, "result.curves[1].rrddata[1]" integer, "result.curves[1].rrddata[2]" integer, "result.curves[1].rrddata[3]" integer, "result.curves[1].line_type" text, "result.curves[1].title" text, "result.curves[2].color" text, "result.curves[2].rrddata[0]" integer, "result.curves[2].rrddata[1]" integer, "result.curves[2].rrddata[2]" integer, "result.curves[2].rrddata[3]" integer, "result.curves[2].line_type" text, "result.curves[2].title" text, "result.curves[3].color" text, "result.curves[3].rrddata[0]" integer, "result.curves[3].rrddata[1]" integer, "result.curves[3].rrddata[2]" integer, "result.curves[3].rrddata[3]" integer, "result.curves[3].line_type" text, "result.curves[3].title" text ) LOCATION ( 'pxf://hdw1:8020/user/nifi/check_mk/memory/*?PROFILE=JSON' ) FORMAT 'CUSTOM' (formatter='pxfwritable_import' ) the problem are the objects in the curves array ... kum=# select * from kum.ext_checkmk_memory; result_code | result.step | result.start_time | result.end_time | result.curves[0].color | result.curves[0].rrddata[0] | result.curves[0].rrddata[1] | result.curves[0].rrddata[2] | result.curves[0].rrddata[3] | result.curves[0].line_type | result.curves[0].title | result .curves[1].color | result.curves[1].rrddata[0] | result.curves[1].rrddata[1] | result.curves[1].rrddata[2] | result.curves[1].rrddata[3] | result.curves[1].line_type | result.curves[1].title | result.curves[2].color | result.curves[2].rrddata[0] | result.curves[2].rrddata [1] | result.curves[2].rrddata[2] | result.curves[2].rrddata[3] | result.curves[2].line_type | result.curves[2].title | result.curves[3].color | result.curves[3].rrddata[0] | result.curves[3].rrddata[1] | result.curves[3].rrddata[2] | result.curves[3].rrddata[3] | result. curves[3].line_type | result.curves[3].title -------------+-------------+-------------------+-----------------+------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+----------------------------+------------------------+------- -----------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+----------------------------+------------------------+------------------------+-----------------------------+------------------------- ----+-----------------------------+-----------------------------+----------------------------+------------------------+------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+-------- --------------------+------------------------ 0 | 1800 | 1491174000 | 1491260400 | | | | | | | | | | | | | | | | | | | | | | | | | | | | ... View more

We access Json HDFS files as an external Table in HAWQ (external table:ext_chkmk_hosts). The "insert into table check_host ... select ..." loads the new data into a HAWQ table (check_hosts) in the same database. We want to schedule all the jobs with Nifi. 1. Load Data from System (InvokeHTTP) 2. Convert the JSON output (JoltTransformJSON) 3. add key:value (timestamp) (JoltTransformJSON) 4. store the data in HDFS (PutHDFS) 5. Access all HDFS JSON file in HAWQ (external table) 6. insert new data into HAWQ Table (insert into check_hosts ... select ... from ext_checkmk_hosts. ... View more