question Re: Remove Duplicate Record values without cache services in Support Questions

Remove Duplicate Record values without cache services

Gtec — Thu, 05 May 2022 05:10:45 GMT

Hi All, Good Morning!

Please help me in removing Duplicate Records without using cache services (Apache NIFI)Error Log

Re: Remove Duplicate Record values without cache services

SAMSAL — Thu, 05 May 2022 19:57:24 GMT

Not sure if this is the best way but try using QueryRecord and create one dynamic property to funnel the data into where your query will look like this:

Select distinct * from FLOWFILE

Re: Remove Duplicate Record values without cache services

Gtec — Fri, 06 May 2022 13:26:05 GMT

Thanks for the valuable information and I got the results , but the Date and Error Message are not sorted/ordered properly.

Do We have any particular Function on Sql Query or any processor to sort in Order for the same

Re: Remove Duplicate Record values without cache services

SAMSAL — Fri, 06 May 2022 17:20:12 GMT

have you tried Selected distinct * order by Date, ERROR_MSG

Re: Remove Duplicate Record values without cache services

Gtec — Mon, 09 May 2022 07:00:09 GMT

No, Not yet and I am waiting from the client end for confirmation since my clients are abroad and they have access for the same, Would check and let you know the results by tomorrow 10th May 2022. Thanks for the help SAMSAL.

Re: Remove Duplicate Record values without cache services

Gtec — Tue, 10 May 2022 10:15:12 GMT

Thanks for the help SAMSAL and my next task is to insert these values into the table. Please let me know which processor would help me.

Re: Remove Duplicate Record values without cache services

SAMSAL — Tue, 10 May 2022 19:28:42 GMT

there are a lot of processors that can help you with that for example : PUTSQL, PutDatabaseRecord, ExecuteSQL

Re: Remove Duplicate Record values without cache services

Gtec — Wed, 11 May 2022 03:48:01 GMT

Sure SAMSAL, Once the table is created I would test the same with above mentioned. Which is HiveDB Table.

Re: Remove Duplicate Record values without cache services

Gtec — Wed, 11 May 2022 09:10:34 GMT

The table has been created, I have to perform Insert Statement on PutHiveStreaming Processor, Insert those Extracted Error Log Data. I am viewing results on Google and If I need any Help will get back here for sure, Thanks

Re: Remove Duplicate Record values without cache services

Gtec — Thu, 12 May 2022 04:04:35 GMT

Please help me in Inserting these data into Hive DB. (Process flow). Hive doesn't accept CSV and it has to convert to Avro. I am stuck on how to Put Insert Statement and direct to PutHiveStreaming.

Re: Remove Duplicate Record values without cache services

SAMSAL — Thu, 12 May 2022 20:31:50 GMT

I never worked with Hive DB, but if it only takes avro format and you have csv you can use processor ConvertRecrod to convert a record from one format to another. In this processor you create CSV Reader service and the writer is Avro

Re: Remove Duplicate Record values without cache services

Gtec — Fri, 13 May 2022 04:28:17 GMT

Sure SAMSOL, In Hive DB, I am having a table and no need for any SQL INSERT STATEMENT.

Say HIVE DB Table has XXX, in that I have to put these Data and Error Field, Just convert to Avro works or ??? have to call insert statement processor