Member since
10-06-2015
273
Posts
202
Kudos Received
81
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3369 | 10-11-2017 09:33 PM | |
2877 | 10-11-2017 07:46 PM | |
2150 | 08-04-2017 01:37 PM | |
1899 | 08-03-2017 03:36 PM | |
1802 | 08-03-2017 12:52 PM |
10-03-2023
03:35 PM
@mburgess I am trying to run a merge query in PUTSQL after using ExecuteSQL to pull from source database. I then pass the results through ConvertAvrotoJSON and then to convertJSONToSQL (set to INSERT since there is no UPSERT for SQL server database). I want to now run the merge query to update if the primary key is matched or insert if not, but I have been getting all sorts of errors Input flowfile has the following attributes sql.args.1.type 12 sql.args.1.value A-001 sql.args.2.type 12 sql.args.2.value DDD sql.args.3.type 12 sql.args.3.value 2B sql.args.4.type 12 sql.args.4.value AAA sql.args.5.type 12 sql.args.5.value 2B sql.args.6.type 12 sql.args.6.value BBB sql.args.7.type 93 sql.args.7.value 1954-08-01 22:46:50.73 sql.catalog CUSDB sql.table CUSTOMERS uuid 50a3daf9-e5d8-422b-b6cb-9c92da723c55 I created the following attributes in UpdateAtributes I directed this flowfile to UpdateAttribute where I created above properties like this sql.args.1.type = 12; sql.args.1.value = ${vkey} sql.args.2.type = 12; sql.args.2.value = ${AP} sql.args.3.type = 12; sql.args.3.value = ${BP} ........ sql.args.7.type = 93; sql.args.7.value = ${LAST_MOD_DATE} But when I check the results of the queue of the UpdateAtributes, I see that sql.args.1.type, sql.args.2.type, ... sql.args.7.type return blank values. I am doing something wrong in the way I have defined those properties? Merge query in PutSQL looks like this MERGE INTO RCDB.dbo.RCDB_WELLHEADER_HZ_NIFI_V2 as t USING (select ? as vkey, ? as ap, ? as bp, ? as lm, ? as lnum, ? as wn, ? as LAST_MOD_DATE) s ON t.EKEY = s.EKEY WHEN MATCHED THEN UPDATE SET t.ap = s.ap, t.bp = s.bp, t.lm = s.lm, t.lnum = s.lnum, t.wn = s.wn, t.LAST_MOD_DATE = s.LAST_MOD_DATE WHEN NOT MATCHED THEN INSERT (vkey, ap, bp, lm, lnum, wn, LAST_MOD_DATE) VALUES (?, ?, ?, ?, ?, ?, to_date(?, 'YYYY-MM-DD HH34:MI:SS')) I get the error that all parameters are blank. Can someone please point me in the direction to resolve this issue. Why are the attributes going blank when I set them in UpdateAttributes?
... View more
08-05-2021
09:17 PM
Hi @vnandigam Good news. Now Spark Atlas integration is supported using CDP cluster. References: 1. https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/atlas-reference/topics/atlas-spark-metadata-collection.html 2. https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-hdp/topics/amb-enable-spark-cm.html
... View more
01-20-2021
07:15 AM
@egarelnabi Hi Im trying this method and I'm also getting Heart Beat message continously. Any idea ? Thanks
... View more
01-12-2021
05:21 AM
@CristoE Since this question already has an accepted solution and is specific to DISTCP replacement for HDFS, It would be much better to start an entirely new question in the community. You can always add a link to this question solution as reference in your new question post. You would get more visibility that way and we would not dilute the answer to this question with suggestions related to ADLS rather then HDFS.
... View more
04-17-2020
02:24 PM
They are actually not the same. SORT BY sorts data inside partition, while ORDER BY is global sort. SORT BY calls sortWithinPartitions() function, while ORDER BY calls sort() Both of these functions call sortInternal(), but with different global flag: def sortWithinPartitions ... sortInternal(global = false, sortExprs) def sort ... sortInternal(global = true, sortExprs)
... View more
04-12-2020
05:25 AM
Below is the very good article for differences between hadoop 2.x and Hadoop 3.x Difference Between Hadoop 2 and Hadoop 3
... View more
12-09-2019
05:05 PM
i found the link with bit of google search ... https://atlas.apache.org/0.8.4/Import-Export-API.html hope that is the one you referring.
... View more
11-15-2019
02:55 PM
Hello, I am begineer on this topic. I want to connect my dataflow to store data in postgresql and google found your topic in the forum. I am trying all the things you said and the service remains "Enabling". Could you explain exactly the steps to enable this communication. From my side: 1. Download postgresql-42.2.8.jar. 2. Move to C:\Program Files\nifi-1.9.2\lib 3. Restart Nifi. 4. Configure DBCPConnectionPool with the following parameters: URL: jdbc:postgresql://127.0.0.1:5432/nifi Diver Class: org/postgresql/Driver.class Driver Location: file:////C:/Program Files/nifi-1.9.2/lib/postgresql-42.2.8.jar Is something wrong? Could you help me? The service remains "ENABLING" Regards,
... View more
08-14-2017
02:46 PM
Thanks Matt, Interesting approach and makes a lot of sense to do things that way. I'll give it a try. Thanks for your help.
... View more
08-03-2017
03:36 PM
4 Kudos
@Marc Parmentier The date format is as follows: {yyyy}-{mm}-{dd}T{hh}:{mm}:{ss}.{zzz}Z {year}-{month}-{day}T{hours}:{minutes}:{seconds}.{timezone_id}Z e.g. 2017-04-18T18:49:44.000Z
... View more