Member since
07-14-2017
5
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
450 | 11-04-2017 12:18 AM |
11-04-2017
12:18 AM
@Matt Burgess JDBC spec does state that REAL maps to (single-precision) float; FLOAT maps to double precision, thus the mapping to in avro should be to double not float, reference table B-1 page 190: http://download.oracle.com/otn-pub/jcp/jdbc-4_3-mrel3-eval-spec/jdbc4.3-fr-spec.pdf So the following code fix resolved the issue. https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java#L527 original case FLOAT:
case REAL:
builder.name(columnName).type().unionOf().nullBuilder().endNull().and().floatType().endUnion().noDefault();
break;
case DOUBLE:
builder.name(columnName).type().unionOf().nullBuilder().endNull().and().doubleType().endUnion().noDefault();
break; Modified case REAL:
builder.name(columnName).type().unionOf().nullBuilder().endNull().and().floatType().endUnion().noDefault();
break;
case FLOAT:
case DOUBLE:
builder.name(columnName).type().unionOf().nullBuilder().endNull().and().doubleType().endUnion().noDefault();
break; Thanks to @Ron Mahoney for finding this issue.
... View more
11-03-2017
11:13 PM
Tested simple Nifi flow ExecuteSQL -> PutDatabasRecord ExecuteSQL reads data from table (named decimal_test) having columns with decimal data type (from Postgres database). The output flowfile (avro file) from ExecuteSQL passed to PutDatabaseRecord which loads the data to the same table (decimal_test) in the Postgres database. ExecuteSQL property 'user avro logical types' is set to true. Here is the result: Decimal column Original Table values in Postgres database Table migrated from Postgres to Postgres using Nifi decimal(18,0) 123456789012345678 123456789012346000 decimal(18,4) 12345678901234.1234 12345678901234.1000 decimal(36,0) 123456789012345678901234567890123456 123456789012346000000000000000000000 decimal(36,4) 12345678901234567890123456789012.1234 12345678901234600000000000000000.0000 Is is a bug in Nifi or am I missing something here..? Please advice. Thanks.
... View more
Labels:
10-30-2017
06:44 PM
@Matt Burgess Thanks for the explanation and suggestions.
... View more
10-30-2017
06:37 PM
@Shu Thanks for the suggestions. The current process is using this Cast to varchar(), but I was wondering if we can get the data as float in AVRO.
... View more
10-27-2017
10:26 PM
Using the ExecuteSQL processor (nifi 1.3) to fetch table from Teradata Database. The processor gives error (org.apache.avro.unresolvedUnionException: not in union ["null","float"]) when the table in the Teradata database has the 'Float' datatype. The process is good when fetching the same table (float datatype) from Postgres database. Please Advise..! Thanks.
... View more
Labels: