Cloudera Community
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
minskychen
user rank
Explorer

Re: Nifi Mysql to hive data format not match

by Explorer in Support Questions
‎03-28-2018 03:36 AM
‎03-28-2018 03:36 AM
@Shu Thank you, Shu ~ Your answer is very timely. And I decided to use String Format in Hive to avoid this problem~ ... View more

Re: Nifi Mysql to hive data format not match

by Explorer in Support Questions
‎03-28-2018 03:31 AM
‎03-28-2018 03:31 AM
@Rahul Soni Thanks for your reply ~ Your answer helps me a lot. I may use the "String" format to store my table in Hive. But I still have a worry, the "String" format may slow down the the execution speed of the data program in the future? ... View more

Re: Nifi Mysql to hive data format not match

by Explorer in Support Questions
‎03-28-2018 03:20 AM
‎03-28-2018 03:20 AM
@Matt Burgess Thanks for your reply. As your answer,I noticed the nifi BigInt problem. It`s true that "if the percision>19, it will be changed into string". the process happened in the "QueryDatabaseTable" processor. I read the source code, and find the function which they write to dell with the jdbc data. #QueryDatabaseTable https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/QueryDatabaseTable.java#L239 #QueryDatabaseTable import this class for data process https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java jdbcCommon.java Tanks a lot for your help~ But I don't know how to solve the problem gently, I might just use the String format in Hive. There also some place should notice, they deal the "TinyInt" and "Boolean" with the same way.The value "0" or "1" will be change into "false"/"true". Different databases have it's own design, we can't get a good way to deal with the Data transfer? ... View more

Nifi Mysql to hive data format not match

by Explorer in Support Questions
‎03-27-2018 06:42 AM
‎03-27-2018 06:42 AM
I use Nifi(QueryDatabaseTable=>PutHiveStreaming) processor to load table from mysql to Hive.But the data format is not match. eg: mysql column:`id` bigint(20) Hive column: id int I will get the erro log like below: 2018-03-27 11:59:22,607 WARN [put-hive-streaming-0] org.apache.hive.hcatalog.data.JsonSerDe Error [org.codehaus.jackson. JsonParseException: Current token (VALUE_STRING) not numeric, can not use numeric value accessors at [Source: java.io.ByteArrayInputStream@1afd4f32; line: 1, column: 9]] parsing json text [{"id": "3520", ...]. ... Caused by: org.codehaus.jackson.JsonParseException: Current token (VALUE_STRING) not numeric, can not use numeric value accessors at [Source: java.io.ByteArrayInputStream@4e94db44; line: 1, column: 9] As you see,The QueryDatabaseTable read the mysql int column as String,like [{"id": "3520", ...].I can't believe this.So I guess, is there something wrong with my operation？ Need Your Help~ ps, if I change the Hive column: id from "int" to "String",I can load the table into Hive smoothly~ Nifi Version : nifi-1.5.0.3.1.1.0-35 JDBC Version : mysql-connector-java-5.1.7.jar/mysql-connector-java-5.1.28.jar(I tried different JDBC version) ... View more
Labels:

Re: Knox sso for ambari and ranger does not work

by Explorer in Support Questions
‎03-20-2017 09:35 AM
‎03-20-2017 09:35 AM
I have meet the same problem,but I don`t know how to setup my own domain. May you have solved this problem,If you have some suggest will be will kind for me. Thanks! ... View more
Contact Me
Online
Offline
Last Visited
‎09-11-2018 04:02 AM
Community Statistics
Member Since ‎03-20-2017 08:18 AM
Last Visited ‎09-11-2018 04:02 AM
Posts 5