Member since
04-20-2018
5
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1424 | 04-08-2020 11:47 PM |
04-08-2020
11:47 PM
The cause has been identified; a wrong table property 'skip.header.line.count'='1' inherited from other CSV format tables that makes this strong behaviour on the ORC external table : missing lines, empty counts (due to null values ?), ...
... View more
04-08-2020
10:49 AM
Hi,
we are facing a strange issue with an ORC external Hive table, some lines can't be retrieved.
The context is HDP 3.1.0 with ORC files generated with a Spark 2.3.2 job into an HDFS partition with an external table mapped on this HDFS directory. Table partitions were added.
When trying to filtering this table with a specific column value, there's no result;but when accessing these files from the Spark interpreter of a Zeppelin notebook, the filter shows the expected lines.
Hive LLAP is not used for this issue and have another behaviour (data is retrieved).
The SQL query (JDBC tool or beeline command) that don't return any result :
select * from tracabilite.tg1pivot where sscc="330232926251080606" and `application`="LOGUSI" and month="04" and day="01";
Another SQL query that returns 2 results (within the same partition):
select * from tracabilite.tg1pivot where sscc="330232926636794272" and `application`="LOGUSI" and month="04" and day="01";
The Zeppelin notebook sample :
val logusi = spark.read.format("orc").option("header",true).load("/DEV/smart_data/TG/application=LOGUSI/year=2020/month=04/day=01") println(logusi.count) logusi.where(col("sscc")==="330232926251080606").show()
Please help to find and fix why some lines are not readable from Hive.
Do you have any tracks to follow, some hive/tez parameters to check or any bugs known ?
Thanks
Olivier
... View more
Labels:
04-20-2018
12:55 PM
Hi, I'm trying to configure a site-to-site connection from a Minifi to a Nifi (v1.1). When I'm trying to set the "Process" Input port "site-to-site" policies, I can't select it ! See the attachment, it is disabled ! But I can modify the site-to-site "Access policies". I'm logged in with an admin account. I don't know why ! Do I miss some Nifi configuration ? Olivier
... View more
Labels:
- Labels:
-
Apache NiFi