About DianaTorres

Eren · ‎04-21-2023

i fix this issue by run command on destination cluster,i think it caused by original version is too old to support ec (hadoop-2.7.5)

DataEngAa · ‎04-20-2023

Thank you @mszurap for you response . I tried the suggested work around already and it seems like the issue still persists . I agree the table has lot of partitions but I am pretty sure the code times out before 5 mins . I have also tried enforcing the hive-site.xml with the updated timeout which also did not help much. Only thing which worked was adding spark.catalog.recoverPartitions(table) before issuing the drop partition command . I am really not sure as why recovering the partitions in the catalog eliminated the metastore warning . Below is the updated code which is working without any warning : spark.sql.catalog.recoverPartitions(orders) spark.sql("alter table orders drop if exists partition(year=2023)") data.write.mode('Overwrite').parquet(hdfsPath) Any help here in understanding the problem will be much appreciated .

mszurap · ‎04-19-2023

Glad to hear that. 🙂 The "hive.stats.autogather" should be unrelated to this, it just controls whether Hive should gather statistics at the end of INSERT statements, disabling it just speeds up the insert query somewhat. If you see something weird or not working with that enabled, feel free to open a support case on our Support Portal. See this and all the hive configuration descriptions in the HiveConf.java source code. Cheers, Miklos

DianaTorres · ‎04-16-2023

Here are some highlights from the month of March DianaTorres_0-1681707483298.png DianaTorres_1-1681707483539.png COMING NEXT! Streaming Data Pipeline Development Check out the FY24 Cloudera Meetup Events Calendar for upcoming & past event details! DianaTorres_2-1681707483368.png 198 new support questions 2 new community articles 592 new members DianaTorres_3-1681707483420.png DianaTorres_4-1681707483479.png DianaTorres_5-1681707483340.png Rank Community Article Author Components/ Labels #1 Processing files within the CML EFS (better I/O performance) - Edge Case with non-tabular data Ryan Cicak @RyanCicak Cloudera Machine Learning (CML) DianaTorres_6-1681707483378.png We would like to recognize the below community members and employees for their efforts over the last month to provide community solutions. See all our top participants at Top Solution Authors leaderboard and all the other leaderboards on our Leaderboards and Badges page. DianaTorres_7-1681707483529.png @MattWho @smdas @steven-matison @mszurap @ckumar @cotopaul @SAMSAL @davehkd @BrianChan @jorgeoa DianaTorres_8-1681707483320.png Share your expertise and answer some of the below open questions. Also, be sure to bookmark the unanswered question page to find additional open questions. Unanswered Community Post Components/ Labels Hortonworks maven repo broken? Hortonworks Data Platform (HDP) Python updates Cloudera Data Platform (CDP) Nifi - Mongo X509 auth Apache NiFi ATLAS-409-00-002: Given type xxx has references Apache Atlas value cannot be converted to a timestamp Apache NiFI NiFi Registry

nabilraza456 · ‎04-16-2023

Hi @DigitalPlumber - Yes it is Public Cloud. I use NiPyAPI on HDF where I was able to add any variable etc using that library. In CDP, I want to get the current processor ID. It can be via expression language or by API. How can I achieve that? For example, I want to fetch all the PutS3 processors ID then how can I get that?

kspyropoulos · ‎04-14-2023

I have also tested the same with Apache Nifi 1.15.3 (open-source/community) and it works. It did not work though when testing with Apache Nifi 1.13.2 (open-source/community) it still fails with the same error message.

mszurap · ‎04-13-2023

This sounds a Spring boot specific issue, a quick googling was leading me to https://stackoverflow.com/questions/42073194/unable-to-detect-database-type please check if the suggestions in that discussion helps.

Asok · ‎04-01-2023

How frequently is this occurring? It is worth checking network drops or you might consider increasing the timeouts on both sides ie. Resource Manager and Zookeeper.

MattWho · ‎03-30-2023

@wffger2 @cotopaul The InstanceIdentifier was introduced when NiFi switched from using the flow.xml.gz to the flow.josn.gz for flow persistence. It is also currently being written to the flows stored in the NiFi-Registry. I say "currently" because this jira exist and is still open: https://issues.apache.org/jira/browse/NIFI-11321 So as @cotopaul points out, the "instanceIdentifier" will be unique for each instantiation of the flow definition, but the "identifier" will be the same. Downside here is every time you want to do a comparison, you'll need to download latest flow definition from DEV and UAT environments and compare for difference based on the Identifiers. Thanks, Matt

RangaReddy · ‎03-30-2023

Hi @Albap Based on the logs, i can see you have created streaming application. By default streaming application will run 24*7, it will stop only when we kill or some interrupted event happen at the system level. Better way to kill/shutdown the spark streaming applications is by using graceful shutdown. If you need further help, please raise an cloudera case we will work on.

Online	Offline
Last Visited	‎08-24-2026 04:08 PM

Member Since	‎11-17-2021 08:08 AM
Last Visited	‎08-24-2026 04:08 PM
Posts	1,168
Kudos received	242

Cloudera Community

Re: How to change the company in the profile

Re: Keycloak SSO Hive REST Catalog

Re: Error connecting to NiFi Registry from NiFi UI...

Re: How to change my Account Email Address?

Re: Cannot erase old /opt/cloudera/parcels

Re: how can i copy hbase data from replication set...

Re: Hive metastore lost connection while executing...

Re: Oozie Hive Job status changes to KILLED but th...

April 2023 Community Highlights

Re: How to get Processor ID in CDF

Re: PutDatabaseRecord "bytes" avro data type fails...

Re: Not able to connect ImpalaDB from Springboot a...

Re: Yarn RM Fatal shutdown cannot find root cause ...

Re: Nifi Flow Definition does not keep the same id...

Re: Multiple spark Spark jobs failed