Thank you for replying, that's the exact solution I eventually settled on.  Best, Shelly ... View more

I create a resource as a file, because the python-env resources are specifically for managing Python packages in requirements.txt, according to the documentation.  Thanks!  ... View more

Hello @Jack_sparrow,  Spark should automatically do it, you can control that with these settings:  Input splits are controlled by spark.sql.files.maxPartitionBytes (default 128MB). If smaller, more splits or parallel tasks will be executed. spark.sql.files.openCostInBytes (default 4MB) influences how Spark coalesces small files. Shuffle parallelism spark.sql.shuffle.partitions (default 200). Configiure around  2–3 times per total executor cores. Also, make sure df.write.parquet() doesn’t set everything into few files only. For that, you can use .repartition(n) to increase the parallelism before writing. ... View more

Thank you for the response. ... View more

@yoonli  This thread is growing in to multiple queries that are not directly related.  Please start a new community question so the information is easier for our community members to follow when they have similar issues. Thank you, Matt ... View more

You did it! I changed my tracking strategy to "Tracking timestamps" and it now populated the "View State" window. Thank you very much for your assistance! ... View more

@HoangNguyen  Keep in mind that the Apache NiFi Variable Registry no longer exist in Apache NiFi 2.x releases and there is no more development of the Apache NIFi 1.x versions.  NiFi Parameter Contexts, which were introduced in later versions of Apache NiFi 1.x, provides similar capability going forward and should be used instead of the variable registry.  You'll be forced to transition to Parameter Contexts in order to move to Apache NiFi 2.x. versions.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

Hello @MoJadallah,  Sorry for no one answering so far.  We do want to give you a good experience on your trial.  I assume you requested your free trial from here:  https://www.cloudera.com/products/cloudera-public-cloud-trial.html?internal_keyplay=ALL&internal_campaign=FY25-Q1-GLOBAL-CDP-5-Day-Trial&cid=FY25-Q1-GLOBAL-CDP-5-Day-Trial&internal_link=WWW-Nav-u01  When was that error hit? Just after the sign-up?  ... View more

Default configurations are the tested and for that reason are set in that way, but those are configurable for a reason, sometimes, and depending on the environment, the use case and much more, they need to be tuned in an specific way.  ... View more

Hello @rizalt  Have you tested the same query directly on beeline?  Can you try it?  ... View more