About fahed

fahed · ‎03-10-2023

For the on-prem cloudera setups where data are stored on local disks, we add a tag mentioning that this disk will for archive and assign HDFS policies to do so. But, on Cloud since HDFS is advised to use as a temporary place and data are mostly stored on cloud object storage. Also, for both Hive managed and external tables the location is cloud object storage. So, is the way to do the archive is just by applying the archive policies on the object storage? Cloudera's docs only mention the on-prem way which is additional disk. Archive link: https://docs.cloudera.com/runtime/7.2.10/scaling-namespaces/topics/hdfs-configure-archival-storage.html

steven-matison · ‎03-02-2023

@fahed What you see with the CDP Public Cloud Data Hubs using GCS (or object store) is a modernization of the platform around object storage. This removes differences across aws, azure, and on-prem (when Ozone is used). It is a change by customer demand so that workloads are able to be built and deployed with minimal changes from on prem to cloud or cloud to cloud. Unfortunately that creates a difference you describe above, but those are risks we are willing to take ourselves in favor of modern data architecture. If you are looking for performance, you should take a look at some of the newer options for databases: impala and kudu (this one uses local disk). Also we have Iceberg coming into this space too.

steven-matison · ‎02-23-2023

@fahed That size is to be able to grow and serve in production manner. At first that disk usage could be low. For DataHubs, My recommendation is to start small and grow as needed. Most of your work load data should be in object store(s) for the data hubs, so dont think of that "hdfs" disk as being size constrained to initial creations of the hub.

Online	Offline
Last Visited	‎03-10-2023 04:30 AM

Member Since	‎02-21-2023 02:16 AM
Last Visited	‎03-10-2023 04:30 AM
Posts	4

Cloudera Community

How to archive data on CDP Public Cloud?

Re: Changing data location from GCS to local disks...

Re: CDP Public Cloud Datalake HDFS usage