Yes, Cloudera's management tools, especially Cloudera Manager, do provide insights and metrics about jobs running on Hadoop, including both MapReduce and Spark jobs. For job completion time estimation: Cloudera Manager: Within the Cloudera Manager interface, you can navigate to the specific service (like YARN or Spark) to view details about running or completed jobs. For each job, there's an estimated time of completion based on the progress and resources available. However, it's worth noting that these estimations can vary based on data skew, resource contention, and other factors. Resource Manager UI: For YARN based jobs, the YARN Resource Manager UI provides information about running applications, including their progress. The percentage completion might give a rough idea, but it doesn't directly estimate the completion time. Spark UI: For Spark jobs, the Spark UI provides insights into job stages, tasks, and their durations. While it doesn’t give a direct "time remaining" estimate, you can use the information about completed stages/tasks to infer how long the remaining stages/tasks might take. That being said, while these tools can provide some insights, predicting the exact completion time for distributed computing jobs can be challenging due to the dynamic nature of distributed resources, data imbalances, etc. To have more accurate estimations, it's recommended to: Monitor Resource Usage: Ensuring you have enough resources (memory, CPU, etc.) for your jobs. Optimize Your Jobs: Depending on the nature of your job, consider optimizing your code or the configuration. Historical Data: Look at the historical runtimes of similar jobs to provide a ballpark figure for future runs. I hope this provides clarity on your query. If you have any more questions or need further insights, please let us know. Resource- Cloudera
... View more
