It is not possible to add an aux jar directly from CM.  Follow the below documents depending on the requirement.  https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/using-hiveql/topics/hive_create_place_udf_jar.html https://docs.cloudera.com/cdw-runtime/1.5.0/integrating-hive-and-bi/topics/hive_setup_jdbcstoragehandler_edb.html ... View more

Hi, From shell find the files that needs to be deleted and save them in a temp file like below, #!/bin/sh today=`date +'%s'` hdfs dfs -ls /file/Path/ | while read line ; do dir_date=$(echo ${line} | awk '{print $6}') difference=$(( ( ${today} - $(date -d ${dir_date} +%s) ) / ( 24*60*60 ) )) filePath=$(echo ${line} | awk '{print $8}') if [ ${difference} -gt 3 ]; then echo -e "$filePath" >> toDelete fi done Then execute arbitrary shell command using form example subprocess.call or sh library so something like below import subprocess file = open('toDelete', 'r') for each in file: subprocess.call(["hadoop", "fs", "-rm", "-f", each]) Also, you can use hdfs fs API in PySpark like below,  from pyspark.sql import SparkSession spark = SparkSession.builder.appName('abc').getOrCreate() def delete_path(spark, path): sc = spark.sparkContext fs = (sc._jvm.org .apache.hadoop .fs.FileSystem .get(sc._jsc.hadoopConfiguration()) ) fs.delete(sc._jvm.org.apache.hadoop.fs.Path(path), True) delete_path(spark, "Your/hdfs/path") ... View more

Hi, Hiveserver2 runs as a java process, When running timestamp-related UDF functions in Hive, the default behavior is to use the system's timezone information to convert timestamp values. PFB   INFO : Executing command(queryId=hive_20220215131716_33c6c90b-207a-43fe-9bcf-4a52cd04de3e): SELECT current_timestamp() INFO : Completed executing command(queryId=hive_20220215131716_33c6c90b-207a-43fe-9bcf-4a52cd04de3e); Time taken: 0.092 seconds INFO : OK +--------------------------+ | _c0 | +--------------------------+ | 2022-02-15 13:17:16.204 | +--------------------------+ 1 row selected (2.413 seconds) [hive@c2757-node3 ~]$ date Tue Feb 15 13:17:26 IST 2022 [hive@c2757-node3 ~]$ timedatectl | grep "Time zone" Time zone: Asia/Kolkata (IST, +0530)   To make Hive return a specific timezone with the timestamp function. please follow the steps below: 1. Go to the Cloudera Manager home page > Hive > Configuration Under "Client Java Configuration Options", append " -Duser.timezone=UTC" in the text string (be mindful of the leading space in front, if you append to the end of existing options). 2 . Under "Java Configuration Options for HiveServer2", append the same thing to the end of the text string " -Duser.timezone=UTC". 3 . Save the configuration, then restart any HiveServer2 instances, and select Actions -> "Deploy Client Configuration" through Cloudera Manager. To confirm the new configuration is working, see below test outputs: Before the change (system's default timezone is IST):   INFO : Executing command(queryId=hive_20220215131716_33c6c90b-207a-43fe-9bcf-4a52cd04de3e): SELECT current_timestamp() INFO : Completed executing command(queryId=hive_20220215131716_33c6c90b-207a-43fe-9bcf-4a52cd04de3e); Time taken: 0.092 seconds INFO : OK +--------------------------+ | _c0 | +--------------------------+ | 2022-02-15 13:17:16.204 | +--------------------------+ 1 row selected (2.413 seconds) [hive@c2757-node3 ~]$ date Tue Feb 15 13:17:26 IST 2022 [hive@c2757-node3 ~]$ timedatectl | grep "Time zone" Time zone: Asia/Kolkata (IST, +0530) After the change to UTC, [hive@c2757-node3 ~]$ timedatectl | grep "Time zone" Time zone: Asia/Kolkata (IST, +0530) [hive@c2757-node3 ~]$ date Tue Feb 15 13:28:05 IST 2022 INFO : Executing command(queryId=hive_20220215075824_6a58bdcb-b1b6-470d-9202-26ccfc60f521): SELECT current_timestamp() INFO : Completed executing command(queryId=hive_20220215075824_6a58bdcb-b1b6-470d-9202-26ccfc60f521); Time taken: 0.079 seconds INFO : OK +--------------------------+ | _c0 | +--------------------------+ | 2022-02-15 07:58:24.785 | +--------------------------+ 1 row selected (2.24 seconds) Like this, you can mention hive to use other timezones. ... View more