Why not using the resource pool and sub pool, if it specific query then pass the resource pool for this query and create resource pool or subpool for this query ... View more

very detailed and useful article. Thank you @asish!  ... View more

Hi @Noel_0317, The error indicates that there are multiple partitions in the where condition. Can you try the below query: INSERT OVERWRITE TABLE db.table_name PARTITION(dt='2023-03-26') select distinct * from db.table_name where dt = '2023-03-26';   Let us know how it goes.   Cheers! ... View more

@Choolake, Thank you for your participation in Cloudera Community. I'm happy to see you resolved your issue. Please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.   ... View more

@Abdul_  As of now hive won't support row delimiter other new line character . Attaching the corresponding Jira for reference HIVE-11996  As a workaround, Recommend to update the input file using external libraries like awk,...etc and upload the input file in the corresponding FileSystem location to read.   Eg -  Through AWK  [root@c2757-node2 ~]# awk -F "\",\"" 'NF < 3 {getline nextline; $0 = $0 nextline} 1' sample_case.txt "IM43163","SOUTH,OFC","10-Jan-23" "IM41763","John:comment added","12-Jan-23" [root@c2757-node2 ~]# awk -F "\",\"" 'NF < 3 {getline nextline; $0 = $0 nextline} 1' sample_case.txt > sample_text.csv Reading from Hive Table 0: jdbc:hive2://c2757-node2.coelab.cloudera.c> select * from table1; . . . INFO : Executing command(queryId=hive_20230616064136_333ff98d-636b-43b1-898d-fca66031fe7f): select * from table1 INFO : Completed executing command(queryId=hive_20230616064136_333ff98d-636b-43b1-898d-fca66031fe7f); Time taken: 0.023 seconds INFO : OK +---------------+---------------------+---------------+ | table1.col_1 | table1.col_2 | table1.col_3 | +---------------+---------------------+---------------+ | IM43163 | SOUTH,OFC | 10-Jan-23 | | IM41763 | John:comment added | 12-Jan-23 | +---------------+---------------------+---------------+ 2 rows selected (1.864 seconds) ... View more

Once the data has been read from database, you don't need to write the same data to  file (i.e. CSV ) .  Instead you can write directly into hive table using DataFrame API's.  Once the Data has been loaded you query the same from hive.    df.write.mode(SaveMode.Overwrite).saveAsTable("hive_records")     Ref - https://spark.apache.org/docs/2.4.7/sql-data-sources-hive-tables.html Sample Code Snippet    df = spark.read \ .format("jdbc") \ .option("url", "jdbc:postgresql://<server name>:5432/<DBNAME>") \ .option("dbtable", "\"<SourceTableName>\"") \ .option("user", "<Username>") \ .option("password", "<Password>") \ .option("driver", "org.postgresql.Driver") \ .load() df.write.mode('overwrite').saveAsTable("<TargetTableName>") From hive INFO : Compiling command(queryId=hive_20230607042851_fa703b79-d6e0-4a4c-936c-efa21ec00a10): select count(*) from TBLS_POSTGRES INFO : Semantic Analysis Completed (retrial = false) INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20230607042851_fa703b79-d6e0-4a4c-936c-efa21ec00a10); Time taken: 0.591 seconds INFO : Executing command(queryId=hive_20230607042851_fa703b79-d6e0-4a4c-936c-efa21ec00a10): select count(*) from TBLS_POSTGRES . . . +------+ | _c0 | +------+ | 122 | +------+      ... View more