Share sample Data file with minimum of 2 records , to understand the structure.  ... View more

Output of below to identify the exact ouptut records details,  explain formatted <query> explain extended <query> explain analyze <query> ... View more

Once the data has been read from database, you don't need to write the same data to  file (i.e. CSV ) .  Instead you can write directly into hive table using DataFrame API's.  Once the Data has been loaded you query the same from hive.    df.write.mode(SaveMode.Overwrite).saveAsTable("hive_records")     Ref - https://spark.apache.org/docs/2.4.7/sql-data-sources-hive-tables.html Sample Code Snippet    df = spark.read \ .format("jdbc") \ .option("url", "jdbc:postgresql://<server name>:5432/<DBNAME>") \ .option("dbtable", "\"<SourceTableName>\"") \ .option("user", "<Username>") \ .option("password", "<Password>") \ .option("driver", "org.postgresql.Driver") \ .load() df.write.mode('overwrite').saveAsTable("<TargetTableName>") From hive INFO : Compiling command(queryId=hive_20230607042851_fa703b79-d6e0-4a4c-936c-efa21ec00a10): select count(*) from TBLS_POSTGRES INFO : Semantic Analysis Completed (retrial = false) INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20230607042851_fa703b79-d6e0-4a4c-936c-efa21ec00a10); Time taken: 0.591 seconds INFO : Executing command(queryId=hive_20230607042851_fa703b79-d6e0-4a4c-936c-efa21ec00a10): select count(*) from TBLS_POSTGRES . . . +------+ | _c0 | +------+ | 122 | +------+      ... View more

mapreduce.output.basename also works since as part of setoutput name assigning the same.  Code snippet form ParquetOutputFormat. protected static void setOutputName(JobContext job, String name) { job.getConfiguration().set("mapreduce.output.basename", name); } JOB CONF -  Configuration conf = getConf(); conf.set("mapreduce.output.basename","parquet_output"); Output  [hive@c1757-node3 ~]$ hdfs dfs -ls /tmp/parquet-sample Found 4 items -rw-r--r-- 2 hive supergroup 0 2023-06-06 17:08 /tmp/parquet-sample/_SUCCESS -rw-r--r-- 2 hive supergroup 271 2023-06-06 17:08 /tmp/parquet-sample/_common_metadata -rw-r--r-- 2 hive supergroup 1791 2023-06-06 17:08 /tmp/parquet-sample/_metadata -rw-r--r-- 2 hive supergroup 2508 2023-06-06 17:08 /tmp/parquet-sample/parquet_output-m-00000.parquet ... View more

Since the output file is .parquet , hope you're using ParquetOutputFormat in the MR job config. In that case ParquetOutputFormat.setOutputname  method will help to set the base name of the output file.   Ref - https://www.javadoc.io/doc/org.apache.parquet/parquet-hadoop/1.12.2/org/apache/parquet/hadoop/ParquetOutputFormat.html https://hadoop.apache.org/docs/r2.8.0/api/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.html#setOutputName(org.apache.hadoop.mapreduce.JobContext,%20java.lang.String) ... View more

It's working expected. Please find the below code snippet >>> columns = ["language","users_count"] >>> data = [("Java", "20000"), ("Python", "100000"), ("Scala", "3000")] >>> df = spark.createDataFrame(data).toDF(*columns) >>> df.write.csv("/tmp/test") >>> df2=spark.read.csv("/tmp/test/*.csv") d>>> df2.show() +------+------+ | _c0| _c1| +------+------+ |Python|100000| | Scala| 3000| | Java| 20000| +------+------+   ... View more

From the error could see the query failed in MoveTask. MoveTask can be loading the partitions as well since the load statement belongs to the partitioned table, Along with HS2 logs HMS logs for the corresponding time period gives a better idea to identify the root cause of the failure.  If it's just timeout issue,  increase client socket timeout value.  ... View more

@Sunil1359   Compilation might be higher if the table has a large number of partitions or if the HMS process is slow when the query runs.  Please check the below on the corresponding time period to find the root cause.  HS2 log HMS log HMS jstack  In Tez engine queries will run in the form of DAG.  In the compilation phase, once the semantic analysis process is completed, the plan will be generated depending on the query you submitted. explain <your query> gives the plan of the query.  Once the plan is generated DAG will be submitted to yarn and the DAG will run depending on the plan.  As part of DAG, Split generation, input file read, shuffle fetch ..etc will be taken care and the end result will be transferred to the client.  ... View more