@srinivasp    I am wondering whether your Ranger policies are also in place.  Please explicitly give the correct permissions to the group/user in Ranger as the beeline authorization depends now on Ranger 🙂     Happy hadooping.  ... View more

@enirys  That's correct to successfully set up an HMS  HA you  MUST ensure the metadata DB should have followed the steps mention in this official document: Configuring High Availability for the Hive Metastore High Availability for Hive Metastore That's should help you sort of the stale metadata issue ... View more

@SparkNewbie  Bingo you are using the derby DB, which is only recommended for testing.   There are three modes for Hive Metastore deployment: Embedded Metastore Local Metastore Remote Metastore In Hive by default, metastore service runs in the same JVM as the Hive service. It uses embedded derby database stored on the local file system in this mode. Thus both metastore service and hive service runs in the same JVM by using embedded Derby Database. But, this mode also has its limitation that, as only one embedded Derby database can access the database files on disk at any one time, so only one Hive session could be open at a time. 21/07/07 23:07:56 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY Derby is an embedded relational database in Java program and used for online transaction processing and has a 3.5 MB disk-space footprint. Depending on your software HDP or Cloudera ensure the hive DB is plugged to an external Mysql database For CDH using Mysql  For HDP using mysql  Check your current hive UI backend metadata databases !! After installing MySQL then you should toggle hive config to point to the external Mysql database .. Once done your commands and the refresh should succeed Please let me know if you need help Happy hadooping   ... View more

@SparkNewbie  Can you add the redcline between your 2 commands spark.sql("create table test1_0522 location 's3a://<test-bucket>/data/test1_0522' stored as PARQUET as select * from my_temp_table") REFRESH TABLE test1_0522; spark.sql("SHOW TABLES").show()   That should resolve the problem. Happy hadooping ... View more

@t1  I tried out the sqoop list-databases  and my output looks correct  [root@bern ~]# sqoop list-databases \ > --connect jdbc:mysql://localhost:3306/ \ > --username root \ > --password welcome ......... SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 21/07/05 17:28:04 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7.3.1.4.0-315 21/07/05 17:28:05 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 21/07/05 17:28:05 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. information_schema ambari druid harsh8 hive mysql oozie performance_schema ranger rangerkms superset Then I run exactly the same sqoop import  and it succeeded but I think its the underlying table format [hdfs@bern ~]$ sqoop import --connect jdbc:mysql://localhost:3306/harsh8 --username root --table staff2 --hive-import --fields-terminated-by "," --hive-import --create-hive-table --hive-table staff2_backup --m 1 SLF4J: Class path contains multiple SLF4J bindings. ..... SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 21/07/05 18:36:40 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7.3.1.4.0-315 21/07/05 18:36:41 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 21/07/05 18:36:41 INFO tool.CodeGenTool: Beginning code generation 21/07/05 18:36:43 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `staff2` AS t LIMIT 1 21/07/05 18:36:43 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `staff2` AS t LIMIT 1 21/07/05 18:36:43 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/3.1.4.0-315/hadoop-mapreduce 21/07/05 18:36:48 WARN orm.CompilationManager: Could not rename /tmp/sqoop-hdfs/compile/358a7be0c1aae1ac531284e68ae3679e/staff2.java to /home/hdfs/./staff2.java. Error: Destination '/home/hdfs/./staff2.java' already exists 21/07/05 18:36:48 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hdfs/compile/358a7be0c1aae1ac531284e68ae3679e/staff2.jar 21/07/05 18:36:49 WARN manager.MySQLManager: It looks like you are importing from mysql. 21/07/05 18:36:49 WARN manager.MySQLManager: This transfer can be faster! Use the --direct 21/07/05 18:36:49 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path. 21/07/05 18:36:49 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql) 21/07/05 18:36:49 INFO mapreduce.ImportJobBase: Beginning import of staff2 21/07/05 18:36:59 INFO client.RMProxy: Connecting to ResourceManager at bern.swiss.ch/192.168.0.139:8050 21/07/05 18:37:09 INFO client.AHSProxy: Connecting to Application History server at bern.swiss.ch/192.168.0.139:10200 21/07/05 18:38:28 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/hdfs/.staging/job_1625500722080_0001 21/07/05 18:40:05 INFO db.DBInputFormat: Using read commited transaction isolation 21/07/05 18:40:23 INFO mapreduce.JobSubmitter: number of splits:1 21/07/05 18:40:32 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1625500722080_0001 21/07/05 18:40:32 INFO mapreduce.JobSubmitter: Executing with tokens: [] 21/07/05 18:40:34 INFO conf.Configuration: found resource resource-types.xml at file:/etc/hadoop/3.1.4.0-315/0/resource-types.xml 21/07/05 18:40:37 INFO impl.YarnClientImpl: Submitted application application_1625500722080_0001 21/07/05 18:40:37 INFO mapreduce.Job: The url to track the job: http://bern.swiss.ch:8088/proxy/application_1625500722080_0001/ 21/07/05 18:40:37 INFO mapreduce.Job: Running job: job_1625500722080_0001 21/07/05 18:46:55 INFO mapreduce.Job: Job job_1625500722080_0001 running in uber mode : false 21/07/05 18:46:55 INFO mapreduce.Job: map 0% reduce 0% 21/07/05 18:50:56 INFO mapreduce.Job: map 100% reduce 0% 21/07/05 18:51:09 INFO mapreduce.Job: Job job_1625500722080_0001 completed successfully 21/07/05 18:51:10 INFO mapreduce.Job: Counters: 32 File System Counters FILE: Number of bytes read=0 ............. HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Other local map tasks=1 Total time spent by all maps in occupied slots (ms)=385416 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=192708 Total vcore-milliseconds taken by all map tasks=192708 Total megabyte-milliseconds taken by all map tasks=394665984 Map-Reduce Framework Map input records=6 Map output records=6 ............ Physical memory (bytes) snapshot=152813568 Virtual memory (bytes) snapshot=3237081088 Total committed heap usage (bytes)=81788928 Peak Map Physical memory (bytes)=152813568 Peak Map Virtual memory (bytes)=3237081088 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=223 21/07/05 18:51:10 INFO mapreduce.ImportJobBase: Transferred 223 bytes in 852.4312 seconds (0.2616 bytes/sec) 21/07/05 18:51:10 INFO mapreduce.ImportJobBase: Retrieved 6 records. 21/07/05 18:51:10 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `staff2` AS t LIMIT 1 21/07/05 18:51:10 INFO hive.HiveImport: Loading uploaded data into Hive My table structure MariaDB [harsh8]> describe staff2; +------------+-------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +------------+-------------+------+-----+---------+-------+ | id | int(11) | NO | PRI | NULL | | | Name | varchar(20) | YES | | NULL | | | Position | varchar(20) | YES | | NULL | | | Salary | int(11) | YES | | NULL | | | Department | varchar(10) | YES | | NULL | | +------------+-------------+------+-----+---------+-------+   My test table contents  MariaDB [harsh8]> select * from staff2; +-----+------------+-------------------+--------+------------+ | id | Name | Position | Salary | Department | +-----+------------+-------------------+--------+------------+ | 100 | Geoffrey | manager | 50000 | Admin | | 101 | Thomas | Oracle Consultant | 15000 | IT | | 102 | Biden | Project Manager | 28000 | PM | | 103 | Carmicheal | Bigdata developer | 30000 | BDS | | 104 | Johnson | Treasurer | 21000 | Accounts | | 105 | Gerald | Director | 30000 | Management | +-----+------------+-------------------+--------+------------+ 6 rows in set (0.09 sec) This is how my comma-delimited [hdfs@bern ~]$ hdfs dfs -cat /tmp/sqoop/hr.txt 100,Geoffrey,manager,50000,Admin 101,Thomas,Oracle Consultant,15000,IT 102,Biden,Project Manager,28000,PM 103,Carmicheal,Bigdata developer,30000,BDS 104,Johnson,Treasurer,21000,Accounts 105,Gerald,Director,30000,Management 106,Paul,Director,30000,Management 105,Mark,CEO,90000,Management 105,Edward,Janitor,30000,Housing 105,Richard,Farm Manager,31000,Agriculture 105,Albert,System Engineer,21000,IT The dataset looks like the above is your format AVRO? Happy hadooping   ... View more

@t1  Is there a way I can re-create your tables?  I could try the same commands I also have MySQL/MariaDB. And keep you posted! ... View more

@t1  How is the root user authentication against the databases?  If username password then I don't see the prompt for the password. Can you run the below and re-share the out put   Added sqoop import \ --connect jdbc:mysql://localhost:3306/sample \ --username root -p \ --table test \ --hive-import \ --fields-terminated-by "," --hive-import --create-hive-table --hive-table sample.tesr --m 4 ... View more

@Guarupe  Can you share your steps, please? Are you suing HUE to run your cmds? Did you use the Impala editor? ... View more

@tomm  A quick thought can you check and share your /etc/hosts ... View more

@ask_bill_brooks  Thanks for the addendum and official context. Happy hadooping ... View more