I have followed the below steps to read the hive table from spark side with credential store: FROM MYSQL  CREATE USER IF NOT EXISTS 'gopi'@'%' IDENTIFIED BY 'gopi'; GRANT ALL PRIVILEGES ON * . * TO 'gopi'@'%'; FLUSH PRIVILEGES; create database if not exists test; use test; CREATE TABLE test.EMPLOYEE( id INT, name varchar(255), salary DECIMAL, dob DATE NOT NULL DEFAULT '2021-05-01', doj TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP(), PRIMARY KEY (id) ); INSERT INTO test.EMPLOYEE (id, name, salary, dob, doj) VALUES (1, "gopi", 10000.00, '1988-06-01', '2020-03-16 09:00:01.000000'); INSERT INTO test.EMPLOYEE (id, name, salary,dob) VALUES (2, "Nishanth", 50000.00, '2018-05-29'); INSERT INTO test.EMPLOYEE (id, name, salary) VALUES (3, "Raja", 30000.00); Create credential store: sudo -u hive hadoop credential create gopi_user.password -v gopi -provider jceks://hdfs/user/hive/gopi_user.jceks hadoop credential list -provider jceks://hdfs/user/hive/gopi_user.jceks sudo -u hive hdfs dfs -chmod 400 /user/hive/gopi_user.jceks FROM HIVE USE db_test; drop table if exists db_test.employee2; CREATE EXTERNAL TABLE db_test.employee2( id INT, name STRING, salary DOUBLE, dob DATE, doj TIMESTAMP ) STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler' TBLPROPERTIES ( "hive.sql.database.type" = "MYSQL", "hive.sql.jdbc.driver" = "com.mysql.cj.jdbc.Driver", "hive.sql.jdbc.url" = "jdbc:mysql://ccycloud-1.nightly7x-us-gr.root.comops.site:3306/test", "hive.sql.dbcp.username" = "gopi", "hive.sql.dbcp.password.keystore" ="jceks://hdfs/user/hive/gopi_user.jceks", "hive.sql.dbcp.password.key" = "gopi_user.password", "hive.sql.query" = "select * from test.EMPLOYEE" ); FROM SPARK sudo -u hive spark-shell \ --jars /opt/cloudera/parcels/CDH/jars/hive-jdbc-handler-3.1.3000.7.2.18.0-622.jar,/usr/share/java/mysql-connector-java.jar scala> spark.sql("SELECT * FROM db_test.employee1").show() +---+--------+-------+----------+-------------------+ | id| name| salary| dob| doj| +---+--------+-------+----------+-------------------+ | 1| gopi|10000.0|1988-06-01|2020-03-16 09:00:01| | 2|Nishanth|50000.0|2018-05-29|2024-02-27 10:39:22| | 3| Raja|30000.0|2021-05-01|2024-02-27 10:39:30| +---+--------+-------+----------+-------------------+ scala> spark.sql("SELECT * FROM db_test.employee2").show() +---+--------+-------+----------+-------------------+ | id| name| salary| dob| doj| +---+--------+-------+----------+-------------------+ | 1| gopi|10000.0|1988-06-01|2020-03-16 09:00:01| | 2|Nishanth|50000.0|2018-05-29|2024-02-27 10:39:22| | 3| Raja|30000.0|2021-05-01|2024-02-27 10:39:30| +---+--------+-------+----------+-------------------+     ... View more

It seems like there might be an issue with the way you're using single quotes in the loop. The variable eachline should be expanded, but it won't if it's enclosed in single quotes. Try using double quotes around the variable and see if that resolves the issue. Here's the corrected loop:       for eachline in "${testarray[@]}" do beeline -f "${eachline}.sql" done   This way, the value of eachline will be correctly expanded when constructing the command. Also, ensure that the SQL files are present in the correct path or provide the full path if needed. If the issue persists, please provide more details on the error message or behavior you're experiencing for further assistance. ... View more

Hive typically relies on the schema definition provided during table creation, and it doesn't perform automatic type conversion while loading data. If there's a mismatch between the data type in the CSV file and the expected data type in the Hive table, it may result in null or incorrect values. Use the CAST function to explicitly convert the data types during the INSERT statement.   INSERT INTO TABLE target_table SELECT CAST(column1 AS INT), CAST(column2 AS STRING), ... FROM source_table;   Preprocess your CSV data before loading it into Hive. You can use tools like Apache NiFi or custom scripts to clean and validate the data before ingestion. Remember to thoroughly validate and clean your data before loading it into Hive to avoid unexpected issues. Also, the choice of method depends on your specific use case and the level of control you want over the data loading process. ... View more

Can you share  2 sample records from the DataFrame for better understanding.  ... View more

Please review the fs.defaultFS configuration in the core-site.xml file within the Hive process directory and ensure that it does not contain any leading or trailing spaces. ... View more

Make sure dfprocessed datafrmae doesn't contains any empty rows.  In Spark, you can identify and filter out empty rows in a DataFrame using the filter operation. Empty rows typically have null or empty values across all columns.   // Identify and filter out empty rows val nonEmptyRowsDF = df.filter(not(df.columns.map(col(_).isNull).reduce(_ || _)))   This code uses the filter operation along with the not function and a condition that checks if any column in a row is null. It then removes rows where all columns are null or empty. If you want to check for emptiness based on specific columns, you can specify those columns in the condition:   val columnsToCheck = Array("column1", "column2", "column3") val nonEmptyRowsDF = df.filter(not(columnsToCheck.map(col(_).isNull).reduce(_ || _)))   Adjust the column names based on your DataFrame structure. The resulting nonEmptyRowsDF will contain rows that do not have null or empty values in the specified columns. ... View more

Observing the provided snippet, it's evident that the job with ID application_1666660764861_0196 is currently in progress. To gather more insights into the ongoing Sqoop job, please review the progress details of this specific application (application_1666660764861_0196) . Check YARN ResourceManager Web UI: Open your web browser and navigate to the YARN ResourceManager Web UI , Look for the specific job ID mentioned in the logs (application_1666660764861_0196). This UI provides details about the running job, its progress, and any errors. Review Hadoop Cluster Logs: Examine the Hadoop cluster logs for any potential issues. Hadoop logs can provide insights into resource constraints, node failures, or other problems that might be affecting your job. Check Database Connection: Ensure that the database you are importing from is accessible and that the connection parameters (such as username, password, JDBC URL) are correct. Sometimes, jobs can hang if there are issues with the database. Verify Network Connectivity: Ensure that there are no network issues between the cluster nodes and the database. Check for any firewalls or network restrictions that might be impacting connectivity. Resource Utilization: Check the resource utilization on your Hadoop cluster. Ensure that there are enough resources (CPU, memory) available for the job to run. By systematically checking these areas, you should be able to gather more information about why the Sqoop job is still running and address any issues that may be preventing it from completing successfully. ... View more

You're right to be concerned about password security, especially in a distributed environment. Spark doesn't inherently provide a built-in secure password handling mechanism like the Hadoop Credential Provider, but there are several approaches you can consider to enhance security when dealing with passwords: Credential Providers: While Spark itself doesn't have a native credential provider, you might consider using Hadoop Credential Providers in combination with Spark. You can store sensitive information like passwords in Hadoop's CredentialProvider API. Then, you'd access these securely stored credentials in your Spark job. Environment Variables: You can set the password as an environment variable on the cluster or machine running Spark. Accessing environment variables in your Spark code helps avoid directly specifying passwords in code or configuration files. Key Management Services (KMS): Some cloud providers offer key management services that allow you to securely store and manage credentials. You can retrieve these credentials dynamically in your Spark application. Secure Storage Systems: Leverage secure storage systems or secret management tools like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault. These tools provide secure storage for sensitive information and offer APIs to retrieve credentials when needed by your application. Secure File Systems: Utilize secure file systems or encryption mechanisms to protect sensitive configuration files. These files could contain the necessary credentials, and access could be restricted using appropriate permissions. Encryption and Secure Communication: Ensure that communication between Spark and external systems is encrypted (e.g., using SSL/TLS) to prevent eavesdropping on the network. Token-Based Authentication: Whenever possible, consider using token-based authentication mechanisms instead of passwords. Tokens can be time-limited and are generally safer for communication over the network. When implementing these measures, it's crucial to balance security with convenience and operational complexity. Choose the approach that aligns best with your security policies, deployment environment, and ease of management for your use case. ... View more

The stacktrace indicates a resemblance to the issue reported in https://issues.apache.org/jira/browse/HIVE-21698. To address this, it is recommended to upgrade to CDP version 7.1.7 or a higher release ... View more