Member since
02-02-2017
101
Posts
29
Kudos Received
15
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
900 | 09-28-2023 11:26 AM | |
943 | 07-05-2023 03:34 PM | |
11796 | 09-22-2018 04:48 AM | |
2830 | 09-19-2018 09:15 PM | |
1595 | 09-17-2018 06:52 PM |
03-20-2024
11:40 PM
1 Kudo
I have followed the below steps to read the hive table from spark side with credential store: FROM MYSQL CREATE USER IF NOT EXISTS 'gopi'@'%' IDENTIFIED BY 'gopi';
GRANT ALL PRIVILEGES ON * . * TO 'gopi'@'%';
FLUSH PRIVILEGES;
create database if not exists test;
use test;
CREATE TABLE test.EMPLOYEE(
id INT,
name varchar(255),
salary DECIMAL,
dob DATE NOT NULL DEFAULT '2021-05-01',
doj TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP(),
PRIMARY KEY (id)
);
INSERT INTO test.EMPLOYEE (id, name, salary, dob, doj) VALUES (1, "gopi", 10000.00, '1988-06-01', '2020-03-16 09:00:01.000000');
INSERT INTO test.EMPLOYEE (id, name, salary,dob) VALUES (2, "Nishanth", 50000.00, '2018-05-29');
INSERT INTO test.EMPLOYEE (id, name, salary) VALUES (3, "Raja", 30000.00); Create credential store: sudo -u hive hadoop credential create gopi_user.password -v gopi -provider jceks://hdfs/user/hive/gopi_user.jceks
hadoop credential list -provider jceks://hdfs/user/hive/gopi_user.jceks
sudo -u hive hdfs dfs -chmod 400 /user/hive/gopi_user.jceks FROM HIVE USE db_test;
drop table if exists db_test.employee2;
CREATE EXTERNAL TABLE db_test.employee2(
id INT,
name STRING,
salary DOUBLE,
dob DATE,
doj TIMESTAMP
)
STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
"hive.sql.database.type" = "MYSQL",
"hive.sql.jdbc.driver" = "com.mysql.cj.jdbc.Driver",
"hive.sql.jdbc.url" = "jdbc:mysql://ccycloud-1.nightly7x-us-gr.root.comops.site:3306/test",
"hive.sql.dbcp.username" = "gopi",
"hive.sql.dbcp.password.keystore" ="jceks://hdfs/user/hive/gopi_user.jceks",
"hive.sql.dbcp.password.key" = "gopi_user.password",
"hive.sql.query" = "select * from test.EMPLOYEE"
); FROM SPARK sudo -u hive spark-shell \
--jars /opt/cloudera/parcels/CDH/jars/hive-jdbc-handler-3.1.3000.7.2.18.0-622.jar,/usr/share/java/mysql-connector-java.jar
scala> spark.sql("SELECT * FROM db_test.employee1").show()
+---+--------+-------+----------+-------------------+
| id| name| salary| dob| doj|
+---+--------+-------+----------+-------------------+
| 1| gopi|10000.0|1988-06-01|2020-03-16 09:00:01|
| 2|Nishanth|50000.0|2018-05-29|2024-02-27 10:39:22|
| 3| Raja|30000.0|2021-05-01|2024-02-27 10:39:30|
+---+--------+-------+----------+-------------------+
scala> spark.sql("SELECT * FROM db_test.employee2").show()
+---+--------+-------+----------+-------------------+
| id| name| salary| dob| doj|
+---+--------+-------+----------+-------------------+
| 1| gopi|10000.0|1988-06-01|2020-03-16 09:00:01|
| 2|Nishanth|50000.0|2018-05-29|2024-02-27 10:39:22|
| 3| Raja|30000.0|2021-05-01|2024-02-27 10:39:30|
+---+--------+-------+----------+-------------------+
... View more
01-12-2024
06:27 AM
@cravani I have none of /var/log/hive, /tmp/hive or /tmp/hive/hive do you think some log files has more info than console
... View more
12-11-2023
02:00 PM
@RS_11 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.
... View more
10-06-2023
04:40 PM
@andrea_pretotto Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.
... View more
10-03-2023
08:39 AM
@cravani Thanks for the help 🙂
... View more
10-02-2023
11:13 PM
Hi @pranav007 The required setup is little bit complex. You can try copy the core-site.xml, hdfs-site.xml, yarn-site.xml, hive-site.xml, mapred-site.xml, krb_configuration files to resource folder. In the spark code, you need to add a two parameters i.e spark.driver.extraJavaOptions and spark.executor.extraJavaOptionsby specifiing the krb_file location. --conf spark.driver.extraJavaOptions="-Djava.security.krb5.conf=KRB5_PATH" \
--conf spark.executor.extraJavaOptions="-Djava.security.krb5.conf=KRB5_PATH" \
... View more
05-31-2020
09:18 AM
1 Kudo
Hi @Shelton , Was you get chance to look into this issue, needed help on this. Thanks
... View more
03-16-2020
05:22 AM
I'm also getting the same issue but for me instead of taking my username.its showing user Anonymous.Can anyone suggest how should i resolve this issue?
... View more