Member since
03-31-2016
33
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2791 | 07-19-2016 11:58 AM |
11-03-2016
05:24 AM
Hello, Earlier we used to point the Hive's external table's location to S3. We now have a requirement to point it to a local filesystem like /tmp etc but not HDFS. Can this be achieved in Hive?
... View more
Labels:
- Labels:
-
Apache Hive
07-19-2016
11:58 AM
Hello, From the logs, I found that it was stuck updating the hdfs location. So, I updated the hdfs location manually by running the command metatool -updateLocation After that I was able to bring up the Hiveserver2 service. Best Regards, Rinku Singh.
... View more
07-19-2016
09:51 AM
Hello @Mukesh Kumar I don't think increasing the timeout number will help here. There must be something that is stopping it from start. From the ambari-ui, I could see the below error: Connection failed on host ip-172-31-31-251.us-west-2.compute.internal:10000 (Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_thrift_port.py", line 200, in execute
check_command_timeout=int(check_command_timeout))
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/hive_check.py", line 74, in check_thrift_port_sasl
timeout=check_command_timeout)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
Fail: Execution of '! beeline -u 'jdbc:hive2://ip-172-31-31-251.us-west-2.compute.internal:10000/;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'' returned 1. Error: Could not open client transport with JDBC Uri: jdbc:hive2://ip-172-31-31-251.us-west-2.compute.internal:10000/;transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0)
Error: Could not open client transport with JDBC Uri: jdbc:hive2://ip-172-31-31-251.us-west-2.compute.internal:10000/;transportMode=binary: java.net.ConnectException: Connection refused (state=08S01,code=0)
)
... View more
07-19-2016
08:44 AM
Hello, I am trying to start Hiveserver2 but it is timing out with the error: "Python script has been killed due to timeout after waiting 900 secs" Tried to find something in the log file but there was no error. Attaching the log file for reference. Best Regards, Rinku Singh.
... View more
Labels:
- Labels:
-
Apache Hive
07-15-2016
05:10 AM
Hi @Sindhu I finally had to add the parameter javax.jdo.option.ConnectionPassword in the hive-site.xml and re-run it. Fixed all the issues and everything is working now. Thank you so much for your help and pointers. Best Regards, Rinku Singh.
... View more
07-14-2016
10:17 AM
Hello @Sindhu I did the following. a. Created a new MySQL instance and migrated the tables into this new database b. Pointed the hive metastore to the new MySQL instance. As expected, the hive metastore service didn't come up. c. I ran the upgrade script command to upgrade the MySQL database schema. d. I restarted the hive metastore service and it came up. e. Now as the schema versions are same, I migrated the tables from the new MySQL database to Postgres database. f. I am now trying to start the hiveserver2 but it is not starting but I am able to login into hive command prompt. g. I also ran the following command [rsingh01@ip-172-31-31-251 bin]$ ./schematool -info -dbType postgres -userName root -passWord password -verbose
WARNING: Use "yarn jar" to launch YARN applications.
Metastore connection URL: jdbc:postgresql://xxxxxxxxxxxxxxxxxxxxxxxxx/cirrus3
Metastore Connection Driver : org.postgresql.Driver
Metastore connection User: root
Hive distribution version: 1.2.1000
Metastore schema version: 1.2.1000
schemaTool completed h. I tried querying the tables but now it is showing the following error: hive> select * from rms_customer;
FAILED: SemanticException Unable to determine if hdfs://xxxxxxxxxxxxxxxxxxxxxxxxxxxx:8020/user/hive/warehouse/rms.db/rms_customer is encrypted: java.lang.IllegalArgumentException: Wrong FS: hdfs://xxxxxxxxxxxxxxxxxxxxxxxxxxxx:8020/user/hive/warehouse/rms.db/rms_customer, expected: hdfs://xxxxxxxxxxxxxxxxxxxxxxxxxxxx:8020 h. I followed the link https://issues.apache.org/jira/browse/HIVE-11116 and tried to list the FSRoot by running the command metatool -listFSRoot but this time it is throwing the error: =========================== [rsingh01@ip-172-31-31-251 bin]$ ./metatool -listFSRoot
WARNING: Use "yarn jar" to launch YARN applications.
Initializing HiveMetaTool..
16/07/14 10:05:43 INFO metastore.ObjectStore: ObjectStore, initialize called
16/07/14 10:05:43 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
16/07/14 10:05:43 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
16/07/14 10:05:44 ERROR Datastore.Schema: Failed initialising database.
Unable to open a test connection to the given database. JDBC url = jdbc:postgresql://xxxxxxxxxxxxxxxxxxxxxxxx/cirrus1, username = root. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
org.postgresql.util.PSQLException: FATAL: password authentication failed for user "root" =========================== I am totally stuck now Best Regards, Rinku Singh.
... View more
07-14-2016
07:29 AM
Hi @Sindhu I found the reason why it says --- relation "compaction_queue" does not exist In postgres, the command will not work: ALTER TABLE COMPACTION_QUEUE ADD COLUMN CQ_HIGHEST_TXN_ID bigint; But the below command will work: ALTER TABLE "COMPACTION_QUEUE" ADD COLUMN "CQ_HIGHEST_TXN_ID" bigint; The table name needs to be in double quotes. As all the upgrade scripts are written normally without double quotes, it will not work. Trying to figure out a way how we can upgrade the database schema version with all these restrictions in postgres.
Best Regards, Rinku Singh.
... View more
07-14-2016
05:28 AM
Hi @Sindhu I ran the above command and found that the database schema version is not compatible with hive version ========================================== [rsingh01]$ ./schematool -info -dbType postgres -userName root -passWord password -verbose
WARNING: Use "yarn jar" to launch YARN applications.
Metastore connection URL: jdbc:postgresql://xxxxxxxxxxxxxxxxxxxxxx/xxxxx Metastore Connection Driver : org.postgresql.Driver
Metastore connection User: root
Hive distribution version: 1.2.1000
Metastore schema version: 1.2.0
org.apache.hadoop.hive.metastore.HiveMetaException: Metastore schema version is not compatible. Hive Version: 1.2.1000, Database Schema Version: 1.2.0
org.apache.hadoop.hive.metastore.HiveMetaException: Metastore schema version is not compatible. Hive Version: 1.2.1000, Database Schema Version: 1.2.0
at org.apache.hive.beeline.HiveSchemaTool.assertCompatibleVersion(HiveSchemaTool.java:196)
at org.apache.hive.beeline.HiveSchemaTool.showInfo(HiveSchemaTool.java:140)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:501)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
*** schemaTool failed *** ============================================ I then tried upgrading the database schema version by running the below command. The dryRun is going through successfully but now I am stuck when I am doing the actual execution of the script ============================================= [rsingh01]$ ./schematool -dbType postgres -userName root -passWord password -upgradeSchema -dryRun
WARNING: Use "yarn jar" to launch YARN applications.
Metastore connection URL: jdbc:postgresql://xxxxxxxxxxxxxxxxxxxxxx/xxxxx Metastore Connection Driver : org.postgresql.Driver
Metastore connection User: root
Starting upgrade metastore schema from version 1.2.0 to 1.2.1000
Upgrade script upgrade-1.2.0-to-1.2.1000.postgres.sql
schemaTool completed [rsingh01]$ ./schematool -dbType postgres -userName root -passWord password -upgradeSchema
WARNING: Use "yarn jar" to launch YARN applications.
Metastore connection URL: jdbc:postgresql://xxxxxxxxxxxxxxxxxxxxxx/xxxxx Metastore Connection Driver : org.postgresql.Driver
Metastore connection User: root
Starting upgrade metastore schema from version 1.2.0 to 1.2.1000
Upgrade script upgrade-1.2.0-to-1.2.1000.postgres.sql
Error: ERROR: relation "compaction_queue" does not exist (state=42P01,code=0)
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent !!
*** schemaTool failed ***
... View more
07-13-2016
01:00 PM
Hi @Sindhu Let me put this in much clear way. There are two databases. 1. I created a database called Test. I then started the Hive service and it came up properly. I then saw that 56 tables were created which were not present earlier. It's obvious that it would have run the schemaTool -initSchema command. 2. Later I created one more database and I copied the 56 tables from MySQL to Postgres(in the new database). Changed the database in Ambari to point to this new database and tried restarting the Hive service. Then I got this error. True that the Hive metastore is trying to run schemaTool -initSchema command in second scenario as well and I want to understand what triggers it to run the schemaTool -initSchema command and how can I resolve it so that I can successfully migrate the metastore and start the Hive service. Best Regards, Rinku Singh.
... View more
07-13-2016
12:18 PM
Hi @Sindhu Just to give more background. I created a new database in postgres which in turn creates a new schema called 'public' with no tables in it. As soon as the connection is made by hive for the first time, it internally runs scripts and creates around 56 tables in the public schema. This applies to Mysql db as well. The issue comes when I have already migrated those 56 tables present in MySQL database to postgres database and then try to connect, I get the above error. Best Regards, Rinku Singh.
... View more