https://hortonworks.com/hadoop-tutorial/manage-security-policy-hive-hbase-knox-ranger/ ... View more

I am setting the Hive metastore db driver using: ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/hdp/current/hive-server2/lib/mysql-connector-java-5.1.38.jar Using python /usr/bin/python Setup ambari-server .Copying /usr/hdp/current/hive-server2/lib/mysql-connector-java-5.1.38.jar to /var/lib/ambari-server/resources If you are updating existing jdbc driver jar for mysql with mysql-connector-java-5.1.38.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts. JDBC driver was successfully initialized. Ambari Server 'setup' completed successfully. and I have checked that the jar is in that path. But, when I try to restart Hiveserver2, I get the following error message: 2017-01-23 20:59:37,414 - Error! Sorry, but we can't find jdbc driver with default name mysql-connector-java.jar in hive lib dir. So, db connection check can fail. Please run 'ambari-server setup --jdbc-db={db_name} --jdbc-driver={path_to_jdbc} on server host.' ... View more

Has anyone tried manually upgrading to Spark 1.5.1 on Hortonworks Sandbox and faced any issues? ... View more

Hi, I am new to HDP and hadoop.I managed to install HDP 2.3 sandbox on Virtual box. I tried a few sample programs and they are working fine from the sandbox. I have installed Eclipse with Scala in my Windows machine. At present ,I use SBT and package my application and deploy the jar in the HDP Sandbox for execution. I would like to execute programs from my Eclipse against the HDP sandbox directly instead of packaging it each and every time. A sample code which I am trying to modify val conf = new SparkConf().setAppName(“Simple Application”).setMaster(“local[2]”).set(“spark.executor.memory”,”1g”) I guess , I have to change the local[2] to the master node / yarn cluster url. How do I get the url from the sandbox ? Any other configurations which has to be done on the Virtual box or on my code ? ... View more

While upgrading HDP 2.0 to HDP 2.1 and Metastore Schema from 0.12 to 0.13 I got the Error: Duplicate column name ‘OWNER_NAME’ (state=42S21,code=1060). The Metastore version is 0.12 in the VERSION Table however the ‘OWNER_NAME’ column in the ‘DBS’ table already exists. Here is the detailed error: +———————————————+ | < HIVE-6386: Add owner filed to database > | +———————————————+ 1 row selected (0.001 seconds) 0: jdbc:mysql://hadoop.domain> ALTER TABLE <code>DBS ADD OWNER_NAME varchar(128) Error: Duplicate column name ‘OWNER_NAME’ (state=42S21,code=1060) Closing: 0: jdbc:mysql://hadoop.domain/hive?createDatabaseIfNotExist=true org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent !! org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent !! at org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:242) at org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:211) at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:489) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: java.io.IOException: Schema script failed, errorcode 2 at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:377) at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:350) at org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:237) … 7 more *** schemaTool failed *** Has anyone run into the same issue? Any idea what is the source of this problem? ... View more