Re: Atlas hive import script (import_hive.sh) is f...

yjagadeesan · ‎09-27-2017

Deleting the kafka topics for atlas and restart of atlas fixed the issue.

yjagadeesan · ‎09-21-2017

Ah ok. Try this %jdbc(hive) You might need to check if the hive configuration is updated in the jdbc interpreter settings.

yjagadeesan · ‎09-21-2017

@Sudheer Velagapudi Here is the list of interpreters that exist in zeppelin. https://zeppelin.apache.org/supported_interpreters.html There is nothing specifically for SQL interpreter though. What exactly are you trying to do here?

yjagadeesan · ‎09-21-2017

This is a kerberized cluster on HDP 2.6.1 import_hive.sh is failing with the errors as below 2017-09-21 10:38:53,105 ERROR - [pool-2-thread-10 - f15b1a2e-6904-49bb-8ed5-b832632d4339:atlas:POST/api/atlas/entities/bedf80dd-deb1-42a9-81cc-40336a3d4546] ~ Unable to update entity by GUID bedf80dd-deb1-42a9-81cc-40336a3d4546 { "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"bedf80dd-deb1-42a9-81cc-40336a3d4546", "version":0, "typeName":"hive_db", "state":"ACTIVE" }, "typeName":"hive_db", "values":{ "name":"default", "location":"hdfs://CLUSTERNAME/apps/hive/warehouse", Using Hive configuration directory [/etc/hive/conf]Log file for import is /usr/hdp/current/atlas-server/logs/import-hive.logException in thread "main" org.apache.atlas.hook.AtlasHookException: HiveMetaStoreBridge.main() failed. at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:650)Caused by: org.apache.atlas.AtlasServiceException: Metadata service API org.apache.atlas.AtlasBaseClient$APIInfo@69c6161d failed with status 500 (Internal Server Error) Response Body ({"error":"Failed to notify for change PARTIAL_UPDATE"}) at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:337) at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:287) at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:429) at org.apache.atlas.AtlasClient.callAPIWithBodyAndParams(AtlasClient.java:1006) at org.apache.atlas.AtlasClient.updateEntity(AtlasClient.java:583) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.updateInstance(HiveMetaStoreBridge.java:526) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:175) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:140) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:134) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:647)Failed to import Hive Data Model!!!

yjagadeesan · ‎06-15-2017

@Manish Gupta After the installation use following steps to replace default service account with custom service account: For zeppelin: 1. Stop zeppelin service from Ambari. 2. Change the zeppelin user from Ambari server using configs.sh (this command is only available from ambari-server host): # /var/lib/ambari-server/resources/scripts/configs.sh -u <AmbariAdminUser> -p <AmbariAdminUserPassword> set localhost <Cluster-name> zeppelin-env zeppelin_user <ZEP-USER> 3. Set the ownership on zeppelin log and run directories. # chown -R <ZEP-USER>:hadoop /var/log/zeppelin # chown -R <ZEP-USER>:hadoop /var/run/zeppelin 4. Start zeppelin service from Ambari

yjagadeesan · ‎03-07-2017

Assumption - HDP 2.5.3 and above versions and kerberized cluster. Create a hplsql-site.xml as per the below template. <configuration> <property> <name>hplsql.conn.default</name> <value>hive2conn</value> <description>The default connection profile</description> </property> <property> <name>hplsql.conn.hiveconn</name> <value>org.apache.hive.jdbc.HiveDriver;jdbc:hive2://</value> <description>HiveServer2 JDBC connection (embedded mode)</description> </property> <property> <name>hplsql.conn.init.hiveconn</name> <value> set hive.execution.engine=mr; use default; </value> <description>Statements for execute after connection to the database</description> </property> <property> <name>hplsql.conn.convert.hiveconn</name>< <value>true</value> <description>Convert SQL statements before execution</description>< </property> <property> <name>hplsql.conn.hive1conn</name> <value>org.apache.hadoop.hive.jdbc.HiveDriver;jdbc:hive://</value> <description>Hive embedded JDBC (not requiring HiveServer)</description> </property> <property> <name>hplsql.conn.hive2conn</name> <value>org.apache.hive.jdbc.HiveDriver;jdbc:hive2://node1.field.hortonworks.com:10500/default;principal=hive/node1.field.hortonworks.com@REALM</value> <description>HiveServer2 JDBC connection</description> </property> <property> <name>hplsql.conn.init.hive2conn</name> <value> set hive.execution.engine=tez; use default; </value> <description>Statements for execute after connection to the database</description> </property> <property> <name>hplsql.conn.convert.hive2conn</name> <value>true</value> <description>Convert SQL statements before execution</description> </property> <property> <name>hplsql.conn.db2conn</name> <value>com.ibm.db2.jcc.DB2Driver;jdbc:db2://localhost:50001/dbname;user;password</value> <description>IBM DB2 connection</description> </property> <property> <name>hplsql.conn.tdconn</name> <value>com.teradata.jdbc.TeraDriver;jdbc:teradata://localhost/database=dbname,logmech=ldap;user;password</value> <description>Teradata connection</description> </property> <property> <name>hplsql.conn.mysqlconn</name> <value>com.mysql.jdbc.Driver;jdbc:mysql://localhost/test;user;password</value> <description>MySQL connection</description> </property> <property> <name>hplsql.dual.table</name> <value>default.dual</value> <description>Single row, single column table for internal operations</description> </property> <property> <name>hplsql.insert.values</name> <value>native</value> <description>How to execute INSERT VALUES statement: native (default) and select</description> </property> <property> <name>hplsql.onerror</name> <value>exception</value> <description>Error handling behavior: exception (default), seterror and stop</description> </property> <property> <name>hplsql.temp.tables</name> <value>native</value> <description>Temporary tables: native (default) and managed</description> </property> <property> <name>hplsql.temp.tables.schema</name> <value></value> <description>Schema for managed temporary tables</description> </property> <property> <name>hplsql.temp.tables.location</name> <value>/tmp/plhql</value> <description>LOcation for managed temporary tables in HDFS</description> </property> </configuration> Modify the LLAP hostname and the hive Principal based on the cluster environment in the following section Note: This is a kerberized cluster <property> <name>hplsql.conn.hive2conn</name> <value>org.apache.hive.jdbc.HiveDriver;jdbc:hive2://<<LLAP_HOSTNAME>>:10500/default;principal=hive/<<LLAPHOSTNAME>>@<<KERBEROS_REALM>></value> <description>HiveServer2 JDBC connection</description> </property> <property> <name>hplsql.conn.init.hive2conn</name> <value> set hive.execution.engine=tez; use default; </value> <description>Statements for execute after connection to the database</description> </property> Update the hive-hplsql jar file with the modified hplsql-site.xml cd /usr/hdp/current/hive-server2-hive2/lib; /usr/jdk64/jdk1.8.0_77/bin/jar uf hive-hplsql-2.1.0.XXX.jar hplsql-site.xml; Note: Please refer to your JDK version path Authenticate the user with the KDC kinit <user principal> Execute the HPLSQL code as below ./hplsql -f /root/myhpl.sql If success then you must be seeing the logs as below, Starting SQL statement SQL statement executed successfully (128 ms) Starting SQL statement SQL statement executed successfully (145 ms)

yjagadeesan · ‎02-23-2017

@prachi sharma Whats your default ssh_username for the image or as per your cloud environment? CBD uses the user 'cloudbreak'

yjagadeesan · ‎12-12-2016

@Nube Technologies Good to hear that.

yjagadeesan · ‎12-09-2016

@Nube Technologies For Hive: You can use Sqoop with the --target-dir parameter set to a directory inside the Hive encryption zone. You need to specify the -D option after sqoop import. sqoop import \-D sqoop.test.import.rootDir=<root-directory> \--target-dir <directory-inside-encryption-zone> \<additional-arguments> For append or incremental export Make sure that the sqoop.test.import.rootDir property points to the encryption zone specified in the --target-dir argument. For HCatalog: No config required. For more information on HDP services for HDFS encryption refer this link below, HDP services for HDFS encryption Let me know if you have any other questions

yjagadeesan · ‎12-08-2016

@Huahua Wei I dont think so.Transition to Failover node does not take much of time. Whichever NameNode is started first will become active. You may choose to start the cluster in a specific order such that your preferred node starts first. What problem exactly are you facing ? Is it the failover or starting up the namenode is taking lot of time?

yjagadeesan · ‎12-07-2016

@Davide Vergari RHEL / Centos 7 is not supported for HDB You can find the link as below, http://hdb.docs.pivotal.io/210/hdb/releasenotes/HAWQ210ReleaseNotes.html

yjagadeesan · ‎12-07-2016

@Baruch AMOUSSOU DJANGBAN Assuming ambari2 user can stop the services, Why don't you try using ambari-agent stop instead of systemctl stop ambari-agent.service

yjagadeesan · ‎09-12-2016

@KheraIs your namenode configured on HA ?

yjagadeesan · ‎09-07-2016

@jigar.patel Can you accept this answer to close this post ? Thanks

yjagadeesan · ‎09-06-2016

Post here if you could not do the distcp inspite of the above mentioned steps.

yjagadeesan · ‎09-06-2016

@jigar.patel There has been already answers. Looks at this links for more info , https://community.hortonworks.com/questions/294/running-distcp-between-two-cluster-one-kerberized.html https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_Sys_Admin_Guides/content/distcp_and_security_settings.html

yjagadeesan · ‎09-01-2016

Thanks @njayakumar Incase of nested partitions how will that be handled ?

yjagadeesan · ‎09-01-2016

@Sunile Manjee I understand KNOX currently do not support HAWQ since it does not have support to web rest api at this moment. HAWQ handles authentication like any other database like oracle ... or internally.

yjagadeesan · ‎09-01-2016

I understand HAWQ can handle Primary key partition. In the HAWQ to Hive migration what is the best suited approach to handle data ingestion ?

yjagadeesan · ‎08-31-2016

@Bjorn Olsen Refer this knox support for high availability in HDP 2.5 https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_release-notes/content/new_features.html

yjagadeesan · ‎08-31-2016

@njayakumar Ranger REST API's are exposed and which can utilized to integrate with the IDM. Documention can be found at https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_security/content/apache_ranger_public_rest_api.html Apache Documentation https://cwiki.apache.org/confluence/display/RANGER/REST+APIs+for+Policy+Management Let me know if you have further questions.

yjagadeesan · ‎08-19-2016

@Bjorn Olsen Knox support for HBASE HA is fixed in the later versions of Apache Knox as you mentioned. Rather than upgrading knox manually you can wait for HDP 2.5 release.

yjagadeesan · ‎06-23-2016

Go with the recommendations above as @Jitendra Yadav has recommended.

yjagadeesan · ‎06-23-2016

@ARUNKUMAR RAMASAMY Yes change it. You might need to restart the HDFS and other services as ambari suggests.

yjagadeesan · ‎06-23-2016

@ARUNKUMAR RAMASAMY Check the heapsize and ulimits(hdfs user)

yjagadeesan · ‎06-21-2016

@Nihar Kanike Look at this link below https://community.hortonworks.com/questions/15506/error-cannot-retrieve-repository-metadata-repomdxm.html The errors are very similar.

yjagadeesan · ‎06-16-2016

@Alexander Feldman It will be interesting to know what process you are trying to monitor. Normally the Alert definitions are tagged to a component name while you define. But might be there are options which I'm not aware of though. One of the example - https://github.com/monolive/ambari-custom-alerts

yjagadeesan · ‎06-16-2016

@Kaliyug Antagonist HDFS attempts to recover from the corruption by itself. I would suggest you to follow this articles and HCC link to try your luck. https://community.hortonworks.com/questions/17917/best-way-of-handling-corrupt-or-missing-blocks.html Fix the corrupt block

yjagadeesan · ‎06-15-2016

@chandramouli muthukumaran No it does not store multiple times. Good Luck with your hadooping.

yjagadeesan · ‎06-15-2016

@chandramouli muthukumaran if you intend to use the /opt/Symantec/hadoop/hdfs/namenode for namenode and /opt/Symantec/hadoop/hdfs/data only then you can remove the other entries. Save and then you might need to restart the HDFS and corresponding services as indicated by ambari. Incase if you have multiple directories for Data then add those entries like this. /opt/Symantec/hadoop/hdfs/data1,/opt/Symantec/hadoop/hdfs/data2,/opt/Symantec/hadoop/hdfs/data3 Hope this helps.

Online	Offline
Last Visited	‎02-25-2021 10:54 PM

Date Registered	‎01-05-2016 02:27 PM
Last Visited	‎02-25-2021 10:54 PM
Total Messages Posted	56
Total Kudos Received	23

Re: Atlas hive import script (import_hive.sh) is f...

Re: Zeppelin Interpreter

Re: How to use custom service accounts, instead of...

Re: How to use sqoop to import the data in to Encr...

Re: How we can distcp between secured and unsecure...

Re: Atlas hive import script (import_hive.sh) is f...

Re: Zeppelin Interpreter

Re: Zeppelin Interpreter

Atlas hive import script (import_hive.sh) is faili...

Re: How to use custom service accounts, instead of...

Configure HIVE HPLSQL

Re: Infrastructure creation failed while deploying...

Re: How to use sqoop to import the data in to Encr...

Re: How to use sqoop to import the data in to Encr...

Re: How to restart Namenode and standby Namenode?

Re: HDB support for RHEL/CentOS 7

Re: How can i write a script to automate stop of a...

Re: Hive View ambari throwing error.

Re: How we can distcp between secured and unsecure...

Re: How we can distcp between secured and unsecure...

Re: How we can distcp between secured and unsecure...

Re: Primary key partition handling from HAWQ to hi...

Re: Does Knox support HAWQ (HDB)?

Primary key partition handling from HAWQ to hive

Re: Does Knox 0.6.0 on HDP 2.4.2 support HBase Hig...

Re: Oracle IDM integration with Ranger

Re: Does Knox 0.6.0 on HDP 2.4.2 support HBase Hig...

Re: Data node down

Re: Data node down

Re: Data node down

Re: Ambari installation failed at Install, Start a...

Re: can I make Ambari to monitor processes running...

Re: HDFS corrupted on a newly installed cluster

Re: Name Node and Data Node Directories

Re: Name Node and Data Node Directories