Re: Atlas hive import script (import_hive.sh) is f...

yjagadeesan · ‎09-27-2017

Deleting the kafka topics for atlas and restart of atlas fixed the issue.

yjagadeesan · ‎09-21-2017

Ah ok. Try this %jdbc(hive) You might need to check if the hive configuration is updated in the jdbc interpreter settings.

yjagadeesan · ‎09-21-2017

@Sudheer Velagapudi Here is the list of interpreters that exist in zeppelin. https://zeppelin.apache.org/supported_interpreters.html There is nothing specifically for SQL interpreter though. What exactly are you trying to do here?

yjagadeesan · ‎09-21-2017

This is a kerberized cluster on HDP 2.6.1 import_hive.sh is failing with the errors as below 2017-09-21 10:38:53,105 ERROR - [pool-2-thread-10 - f15b1a2e-6904-49bb-8ed5-b832632d4339:atlas:POST/api/atlas/entities/bedf80dd-deb1-42a9-81cc-40336a3d4546] ~ Unable to update entity by GUID bedf80dd-deb1-42a9-81cc-40336a3d4546 { "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"bedf80dd-deb1-42a9-81cc-40336a3d4546", "version":0, "typeName":"hive_db", "state":"ACTIVE" }, "typeName":"hive_db", "values":{ "name":"default", "location":"hdfs://CLUSTERNAME/apps/hive/warehouse", Using Hive configuration directory [/etc/hive/conf]Log file for import is /usr/hdp/current/atlas-server/logs/import-hive.logException in thread "main" org.apache.atlas.hook.AtlasHookException: HiveMetaStoreBridge.main() failed. at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:650)Caused by: org.apache.atlas.AtlasServiceException: Metadata service API org.apache.atlas.AtlasBaseClient$APIInfo@69c6161d failed with status 500 (Internal Server Error) Response Body ({"error":"Failed to notify for change PARTIAL_UPDATE"}) at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:337) at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:287) at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:429) at org.apache.atlas.AtlasClient.callAPIWithBodyAndParams(AtlasClient.java:1006) at org.apache.atlas.AtlasClient.updateEntity(AtlasClient.java:583) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.updateInstance(HiveMetaStoreBridge.java:526) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:175) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:140) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:134) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:647)Failed to import Hive Data Model!!!

yjagadeesan · ‎06-15-2017

@Manish Gupta After the installation use following steps to replace default service account with custom service account: For zeppelin: 1. Stop zeppelin service from Ambari. 2. Change the zeppelin user from Ambari server using configs.sh (this command is only available from ambari-server host): # /var/lib/ambari-server/resources/scripts/configs.sh -u <AmbariAdminUser> -p <AmbariAdminUserPassword> set localhost <Cluster-name> zeppelin-env zeppelin_user <ZEP-USER> 3. Set the ownership on zeppelin log and run directories. # chown -R <ZEP-USER>:hadoop /var/log/zeppelin # chown -R <ZEP-USER>:hadoop /var/run/zeppelin 4. Start zeppelin service from Ambari

yjagadeesan · ‎03-07-2017

Assumption - HDP 2.5.3 and above versions and kerberized cluster. Create a hplsql-site.xml as per the below template. <configuration> <property> <name>hplsql.conn.default</name> <value>hive2conn</value> <description>The default connection profile</description> </property> <property> <name>hplsql.conn.hiveconn</name> <value>org.apache.hive.jdbc.HiveDriver;jdbc:hive2://</value> <description>HiveServer2 JDBC connection (embedded mode)</description> </property> <property> <name>hplsql.conn.init.hiveconn</name> <value> set hive.execution.engine=mr; use default; </value> <description>Statements for execute after connection to the database</description> </property> <property> <name>hplsql.conn.convert.hiveconn</name>< <value>true</value> <description>Convert SQL statements before execution</description>< </property> <property> <name>hplsql.conn.hive1conn</name> <value>org.apache.hadoop.hive.jdbc.HiveDriver;jdbc:hive://</value> <description>Hive embedded JDBC (not requiring HiveServer)</description> </property> <property> <name>hplsql.conn.hive2conn</name> <value>org.apache.hive.jdbc.HiveDriver;jdbc:hive2://node1.field.hortonworks.com:10500/default;principal=hive/node1.field.hortonworks.com@REALM</value> <description>HiveServer2 JDBC connection</description> </property> <property> <name>hplsql.conn.init.hive2conn</name> <value> set hive.execution.engine=tez; use default; </value> <description>Statements for execute after connection to the database</description> </property> <property> <name>hplsql.conn.convert.hive2conn</name> <value>true</value> <description>Convert SQL statements before execution</description> </property> <property> <name>hplsql.conn.db2conn</name> <value>com.ibm.db2.jcc.DB2Driver;jdbc:db2://localhost:50001/dbname;user;password</value> <description>IBM DB2 connection</description> </property> <property> <name>hplsql.conn.tdconn</name> <value>com.teradata.jdbc.TeraDriver;jdbc:teradata://localhost/database=dbname,logmech=ldap;user;password</value> <description>Teradata connection</description> </property> <property> <name>hplsql.conn.mysqlconn</name> <value>com.mysql.jdbc.Driver;jdbc:mysql://localhost/test;user;password</value> <description>MySQL connection</description> </property> <property> <name>hplsql.dual.table</name> <value>default.dual</value> <description>Single row, single column table for internal operations</description> </property> <property> <name>hplsql.insert.values</name> <value>native</value> <description>How to execute INSERT VALUES statement: native (default) and select</description> </property> <property> <name>hplsql.onerror</name> <value>exception</value> <description>Error handling behavior: exception (default), seterror and stop</description> </property> <property> <name>hplsql.temp.tables</name> <value>native</value> <description>Temporary tables: native (default) and managed</description> </property> <property> <name>hplsql.temp.tables.schema</name> <value></value> <description>Schema for managed temporary tables</description> </property> <property> <name>hplsql.temp.tables.location</name> <value>/tmp/plhql</value> <description>LOcation for managed temporary tables in HDFS</description> </property> </configuration> Modify the LLAP hostname and the hive Principal based on the cluster environment in the following section Note: This is a kerberized cluster <property> <name>hplsql.conn.hive2conn</name> <value>org.apache.hive.jdbc.HiveDriver;jdbc:hive2://<<LLAP_HOSTNAME>>:10500/default;principal=hive/<<LLAPHOSTNAME>>@<<KERBEROS_REALM>></value> <description>HiveServer2 JDBC connection</description> </property> <property> <name>hplsql.conn.init.hive2conn</name> <value> set hive.execution.engine=tez; use default; </value> <description>Statements for execute after connection to the database</description> </property> Update the hive-hplsql jar file with the modified hplsql-site.xml cd /usr/hdp/current/hive-server2-hive2/lib; /usr/jdk64/jdk1.8.0_77/bin/jar uf hive-hplsql-2.1.0.XXX.jar hplsql-site.xml; Note: Please refer to your JDK version path Authenticate the user with the KDC kinit <user principal> Execute the HPLSQL code as below ./hplsql -f /root/myhpl.sql If success then you must be seeing the logs as below, Starting SQL statement SQL statement executed successfully (128 ms) Starting SQL statement SQL statement executed successfully (145 ms)

yjagadeesan · ‎02-23-2017

@prachi sharma Whats your default ssh_username for the image or as per your cloud environment? CBD uses the user 'cloudbreak'

yjagadeesan · ‎12-12-2016

@Nube Technologies Good to hear that.

yjagadeesan · ‎12-09-2016

@Nube Technologies For Hive: You can use Sqoop with the --target-dir parameter set to a directory inside the Hive encryption zone. You need to specify the -D option after sqoop import. sqoop import \-D sqoop.test.import.rootDir=<root-directory> \--target-dir <directory-inside-encryption-zone> \<additional-arguments> For append or incremental export Make sure that the sqoop.test.import.rootDir property points to the encryption zone specified in the --target-dir argument. For HCatalog: No config required. For more information on HDP services for HDFS encryption refer this link below, HDP services for HDFS encryption Let me know if you have any other questions

yjagadeesan · ‎12-08-2016

@Huahua Wei I dont think so.Transition to Failover node does not take much of time. Whichever NameNode is started first will become active. You may choose to start the cluster in a specific order such that your preferred node starts first. What problem exactly are you facing ? Is it the failover or starting up the namenode is taking lot of time?

yjagadeesan · ‎12-07-2016

@Baruch AMOUSSOU DJANGBAN Assuming ambari2 user can stop the services, Why don't you try using ambari-agent stop instead of systemctl stop ambari-agent.service

yjagadeesan · ‎09-07-2016

@jigar.patel Can you accept this answer to close this post ? Thanks

yjagadeesan · ‎09-06-2016

Post here if you could not do the distcp inspite of the above mentioned steps.

yjagadeesan · ‎09-06-2016

@jigar.patel There has been already answers. Looks at this links for more info , https://community.hortonworks.com/questions/294/running-distcp-between-two-cluster-one-kerberized.html https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_Sys_Admin_Guides/content/distcp_and_security_settings.html

yjagadeesan · ‎09-01-2016

Thanks @njayakumar Incase of nested partitions how will that be handled ?

yjagadeesan · ‎09-01-2016

@Sunile Manjee I understand KNOX currently do not support HAWQ since it does not have support to web rest api at this moment. HAWQ handles authentication like any other database like oracle ... or internally.

yjagadeesan · ‎09-01-2016

I understand HAWQ can handle Primary key partition. In the HAWQ to Hive migration what is the best suited approach to handle data ingestion ?

yjagadeesan · ‎08-31-2016

@njayakumar Ranger REST API's are exposed and which can utilized to integrate with the IDM. Documention can be found at https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_security/content/apache_ranger_public_rest_api.html Apache Documentation https://cwiki.apache.org/confluence/display/RANGER/REST+APIs+for+Policy+Management Let me know if you have further questions.

yjagadeesan · ‎06-23-2016

Go with the recommendations above as @Jitendra Yadav has recommended.

yjagadeesan · ‎06-23-2016

@ARUNKUMAR RAMASAMY Yes change it. You might need to restart the HDFS and other services as ambari suggests.

yjagadeesan · ‎06-23-2016

@ARUNKUMAR RAMASAMY Check the heapsize and ulimits(hdfs user)

yjagadeesan · ‎06-16-2016

@Alexander Feldman It will be interesting to know what process you are trying to monitor. Normally the Alert definitions are tagged to a component name while you define. But might be there are options which I'm not aware of though. One of the example - https://github.com/monolive/ambari-custom-alerts

yjagadeesan · ‎06-15-2016

@chandramouli muthukumaran No it does not store multiple times. Good Luck with your hadooping.

yjagadeesan · ‎06-15-2016

@chandramouli muthukumaran if you intend to use the /opt/Symantec/hadoop/hdfs/namenode for namenode and /opt/Symantec/hadoop/hdfs/data only then you can remove the other entries. Save and then you might need to restart the HDFS and corresponding services as indicated by ambari. Incase if you have multiple directories for Data then add those entries like this. /opt/Symantec/hadoop/hdfs/data1,/opt/Symantec/hadoop/hdfs/data2,/opt/Symantec/hadoop/hdfs/data3 Hope this helps.

yjagadeesan · ‎06-13-2016

Thanks @Jonathan Hurley I was able to get the "under replicated count" through the AMS Metrics and here is the REST API call. curl -u admin:admin -H "X-Requested-by:ambari" -i -k -X GET http://sandbox.hortonworks.com:6188/ws/v1/timeline/metrics?metricNames=kafka.controller.KafkaController.ActiveControllerCount

yjagadeesan · ‎06-13-2016

Currently ambari alerts for kafka is just to monitor the kafka broker process on a particular port. Is there a way we can monitor and alert any other parameters like under replicated counts or active controller count etc... ? After upgrading to Ambari 2.2.2 we are able to create and add new widgets for these fields. But there is nothing in the alerts section. Has anyone been successful in setting up the custom alerting for these parameters?

yjagadeesan · ‎06-07-2016

Never mind. I was able to figure this. Thanks.

yjagadeesan · ‎06-07-2016

@Constantin Stanca Thanks for the Post. Is there a way we can use REST API call to retrieve the metrics (eg - Active controller count or Broker Topics etc... )

yjagadeesan · ‎06-06-2016

You can try as below li <- read.table(textConnection(c), sep = ",");

yjagadeesan · ‎06-06-2016

@Simran Kaur When you convert using as.list then your data will look like the above. Whats your problem here? how do you want the data to look like? What are you trying to do here?

Online	Offline
Last Visited	‎04-11-2022 02:16 PM

Date Registered	‎01-05-2016 02:27 PM
Last Visited	‎04-11-2022 02:16 PM
Total Messages Posted	56
Total Kudos Received	23

Re: Atlas hive import script (import_hive.sh) is f...

Re: Zeppelin Interpreter

Re: How to use custom service accounts, instead of...

Re: How to use sqoop to import the data in to Encr...

Re: How we can distcp between secured and unsecure...

Re: Atlas hive import script (import_hive.sh) is f...

Re: Zeppelin Interpreter

Re: Zeppelin Interpreter

Atlas hive import script (import_hive.sh) is faili...

Re: How to use custom service accounts, instead of...

Configure HIVE HPLSQL

Re: Infrastructure creation failed while deploying...

Re: How to use sqoop to import the data in to Encr...

Re: How to use sqoop to import the data in to Encr...

Re: How to restart Namenode and standby Namenode?

Re: How can i write a script to automate stop of a...

Re: How we can distcp between secured and unsecure...

Re: How we can distcp between secured and unsecure...

Re: How we can distcp between secured and unsecure...

Re: Primary key partition handling from HAWQ to hi...

Re: Does Knox support HAWQ (HDB)?

Primary key partition handling from HAWQ to hive

Re: Oracle IDM integration with Ranger

Re: Data node down

Re: Data node down

Re: Data node down

Re: can I make Ambari to monitor processes running...

Re: Name Node and Data Node Directories

Re: Name Node and Data Node Directories

Re: Kafka Custom alert with ambari 2.2.2

Kafka Custom alert with ambari 2.2.2

Re: Kafka Topics and Brokers Metrics with Ambari 2...

Re: Kafka Topics and Brokers Metrics with Ambari 2...

Re: Saving a list in HDFS with headers using R

Re: Saving a list in HDFS with headers using R