Member since
01-05-2016
56
Posts
23
Kudos Received
9
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1109 | 09-27-2017 06:11 AM | |
843 | 09-21-2017 06:36 PM | |
573 | 06-15-2017 01:28 PM | |
1065 | 12-09-2016 08:39 PM | |
1159 | 09-06-2016 04:57 PM |
09-27-2017
06:11 AM
1 Kudo
Deleting the kafka topics for atlas and restart of atlas fixed the issue.
... View more
09-21-2017
06:39 PM
1 Kudo
Ah ok. Try this %jdbc(hive) You might need to check if the hive configuration is updated in the jdbc interpreter settings.
... View more
09-21-2017
06:36 PM
1 Kudo
@Sudheer Velagapudi
Here is the list of interpreters that exist in zeppelin.
https://zeppelin.apache.org/supported_interpreters.html
There is nothing specifically for SQL interpreter though. What exactly are you trying to do here?
... View more
09-21-2017
06:32 PM
This is a kerberized cluster on HDP 2.6.1 import_hive.sh is failing with the errors as below 2017-09-21 10:38:53,105 ERROR - [pool-2-thread-10 - f15b1a2e-6904-49bb-8ed5-b832632d4339:atlas:POST/api/atlas/entities/bedf80dd-deb1-42a9-81cc-40336a3d4546] ~ Unable to update entity by GUID bedf80dd-deb1-42a9-81cc-40336a3d4546 { "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"bedf80dd-deb1-42a9-81cc-40336a3d4546", "version":0, "typeName":"hive_db", "state":"ACTIVE" }, "typeName":"hive_db", "values":{ "name":"default", "location":"hdfs://CLUSTERNAME/apps/hive/warehouse", Using Hive configuration directory [/etc/hive/conf]Log file for import is /usr/hdp/current/atlas-server/logs/import-hive.logException in thread "main" org.apache.atlas.hook.AtlasHookException: HiveMetaStoreBridge.main() failed. at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:650)Caused by: org.apache.atlas.AtlasServiceException: Metadata service API org.apache.atlas.AtlasBaseClient$APIInfo@69c6161d failed with status 500 (Internal Server Error) Response Body ({"error":"Failed to notify for change PARTIAL_UPDATE"}) at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:337) at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:287) at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:429) at org.apache.atlas.AtlasClient.callAPIWithBodyAndParams(AtlasClient.java:1006) at org.apache.atlas.AtlasClient.updateEntity(AtlasClient.java:583) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.updateInstance(HiveMetaStoreBridge.java:526) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:175) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:140) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:134) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:647)Failed to import Hive Data Model!!!
... View more
Labels:
- Labels:
-
Apache Atlas
06-15-2017
01:28 PM
1 Kudo
@Manish Gupta After the installation use following steps to replace default service account with custom service account: For zeppelin: 1. Stop zeppelin service from Ambari. 2. Change the zeppelin user from Ambari server using configs.sh (this command is only available from ambari-server host): # /var/lib/ambari-server/resources/scripts/configs.sh -u <AmbariAdminUser> -p <AmbariAdminUserPassword> set localhost <Cluster-name> zeppelin-env zeppelin_user <ZEP-USER> 3. Set the ownership on zeppelin log and run directories. # chown -R <ZEP-USER>:hadoop /var/log/zeppelin
# chown -R <ZEP-USER>:hadoop /var/run/zeppelin 4. Start zeppelin service from Ambari
... View more
03-07-2017
05:07 PM
3 Kudos
Assumption - HDP 2.5.3 and above versions and kerberized cluster. Create a hplsql-site.xml as per the below template.
<configuration>
<property>
<name>hplsql.conn.default</name>
<value>hive2conn</value>
<description>The default connection profile</description>
</property>
<property>
<name>hplsql.conn.hiveconn</name>
<value>org.apache.hive.jdbc.HiveDriver;jdbc:hive2://</value>
<description>HiveServer2 JDBC connection (embedded mode)</description>
</property>
<property>
<name>hplsql.conn.init.hiveconn</name>
<value>
set hive.execution.engine=mr;
use default;
</value>
<description>Statements for execute after connection to the database</description>
</property>
<property>
<name>hplsql.conn.convert.hiveconn</name><
<value>true</value>
<description>Convert SQL statements before execution</description><
</property>
<property>
<name>hplsql.conn.hive1conn</name>
<value>org.apache.hadoop.hive.jdbc.HiveDriver;jdbc:hive://</value>
<description>Hive embedded JDBC (not requiring HiveServer)</description>
</property>
<property>
<name>hplsql.conn.hive2conn</name>
<value>org.apache.hive.jdbc.HiveDriver;jdbc:hive2://node1.field.hortonworks.com:10500/default;principal=hive/node1.field.hortonworks.com@REALM</value>
<description>HiveServer2 JDBC connection</description>
</property>
<property>
<name>hplsql.conn.init.hive2conn</name>
<value>
set hive.execution.engine=tez;
use default;
</value>
<description>Statements for execute after connection to the database</description>
</property>
<property>
<name>hplsql.conn.convert.hive2conn</name>
<value>true</value>
<description>Convert SQL statements before execution</description>
</property>
<property>
<name>hplsql.conn.db2conn</name>
<value>com.ibm.db2.jcc.DB2Driver;jdbc:db2://localhost:50001/dbname;user;password</value>
<description>IBM DB2 connection</description>
</property>
<property>
<name>hplsql.conn.tdconn</name>
<value>com.teradata.jdbc.TeraDriver;jdbc:teradata://localhost/database=dbname,logmech=ldap;user;password</value>
<description>Teradata connection</description>
</property>
<property>
<name>hplsql.conn.mysqlconn</name>
<value>com.mysql.jdbc.Driver;jdbc:mysql://localhost/test;user;password</value>
<description>MySQL connection</description>
</property>
<property>
<name>hplsql.dual.table</name>
<value>default.dual</value>
<description>Single row, single column table for internal operations</description>
</property>
<property>
<name>hplsql.insert.values</name>
<value>native</value>
<description>How to execute INSERT VALUES statement: native (default) and select</description>
</property>
<property>
<name>hplsql.onerror</name>
<value>exception</value>
<description>Error handling behavior: exception (default), seterror and stop</description>
</property>
<property>
<name>hplsql.temp.tables</name>
<value>native</value>
<description>Temporary tables: native (default) and managed</description>
</property>
<property>
<name>hplsql.temp.tables.schema</name>
<value></value>
<description>Schema for managed temporary tables</description>
</property>
<property>
<name>hplsql.temp.tables.location</name>
<value>/tmp/plhql</value>
<description>LOcation for managed temporary tables in HDFS</description>
</property>
</configuration>
Modify
the LLAP hostname and the hive Principal based on the cluster environment in the following section Note: This is a kerberized cluster
<property>
<name>hplsql.conn.hive2conn</name>
<value>org.apache.hive.jdbc.HiveDriver;jdbc:hive2://<<LLAP_HOSTNAME>>:10500/default;principal=hive/<<LLAPHOSTNAME>>@<<KERBEROS_REALM>></value>
<description>HiveServer2 JDBC connection</description>
</property>
<property>
<name>hplsql.conn.init.hive2conn</name>
<value>
set hive.execution.engine=tez;
use default;
</value>
<description>Statements for execute after connection to the database</description>
</property>
Update the
hive-hplsql jar file with the modified hplsql-site.xml
cd /usr/hdp/current/hive-server2-hive2/lib;
/usr/jdk64/jdk1.8.0_77/bin/jar uf hive-hplsql-2.1.0.XXX.jar hplsql-site.xml; Note: Please refer to your JDK version path
Authenticate the user with the KDC
kinit <user principal>
Execute the HPLSQL code as below
./hplsql -f /root/myhpl.sql
If success then you must be seeing the logs as below, Starting SQL statement
SQL statement executed successfully (128 ms)
Starting SQL statement
SQL statement executed successfully (145 ms)
... View more
02-23-2017
11:35 PM
@prachi sharma Whats your default ssh_username for the image or as per your cloud environment? CBD uses the user 'cloudbreak'
... View more
12-12-2016
03:20 AM
@Nube Technologies Good to hear that.
... View more
12-09-2016
08:39 PM
2 Kudos
@Nube Technologies For Hive: You can use Sqoop with the --target-dir parameter set to a directory inside the Hive encryption zone. You need to specify the -D option after sqoop import. sqoop import \-D sqoop.test.import.rootDir=<root-directory> \--target-dir <directory-inside-encryption-zone> \<additional-arguments>
For append or incremental export Make sure that the sqoop.test.import.rootDir property points to the encryption zone specified in the --target-dir argument. For HCatalog: No config required. For more information on HDP services for HDFS encryption refer this link below, HDP services for HDFS encryption Let me know if you have any other questions
... View more
12-08-2016
02:16 PM
1 Kudo
@Huahua Wei I dont think so.Transition to Failover node does not take much of time. Whichever NameNode is started first will become active. You may choose to start the cluster in a specific order such that your preferred node starts first. What problem exactly are you facing ? Is it the failover or starting up the namenode is taking lot of time?
... View more
12-07-2016
02:41 PM
@Baruch AMOUSSOU DJANGBAN Assuming ambari2 user can stop the services, Why don't you try using ambari-agent stop instead of systemctl stop ambari-agent.service
... View more
09-07-2016
07:35 PM
@jigar.patel Can you accept this answer to close this post ? Thanks
... View more
09-06-2016
05:01 PM
Post here if you could not do the distcp inspite of the above mentioned steps.
... View more
09-06-2016
04:57 PM
1 Kudo
@jigar.patel There has been already answers. Looks at this links for more info , https://community.hortonworks.com/questions/294/running-distcp-between-two-cluster-one-kerberized.html https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_Sys_Admin_Guides/content/distcp_and_security_settings.html
... View more
09-01-2016
01:09 PM
1 Kudo
Thanks @njayakumar Incase of nested partitions how will that be handled ?
... View more
09-01-2016
12:58 PM
@Sunile Manjee I understand KNOX currently do not support HAWQ since it does not have support to web rest api at this moment. HAWQ handles authentication like any other database like oracle ... or internally.
... View more
09-01-2016
12:55 PM
1 Kudo
I understand HAWQ can handle Primary key partition. In the HAWQ to Hive migration what is the best suited approach to handle data ingestion ?
... View more
Labels:
- Labels:
-
Apache Hive
08-31-2016
06:05 PM
1 Kudo
@njayakumar Ranger REST API's are exposed and which can utilized to integrate with the IDM. Documention can be found at https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_security/content/apache_ranger_public_rest_api.html Apache Documentation https://cwiki.apache.org/confluence/display/RANGER/REST+APIs+for+Policy+Management Let me know if you have further questions.
... View more
06-23-2016
02:42 PM
Go with the recommendations above as @Jitendra Yadav has recommended.
... View more
06-23-2016
02:33 PM
@ARUNKUMAR RAMASAMY Yes change it. You might need to restart the HDFS and other services as ambari suggests.
... View more
06-16-2016
03:44 PM
@Alexander Feldman It will be interesting to know what process you are trying to monitor. Normally the Alert definitions are tagged to a component name while you define. But might be there are options which I'm not aware of though. One of the example - https://github.com/monolive/ambari-custom-alerts
... View more
06-15-2016
06:11 PM
@chandramouli muthukumaran No it does not store multiple times. Good Luck with your hadooping.
... View more
06-15-2016
06:01 PM
@chandramouli muthukumaran if you intend to use the /opt/Symantec/hadoop/hdfs/namenode for namenode and /opt/Symantec/hadoop/hdfs/data only then you can remove the other entries. Save and then you might need to restart the HDFS and corresponding services as indicated by ambari. Incase if you have multiple directories for Data then add those entries like this. /opt/Symantec/hadoop/hdfs/data1,/opt/Symantec/hadoop/hdfs/data2,/opt/Symantec/hadoop/hdfs/data3 Hope this helps.
... View more
06-13-2016
07:25 PM
Thanks @Jonathan Hurley I was able to get the "under replicated count" through the AMS Metrics and here is the REST API call. curl -u admin:admin -H "X-Requested-by:ambari" -i -k -X GET http://sandbox.hortonworks.com:6188/ws/v1/timeline/metrics?metricNames=kafka.controller.KafkaController.ActiveControllerCount
... View more
06-13-2016
06:59 PM
1 Kudo
Currently ambari alerts for kafka is just to monitor the kafka broker process on a particular port. Is there a way we can monitor and alert any other parameters like under replicated counts or active controller count etc... ? After upgrading to Ambari 2.2.2 we are able to create and add new widgets for these fields. But there is nothing in the alerts section. Has anyone been successful in setting up the custom alerting for these parameters?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Kafka
06-07-2016
09:05 PM
Never mind. I was able to figure this. Thanks.
... View more
06-07-2016
07:34 PM
@Constantin Stanca Thanks for the Post. Is there a way we can use REST API call to retrieve the metrics (eg - Active controller count or Broker Topics etc... )
... View more
06-06-2016
07:25 PM
You can try as below li <- read.table(textConnection(c), sep = ",");
... View more
06-06-2016
05:16 PM
@Simran Kaur When you convert using as.list then your data will look like the above. Whats your problem here? how do you want the data to look like? What are you trying to do here?
... View more