Re: Hortonworks (Hive - ldap integration failure)

Prakashcit · ‎05-30-2021

[hdfs@c****-node* hive-testbench-hive14]$ ./tpcds-build.sh Building TPC-DS Data Generator make: Nothing to be done for `all’. TPC-DS Data Generator built, you can now use tpcds-setup.sh to generate data. [hdfs@c4237-node2 hive-testbench-hive14]$ ./tpcds-setup.sh 2 TPC-DS text data generation complete. Loading text data into external tables. make: *** [time_dim] Error 1 make: *** Waiting for unfinished jobs.... make: *** [date_dim] Error 1 Data loaded into database tpcds_bin_partitioned_orc_2. INFO : OK +---------------------+ | database_name | +---------------------+ | default | | information_schema | | sys | +---------------------+ 3 rows selected (1.955 seconds) 0: jdbc:hive2://c4237-node2.coelab.cloudera.c> tpcds_bin_partitioned_orc_2 database is not created, I have some issues in testing the tpcds queries sudo -u hdfs -s 13 cd /home/hdfs 14 wget https://github.com/hortonworks/hive-testbench/archive/hive14.zip 15 unzip hive14.zip 17 export JAVA_HOME=/usr/jdk64/jdk1.8.0_77 18 export PATH=$JAVA_HOME/bin:$PATH ./tpcds-build.sh beeline -i testbench.settings -u "jdbc:hive2://c****-node9.coe***.*****.com:10500/tpcds_bin_partitioned_orc_2" I'm not able to test the tpcds queries, any help would be appreciated.

Prakashcit · ‎02-22-2021

@ARP There is a bug https://issues.apache.org/jira/browse/HIVE-24693 Kindly use this work around properties and test your jobs. set hive.parquet.timestamp.time.unit=nanos; set hive.parquet.write.int64.timestamp=true;

Prakashcit · ‎02-16-2021

@ARP try increasing fs.s3a.connection.maximum to 1500 and follow this doc for the fine S3 tuning parameters. https://docs.cloudera.com/documentation/enterprise/latest/topics/admin_hive_on_s3_tuning.html

Prakashcit · ‎02-16-2021

@bigdataNico Kindly run hdfs dfs -rmr /user/hive/ .yarn/package/LLAP/* and restart the LLAP.

Prakashcit · ‎01-25-2021

You have to delete the files from HDFS use this command::hdfs dfs -rm -rf /user/hive/.yarn/package/LLAP After the deleting run the below commands:: hdfs dfs -mkdir -p / user/hive/.yarn/package/LLAP hdfs dfs -chown hive: hadoop / user/hive/.yarn/package/LLAP hdfs dfs -chmod 755 /user/hive/.yarn/package/LLAP Restart the Hiveserver2 after executing these commands.

Prakashcit · ‎01-25-2021

@kevinmat0510 You have to delete the files from HDFS use this command::hdfs dfs -rm -rf /user/hive/.yarn/package/LLAP

Prakashcit · ‎01-25-2021

@kevinmat0510 Hive 3 architecture is changed to support ACID v2 and Hive 3 buckets generation is automatic which splits data implicitly. https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/using-hiveql/content/hive_3_internals.html You can't disable the generation of buckets, it's a complete architecture change of Hive 3 and refer the Hive 3 ACID support details in the document. Thanks, Prakash

Prakashcit · ‎01-25-2021

@linzhongwei Relevant bug::https://issues.apache.org/jira/browse/HIVE-23111

Prakashcit · ‎01-24-2021

@AnkitP It's not able to make connection to the namenode hostname/IPaddress to hostname:8020, can you check whether you're able to perform telnet namenode-hostname 8020, in case if you're getting connection error then you need to open this port for connection. Thanks, Prakash

Prakashcit · ‎01-24-2021

@shrimaha It looks like you're using wrong jdbc connection string to connect to Hive, could you please check your connection string details once. https://community.cloudera.com/t5/Community-Articles/Working-with-Beeline/ta-p/247606 https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_data-access/content/beeline-vs-hive-cli.html Thanks, Prakash

Prakashcit · ‎01-24-2021

@bigdataNico It's a known issue with the cloudbreak deployment. Kindly refer the "Known issues: Data lake" section in the below document. https://docs.cloudera.com/HDPDocuments/Cloudbreak/Cloudbreak-2.9.0/release-notes/content/cb_known-issues.html Delete the "/user/hive/.yarn/package/LLAP" file, and then create a new directory in this location with the relevant permissions for the hive user. Start HiveServer2. Thanks, Prakash

Prakashcit · ‎12-16-2020

@anrathen Can you adjust hive proxy settings and test once hadoop.proxyuser.hive.hosts=* hadoop.proxyuser.hive.groups=*

Prakashcit · ‎12-05-2020

@henimaher It could be due to bug https://issues.apache.org/jira/browse/HIVE-21866 perform yarn app -destroy llap0 and start LLAP service

Prakashcit · ‎08-24-2020

https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/ambari-release-notes/content/known_issues.html

Prakashcit · ‎08-17-2020

@AdityaShaw Yes with the help of Yarn ACL's you can control the users submitting applications to specific yarn queue. Kindly follow these documents to enable yarn acl. https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_yarn-resource-management/content/controlling_access_to_queues_with_acls.html https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html

Prakashcit · ‎08-17-2020

@manicool7 Can you elaborate more on the issue here and are you ingesting data from any third party like (Tableau, qlik sense) ? Attach the error details as well

Prakashcit · ‎08-17-2020

@Sham Can you collect yarn logs -applicationId application_1594037379069_89091 > app89091.txt and attach it here.

Prakashcit · ‎08-14-2020

@Nagamalleswara Thanks for your response, you need to open a support case with cloudera to get the patch. Mention about the bug and the issue, your issue will be analysed and appropriate patch will be released.

Prakashcit · ‎08-12-2020

@Nagamalleswara It's due to these bugs https://issues.apache.org/jira/browse/HIVE-22416 and https://issues.apache.org/jira/browse/HIVE-9120 You need to have a fix for these bugs to overcome this issue. Thanks, Prakash

Prakashcit · ‎07-09-2020

@Heri Suggestion is to use triggers and create new table with the timestamp field and run the sqoop incremental job on the newly created table. https://stackoverflow.com/questions/34806245/incremental-data-load-using-sqoop-without-primary-key-or-timestamp

Prakashcit · ‎07-09-2020

@Mithun07 If the sqoop import is failed with large data set then it will not write anything in hdfs only the successful imports will right the data in hdfs.

Prakashcit · ‎04-23-2020

@TR7_BRYLE Did we restart the agents after making the changes? can you attach the error stack trace.

Prakashcit · ‎04-20-2020

@TR7_BRYLE This issue occurs because the Java is restricting the TLSv1 from (1.8.0-171) used by the Ambari Agents. By default, ambari-agent connects to TLSv1, unless specified by force_https_protocol=PROTOCOL_TLSv1_2 in ambari-agent.ini. Hence, the Ambari- Agent is not able to connect and communicate to Ambari server. To resolve this issue, add the following property in ambari-agent.ini [/etc/ambari-agent/conf/ambari-agent.ini] file under [security] and restart ambari-agent. force_https_protocol = PROTOCOL_TLSv1_2 https://community.cloudera.com/t5/Support-Questions/ambari-agents-cannot-reach-ambari-server-after-changing/td-p/193251 Restart the ambari agents Thanks, Prakash

Prakashcit · ‎04-20-2020

@banshidhar_saho Can we enable both Kerberos and LDAP authentication for HiveServer2 at the same time? --> No that's not possible, you can either use kerberos or LDAP, we can't use both at the same time. Secondly, can you share a sample workflow.xml and job.properties for an Oozie job that has a HiveServer2 action where LDAP authentication is enabled for Hiveserver2. Kindly refer this documentation to specify the ldap credentials in hive action. https://oozie.apache.org/docs/5.1.0/DG_Hive2ActionExtension.html Thanks, Prakash

Prakashcit · ‎02-21-2020

@Folks You can follow this approach for any other kerberos keytab related issues with LLAP.

Prakashcit · ‎02-20-2020

LLAP Service sometimes may fail to startup after HDP upgrade with the following error: Error:: Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: hive/prakash.cloudera.com@CLOUDERA.COM from keytab /grid/0/yarn/local/usercache/hive/appcache/application_**********/container_***********_01_000001/keytabs/llap0.keytab javax.security.auth.login.LoginException: Pre-authentication information was invalid (24) To resolve this issue, run the following command and start LLAP: yarn app -destroy llap0 If the issue persists, then do the following Check the hive.service.keytab file is available in all the hosts, run kinit and test the connectivity. If the file is missing, recreate the keytabs on the missing hosts either through Ambari or manually (see this article for manual keytab creation). Verify the file in HDFS and copy it to the local. Perform kinit with the copied keytab. hdfs:///user/hive/.yarn/keytabs/hive/hive.service.keytab hdfs dfs -copyToLocal /user/hive/.yarn/keytabs/hive/hive.service.keytab /tmp/hive.keytab kinit -kt /tmp/hive.keytab hive/prakash.cloudera.com@CLOUDERA.COM kinit: Preauthentication failed while getting initial credentials If you get the error, to move the keytab file from hdfs location. hdfs dfs -cp /user/hive/.yarn/keytabs/hive/hive.service.keytab /tmp/ hdfs dfs -rm /user/hive/.yarn/keytabs/hive/hive.service.keytab Start LLAP component.

Prakashcit · ‎02-14-2020

@hesham_eldib Could you please disable these properties hive.metastore.metrics.enabled and hive.server2.metrics.enabled from Advanced hiveserver2-site. Start Hiveserver2

Prakashcit · ‎02-13-2020

@Asoka I think you're passing the hidden file in the --password-file option sqoop export --connect jdbc:mysql://hostname/test --connection-manager org.apache.sqoop.manager.MySQLManager --export-dir /tmp/version --table HiveVersion --username hive --password-file file:///tmp/mysql-pass.txt This command worked for me

Prakashcit · ‎01-20-2020

@ana24 You can make use Hive CAST function to find the timestamp difference.Please refer this article and frame your queries according to your case. http://sqlandhadoop.com/how-to-subtract-timestamp-date-time-in-hive/

Prakashcit · ‎01-20-2020

@saivenkatg55 It looks like query is not retrieving any results from the specific table.Could you please attach the "show create table pcr_project" output. Meanwhile verify the data and it's ownership in the hdfs path.

Online	Offline
Last Visited	‎08-27-2021 09:21 AM

Date Registered	‎04-30-2019 07:39 AM
Last Visited	‎08-27-2021 09:21 AM
Total Messages Posted	53
Total Kudos Received	5

Re: Hortonworks (Hive - ldap integration failure)

Re: How to use Hive testbench to perform benchmark...

Re: Tez mapper Slow Read from S3

Re: Tez mapper Slow Read from S3

Re: Start hive on tez with llap failed

Re: Start Hive on Tez with llap FAILED

Re: Disable creation of Bucket files in Hive 3.1.0...

Re: Disable creation of Bucket files in Hive 3.1.0...

Re: hive metastore memory leak

Re: Not able start ambari services

Re: org.apache.thrift.transport.TTransportExceptio...

Re: Start Hive on Tez with llap FAILED

Re: SSIS Hive Task using WebHCAT

Re: Start Hive & HiveServer2

Re: Hiveserver interactive(LLAP) startup failure j...

Re: How to restrict yarn queue access when Hive Im...

Re: Hive text data ingestion issue

Re: Hello Team, getting Vertex errror while runnin...

Re: beeline ( HiveServer2 ) not writing console lo...

Re: beeline ( HiveServer2 ) not writing console lo...

Re: Sqoop: Incremental loads in Hive without a tim...

Re: sqoop - job failure during data import

Re: Ambari-agent unable to communicate to ambari s...

Re: Ambari-agent unable to communicate to ambari s...

Re: HiveServer2 authentication - LDAP and Kerberos

Re: Hiveserver interactive(LLAP) startup failure j...

Hiveserver interactive(LLAP) startup failure javax...

Re: Failed to start HiveServer2 using Ambari 2.7 a...

Re: What is the format for a sqoop-export password...

Re: How to substract minutes

Re: Unable to select the table in hive