About pra_big

pra_big · ‎10-10-2019

I need clarification about Metastore URI configuration of Hive data server. The problem is that in the official cloudera documentation you can see that it is a requirement to block external applications accessing Hive Metastore in order to secure cluster with Sentry: https://www.cloudera.com/documentation/enterprise/latest/topics/sg_sentry_service_config.html#concept_z5b_42s_p4__section_lvc_4g4_rp Block the external applications from accessing the Hive metastore: In the Cloudera Manager Admin Console, select the Hive service. On the Hive service page, click the Configuration tab. In the search well on the right half of the Configuration page, search for Hive Metastore Access Control and Proxy User Groups Override to locate the hadoop.proxyuser.hive.groups parameter and click the plus sign. Enter hive into the text box and click the plus sign again. Enter hue into the text box. Enter sentry into the text box. Click Save Changes. ODI is also an external application so its direct access requirement to Metastore contradicts with this requirement. Why are we setting Metastore URI in Hive data server? What is it used for? If it is a requirement, how can we explain this contradiction? Another problem is that we are configuring Metastore HA by default in our BDA. However how can we configure ODI Metadata Uri to support metadata HA configuration? Is there any way to write a Uri supporting connecting more than one Metastore? If not, how can we explain this situation?

pra_big · ‎06-13-2019

As a company policy, we are required to shutdown unused services. Oozie is one of such services running on BDA. We are aware of the stuation that we should not remove oozie service because it may broke mammoth functionality. However we want to keep it stopped. The problem is that with oozie stopped Cloudera Navigator Metadata service periodically throws ERROR "[CDHExecutor-0-CDHUrlClassLoader@53df0527]: Error extracting Oozie workflows." I will be uploading Cloudera Navigator Metadata service log where you can see a lot of these errors. Is there a way to configure Cloudera Navigator Metadata service so it does not do extraction from oozie? I looked at the official documentation but could not see a way to do this

pra_big · ‎05-29-2019

This flaw affects Tomcat on oracle Enterprise Linux only when a specific context is configured with readonly=false. The default configuration has a readonly context, so it is not affected. Ensure that readonly is set to true (by default, it is true even not mentioned in web.xml) for the DefaultServlet, WebDAV servlet or application context. Example – Depending upon what version you are, there are many web.xml for each service. [root@labUSbda07 ~]# vi /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/etc/oozie/tomcat-conf.http/conf/web.xml [root@labUSbda07 ~]# vi /opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/etc/oozie/tomcat-conf.http/conf/web.xml more /opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/etc/oozie/tomcat-conf.http/conf/web.xml Servlet content on my Lab server default org.apache.catalina.servlets.DefaultServlet debug 0 listings false 1 Readonly parameter’s default value is picked here

pra_big · ‎05-29-2019

Customer found the tomcat security vulnerability CVE-2017-12615 on the third node of CDH 5.15 cluster . How to fix this CVE-2017-12615 issue?

pra_big · ‎04-24-2019

Please tell me after the Refresh Dynamic Resource Pools there is an error "Failed to perform refresh of roles in YARN, Impala, and (in CDH 5.8 and later) Hive services." Refresh the ResourceManager's pool ACLs, states, scheduler specific properties, and hosts allow/exclude information. Process yarn-RESOURCEMANAGER-refresh (id=12038) on host bda3node04.vtb24.ru (id=6) exited with 255 and expected 0 Refresh the ResourceManager's pool ACLs, states, scheduler specific properties, and hosts allow/exclude information. Process yarn-RESOURCEMANAGER-refresh (id=12039) on host bda3node03.vtb24.ru (id=5) exited with 255 and expected 0 How to deal with this?

pra_big · ‎04-24-2019

Please tell me after the Refresh Dynamic Resource Pools there is an error "Failed to perform refresh of roles in YARN, Impala, and (in CDH 5.8 and later) Hive services." Refresh the ResourceManager's pool ACLs, states, scheduler specific properties, and hosts allow/exclude information. Process yarn-RESOURCEMANAGER-refresh (id=12038) on host bda3node04.vtb24.ru (id=6) exited with 255 and expected 0 Refresh the ResourceManager's pool ACLs, states, scheduler specific properties, and hosts allow/exclude information. Process yarn-RESOURCEMANAGER-refresh (id=12039) on host bda3node03.vtb24.ru (id=5) exited with 255 and expected 0 How to deal with this?

pra_big · ‎04-09-2019

Thanks Gentleman, Let me re-install. I hope it works

pra_big · ‎04-08-2019

We're using cloudera with anaconda parcel on bda production cluster . I tried to execute pyspark code that imports pyarrow package , then i faced with error below . Traceback (most recent call last): File "/opt/cloudera/parcels/SPARK2-2.3.0.cloudera2-1.cdh5.13.3.p0.316101/lib/spark2/python/lib/pyspark.zip/pyspark/sql/utils.py", line 140, in require_minimum_pyarrow_version File "/opt/cloudera/parcels/Anaconda-3.6.5_2/lib/python3.6/site-packages/pyarrow/__init__.py", line 47, in <module> from pyarrow.lib import cpu_count, set_cpu_count ImportError: libboost_system.so.1.66.0: cannot open shared object file: No such file or directory During handling of the above exception, another exception occurred: Traceback (most recent call last): File "lbf_livArea_scr_2.py", line 51, in <module> @pandas_udf(schema, PandasUDFType.GROUPED_MAP) File "/opt/cloudera/parcels/SPARK2-2.3.0.cloudera2-1.cdh5.13.3.p0.316101/lib/spark2/python/lib/pyspark.zip/pyspark/sql/udf.py", line 45, in _create_udf File "/opt/cloudera/parcels/SPARK2-2.3.0.cloudera2-1.cdh5.13.3.p0.316101/lib/spark2/python/lib/pyspark.zip/pyspark/sql/utils.py", line 143, in require_minimum_pyarrow_version ImportError: PyArrow >= 0.8.0 must be installed; however, it was not found. Also output of conda list is below . [ihsany@gbbdap02 ~]$ dzdo /opt/cloudera/parcels/Anaconda/bin/conda list |grep arrow arrow-cpp 0.9.0 py36h1ae9da6_7 <unknown> pyarrow 0.9.0 py36_1 <unknown> [ihsany@gbbdap02 ~]$ dzdo /opt/cloudera/parcels/Anaconda/bin/conda list |grep boost libboost 1.65.1 habcd387_4 <unknown>

pra_big · ‎03-07-2019

Is it possible, an YARN DRP has priority against other pools? ie. If i already have 2 DRP created DRP1 = 500Gb DRP2 = 200Gb and i want to add new one DRP3 = 700Gb that have priority against other pools and use all available resources are directed to DRP3. Is this possible?

pra_big · ‎03-07-2019

Thank you very much Harsh

pra_big · ‎02-21-2019

can we deploy the HttpFS role on more than one node? is it best practice? we had a problem with one of the namenodes (on node02) in our cluster and it was down for almost 20 hours. once node02 was up, the active namenode ,on node01, started to alert that its not syncronized after restart of the namenode on node02 the alert stopped now active namenode is on node01 standby namenode is on node02 we can see that new logs are created on opt/hadoop/dfs/nn/current on the actine namenode on node01 but no new files . on the standby namenode no node02 - is it OK ??

pra_big · ‎02-13-2019

Below issue can happen if certifcate is expired? I see in some logs that certificates are expired. Please send documentation for certification renewal. 2019-02-13 23:31:58,038 WARN 1168879507@agentServer-54778:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_expired 2019-02-13 23:31:58,703 WARN 1168879507@agentServer-54778:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_expired 2019-02-13 23:32:01,494 INFO 1645307921@scm-web-99151:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 192.168.10.51 2019-02-13 23:32:03,490 WARN 1168879507@agentServer-54778:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_expired

pra_big · ‎01-28-2019

We see the same issue on out node05 28/Jan/2019 11:00:36 +0000] 62064 Metadata-Plugin navigator_plugin_pipeline INFO Stopping Navigator Plugin Pipeline '' for yarn-RESOURCEMANAGER (log dir: None) [28/Jan/2019 11:00:36 +0000] 62064 Metadata-Plugin navigator_plugin INFO Refreshing Metadata Plugin for hive-WEBHCAT with pipelines [] [28/Jan/2019 11:00:36 +0000] 62064 Metadata-Plugin navigator_plugin_pipeline INFO Stopping Navigator Plugin Pipeline '' for hive-WEBHCAT (log dir: None) [28/Jan/2019 11:00:36 +0000] 62064 Audit-Plugin throttling_logger INFO Unable to communicate with audit server (host: nlbv015c060605.em2.oraclecloud.com, port: 7186): [Errno 111] Connection refused [28/Jan/2019 11:00:36 +0000] 62064 Audit-Plugin throttling_logger INFO Unable to send data to nav server. Will try again. [28/Jan/2019 11:00:36 +0000] 62064 Metadata-Plugin throttling_logger ERROR Unable to send message to metadata server. Traceback (most recent call last): File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.13.1-py2.6.egg/cmf/audit/navigator_sender.py", line 162, in _send_json_data elif isinstance(getattr(e, 'reason'), socket.timeout): AttributeError: 'HTTPError' object has no attribute 'reason' [28/Jan/2019 11:00:55 +0000] 62064 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('GenericMonitor HUE-KT_RENEWER for None',) [28/Jan/2019 11:00:55 +0000] 62064 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('GenericMonitor HIVE-HIVESERVER2 for None',) [28/Jan/2019 11:00:55 +0000] 62064 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('GenericMonitor HIVE-HIVEMETASTORE for None',) [28/Jan/2019 11:00:55 +0000] 62064 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('GenericMonitor HIVE-WEBHCAT for None',) [28/Jan/2019 11:00:57 +0000] 62064 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('GenericMonitor HUE-HUE_SERVER for None',) -bash-4.1# dcli -C cat /etc/cloudera-scm-agent/config.ini | grep server_host 10.196.64.68: server_host=nlbv015c060605.em2.oraclecloud.com 10.196.64.44: server_host=nlbv015c060605.em2.oraclecloud.com 10.196.64.86: server_host=nlbv015c060605.em2.oraclecloud.com 10.196.64.104: server_host=nlbv015c060605.em2.oraclecloud.com 10.196.64.8: server_host=nlbv015c060605.em2.oraclecloud.com -bash-4.1# hostname -f nlbv012c060605.em2.oraclecloud.com -bash-4.1# -bash-4.1# dcli -C cat /etc/cloudera-scm-agent/config.ini | grep server_port 10.196.64.68: server_port=7182 10.196.64.44: server_port=7182 10.196.64.86: server_port=7182 10.196.64.104: server_port=7182 10.196.64.8: server_port=7182 -bash-4.1#

pra_big · ‎01-22-2019

Remove the dead host/decommissioned host from mammoth -c output or CM. We have already deleted the host. As we are about to start the upgrade process from 5.14.2 to 6.0. So as a prerequisites, When running ./mammoth -c it is giving information about the hosts which is not part of cluster. We are also thinking to remove it from scm database table hosts On mysql, under scm database, also I am able to see : mysql> mysql> select * from HOSTS; +---------+-------------------------+--------------------------------------+-----------------------------+---------------+----------+--------+- | HOST_ID | OPTIMISTIC_LOCK_VERSION | HOST_IDENTIFIER | NAME | IP_ADDRESS | RACK_ID | STATUS | +---------+-------------------------+--------------------------------------+-----------------------------+---------------+----------+--------+- | 1 | 248 | 260772a1-a89a-42b8-af4c-0406ac0c21bd | bdk1n07.bnet.luxds.net | 192.168.11.16 | /default | NA | | 2 | 251 | 19103582-a94d-4961-aeb8-5a2023480fa5 | bdk1n09.bnet.luxds.net | 192.168.11.18 | /default | NA | | 3 | 254 | e57f3aa9-ab4f-4b3c-925d-2be272237928 | bdk1n08.bnet.luxds.net | 192.168.11.17 | /default | NA | | 4 | 89 | 0317c86d-b693-4280-ba25-0bbcc46e567c | xl11lsrv0428.bnet.luxds.net | 10.178.65.98 | /default | NA | +---------+-------------------------+--------------------------------------+-----------------------------+---------------+----------+--------+- One with hostId "0317c86d-b693-4280-ba25-0bbcc46e567c"(which was edge node before) is removed from cloudera, so is there anyway to clean this node from CM, because on the screen of cloudera - hosts I am just able to see 3 nodes. Is that server xl11lsrv0428.bnet.luxds.net | 10.178.65.98 still running separately? it is running seperately and even re-imaged. Is CM agent still running /stopped on the server xl11lsrv0428.bnet.luxds.net | 10.178.65.98? No CM agent is working on it currently Is it showing in CM portal? Ans. on the CM, no entry as xl11lsrv0428.

pra_big · ‎01-16-2019

Where the HDFS balancer logs are stored.

pra_big · ‎01-09-2019

Same issue in my environment. Error: flush failed for required journal (JournalAndStream(mgr=QJM to [10.196.64.44:8485, 10.196.64.68:8485, 10.196.64.86:8485], stream=QuorumOutputStream starting at txid 434560443)) We are running cloudera on VM machine (cloud)

pra_big · ‎01-04-2019

we config the core-site.xml with the credentials (Account + key) and restart What do you mean by restart, Is it HDFS service restart across cluster

pra_big · ‎01-04-2019

i have used the below command to copy 36TB to blob using snaoshot. HADOOP_CLIENT_OPTS="-Xmx40G" hadoop distcp -update -delete $SNAPSHOT_PATH wasbs://buclusterbackup@blobplatformdataxe265ecb.blob.core.windows.net/sep_backup/application_data getting Azure exception errors and Java IO error. i re ran with -skipcrccheck still the same error.

pra_big · ‎12-29-2018

Where i should add ha.zookeeper.session-timeout.ms and what is the correct setting? Should I add the property in hdfs-site.xml

pra_big · ‎12-13-2018

Same problem happened with me [root@ukfhbda1-db01 ~]# klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: oracle@GDC.LOCAL Valid starting Expires Service principal 12/13/18 12:39:27 12/14/18 12:39:27 krbtgt/GDC.LOCAL@GDC.LOCAL renew until 12/20/18 12:39:27 [root@ukfhbda1-db01 ~]# [root@ukfhbda1-db01 ~]# beeline Beeline version 1.1.0-cdh5.14.2 by Apache Hive beeline> !connect 'jdbc:hive2://ukfhbda1-db04.gdc.local:10000/default;principal=hive/_HOST@GDC.LOCAL' scan complete in 2ms Connecting to jdbc:hive2://ukfhbda1-db04.gdc.local:10000/default;principal=hive/_HOST@GDC.LOCAL Connected to: Apache Hive (version 1.1.0-cdh5.14.2) Driver: Hive JDBC (version 1.1.0-cdh5.14.2) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://ukfhbda1-db04.gdc.local:10000> oracle . . . . . . . . . . . . . . . . . . . . . . .> Experian123 . . . . . . . . . . . . . . . . . . . . . . .> create role admin_role; Error: Error while compiling statement: FAILED: ParseException line 1:0 cannot recognize input near 'oracle' 'Experian123' 'create' (state=42000,code=40000) 0: jdbc:hive2://ukfhbda1-db04.gdc.local:10000> 0: jdbc:hive2://ukfhbda1-db04.gdc.local:10000> 0: jdbc:hive2://ukfhbda1-db04.gdc.local:10000> Closing: 0: jdbc:hive2://ukfhbda1-db04.gdc.local:10000/default;principal=hive/_HOST@GDC.LOCAL [root@ukfhbda1-db01 ~]# su - oracle [oracle@ukfhbda1-db01 ~]$ beeline Beeline version 1.1.0-cdh5.14.2 by Apache Hive beeline> !connect 'jdbc:hive2://ukfhbda1-db04.gdc.local:10000/default;principal=hive/_HOST@GDC.LOCAL' scan complete in 2ms Connecting to jdbc:hive2://ukfhbda1-db04.gdc.local:10000/default;principal=hive/_HOST@GDC.LOCAL Connected to: Apache Hive (version 1.1.0-cdh5.14.2) Driver: Hive JDBC (version 1.1.0-cdh5.14.2) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://ukfhbda1-db04.gdc.local:10000> create role admin_role; INFO : Compiling command(queryId=hive_20181213140707_3023a4fb-b861-469a-b271-f69482c8dd34): create role admin_role INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20181213140707_3023a4fb-b861-469a-b271-f69482c8dd34); Time taken: 0.115 seconds INFO : Executing command(queryId=hive_20181213140707_3023a4fb-b861-469a-b271-f69482c8dd34): create role admin_role INFO : Starting task [Stage-0:DDL] in serial mode ERROR : Error processing Sentry command: java.net.ConnectException: Connection refused (Connection refused). ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. SentryUserException: java.net.ConnectException: Connection refused (Connection refused) INFO : Completed executing command(queryId=hive_20181213140707_3023a4fb-b861-469a-b271-f69482c8dd34); Time taken: 15.015 seconds Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. SentryUserException: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=1) 0: jdbc:hive2://ukfhbda1-db04.gdc.local:10000> grant role admin_role to group hive; INFO : Compiling command(queryId=hive_20181213141212_24f592d7-adcf-4a91-8d15-aa46a7220138): grant role admin_role to group hive INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20181213141212_24f592d7-adcf-4a91-8d15-aa46a7220138); Time taken: 0.172 seconds INFO : Executing command(queryId=hive_20181213141212_24f592d7-adcf-4a91-8d15-aa46a7220138): grant role admin_role to group hive INFO : Starting task [Stage-0:DDL] in serial mode ERROR : Error processing Sentry command: java.net.ConnectException: Connection refused (Connection refused). ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. SentryUserException: java.net.ConnectException: Connection refused (Connection refused) INFO : Completed executing command(queryId=hive_20181213141212_24f592d7-adcf-4a91-8d15-aa46a7220138); Time taken: 15.014 seconds Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. SentryUserException: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=1) 0: jdbc:hive2://ukfhbda1-db04.gdc.local:10000> grant all on server server1 to role admin_role; INFO : Compiling command(queryId=hive_20181213141212_08a3e86b-4c85-4ed5-ae99-9c22ca937130): grant all on server server1 to role admin_role INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20181213141212_08a3e86b-4c85-4ed5-ae99-9c22ca937130); Time taken: 0.079 seconds INFO : Executing command(queryId=hive_20181213141212_08a3e86b-4c85-4ed5-ae99-9c22ca937130): grant all on server server1 to role admin_role INFO : Starting task [Stage-0:DDL] in serial mode ERROR : Error processing Sentry command: java.net.ConnectException: Connection refused (Connection refused). ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. SentryUserException: java.net.ConnectException: Connection refused (Connection refused) INFO : Completed executing command(queryId=hive_20181213141212_08a3e86b-4c85-4ed5-ae99-9c22ca937130); Time taken: 15.014 seconds Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. SentryUserException: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=1)

pra_big · ‎12-07-2018

We are getting below alerts. Ki ndly guide on how to troubleshoot T h e h e a l t h t e s t r e s u l t f o r C A T A L O G S E R V E R _ H O S T _ H E A L T H h a s b e c o m e b a d. T h e h e a l t h o f t h i s r o l e ' s h o s t i s b a d T h e h e a l t h t e s t r e s u l t f o r S T A T E S T O R E _ H O S T _ H E A L T H h a s b e c o m e b a d : T h e h e a l t h o f t h i s r o l e ' s h o s t i s b a d . T h e h e a l t h t e s t r e s u l t f o r I M P A L A D _ H O S T _ H E A L T H h a s b e c o m e b a d : T h e h e a l t h o f t h i s r o l e ' s h o s t i s b a d T h e h e a l t h t e s t r e s u l t f o r I M P A L A _ S T A T E S T O R E _ H E A L T H h a s b e c o m e b ad T h e h e a l t h t e s t r e s u l t f o r I M P A L A _ C A T A L O G S E R V E R _ H E A L T H h a s b e c o m e b a d

Online	Offline
Last Visited	‎10-17-2019 12:26 PM

Date Registered	‎12-07-2018 01:28 AM
Last Visited	‎10-17-2019 12:26 PM
Total Messages Posted	21

Need Clarification for Hive Data Server Configurat...

ERROR in cloudera navugator metadata service "[CDH...

Re: tomcat security vulnerability CVE-2017-12615

tomcat security vulnerability CVE-2017-12615

Re: Dynamic resource pools edit and refresh via AP...

Re: ResourceManager dynamic pool update fails due ...

Re: pyspark / pyarrow problem

pyspark / pyarrow problem

Is it possible, an YARN DRP has priority against o...

Re: can we deploy the HttpFS role on more than one...

can we deploy the HttpFS role on more than one nod...

Re: issue with cloudera management services after ...

Re: Cloudera Manager distribution error

Re: can't delete deleted node from the cluster CM

Re: Where the HDFS balancer logs are stored

Re: Error: flush failed for required journal

Re: Copy data from cloudera hdfs to azure blob sto...

Re: Handling Distcp of large files between 2 clust...

Re: Failover Controllers Health Bad leads to compl...

Re: Exercise 3: Could not open client transport wi...

Health test result for IMPALAD_HOST_HEALTH has bec...