Member since
09-14-2017
120
Posts
11
Kudos Received
5
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 5036 | 06-17-2021 06:55 AM | |
| 2645 | 01-13-2021 01:56 PM |
12-18-2021
10:14 AM
Hello, After upgrading to CM 7.4.4/CR 7.1.7 the Activity Monitor never starts. It is always down and cannot be started after multiple attempts. It gives error messages below any thoughts? Command (Start this Activity Monitor (17683)) has failed Supervisor returned FATAL. Please check the role log file, stderr, or stdout. The health test result for ACTIVITY_MONITOR_SCM_HEALTH has become bad: This role's process failed to start. STDOUT messages: 13:06:12.204 [main] ERROR org.hibernate.engine.jdbc.batch.internal.BatchingBatch - HHH000315: Exception executing batch [java.sql.BatchUpdateException: Data truncation: Out of range value for column 'METRIC_ID' at row 1], SQL: insert into CMON_METRIC_INFO (NAME, METRIC_ID) values (?, ?)
13:06:12.209 [main] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - Data truncation: Out of range value for column 'METRIC_ID' at row 1
Sat Dec 18 13:06:17 EST 2021
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64
CONF_DIR=/var/run/cloudera-scm-agent/process/7594-cloudera-mgmt-ACTIVITYMONITOR 13:06:30.645 [main] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - Data truncation: Out of range value for column 'METRIC_ID' at row 1 at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:101)
... 2 more
Caused by: java.sql.BatchUpdateException: Data truncation: Out of range value for column 'METRIC_ID' at row 1
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.Util.getInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createBatchUpdateException(SQLError.java:1163)
at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1823)
at com.mysql.jdbc.PreparedStatement.executeBatchInternal(PreparedStatement.java:1307)
at com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:970)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeBatch(NewProxyPreparedStatement.java:2544)
at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:121)
... 22 more
Caused by: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Out of range value for column 'METRIC_ID' at row 1
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3976)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2495)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1903)
at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2124)
at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1801)
... 26 more
... View more
Labels:
- Labels:
-
Cloudera Manager
12-14-2021
12:29 PM
1 Kudo
The script provided in https://github.com/cloudera/cloudera-scripts-for-log4j to delete the JndiLookup.class from log4j jar may have an issue if command zip or unzip is not installed on the cluster nodes. The script should initially check for yum list zip and unzip modules if available and abort if not found. Else it will give a finished message even though it had errors and didnt run successfully like below: Completed removing JNDI from /opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/share/doc/search-1.0.0.7.1.6.0/examples/test-documents/testJPEG_EXIF.jpg.tar.gz Backing up to '/tmp/tmp.FsVTS5Rg9y//opt/cloudera/cm/lib/solr-upgrade-1.0.0.7.1.7.0-547.tar.gz.backup' Patching '/opt/cloudera/cm/lib/solr-upgrade-1.0.0.7.1.7.0-547.tar.gz' Running on '/tmp/tmp.Yxjh6FQYgS' Backing up files to '/tmp/tmp.TsL7gbbmHR' Completed removing JNDI from jar files ./cm_cdp_cdh_log4j_jndi_removal.sh: line 114: unzip: command not found grep: /tmp/unzip_target/**/*.jar: No such file or directory Completed removing JNDI from nar files Recompressing Completed removing JNDI from /opt/cloudera/cm/lib/solr-upgrade-1.0.0.7.1.7.0-547.tar.gz INFO : Finished
... View more
07-28-2021
07:43 AM
After openjdk 1.8.0.292 install we had to fix the mysql scm database ssl connection issue using the property useSSL=false like com.cloudera.cmf.orm.hibernate.connection.url=jdbc:mysql://<mysql-host>/scm?useUnicode=true&characterEncoding=UTF-8&useSSL=false Now same ssl connection issue is happening for nav, navms, amon, rman databases and these components wont start. Can we set the useSSL for these databases also in /etc/cloudera-scm-server/db.properties file like below: com.cloudera.cmf.orm.hibernate.connection.url=jdbc:mysql://localhost/nav?useUnicode=true&characterEncoding=UTF-8&useSSL=false com.cloudera.cmf.orm.hibernate.connection.url=jdbc:mysql://localhost/navms?useUnicode=true&characterEncoding=UTF-8&useSSL=false com.cloudera.cmf.orm.hibernate.connection.url=jdbc:mysql://localhost/rman?useUnicode=true&characterEncoding=UTF-8&useSSL=false
... View more
06-17-2021
06:55 AM
1 Kudo
Darren, This got resolved with the help of Cloudera Support engineer. There were couple of issues. First the IDP and SP entity_id should be different values. Next there is a SAML property that needs to be set by the IDP which is <saml:AudienceRestriction><saml:Audience /></saml:AudienceRestriction></saml:Conditions> After setting the Audience property to the entity_id of the SP the error went away and HUE SAML is working again. See below error before setting Audience property: response DEBUG conditions: <?xml version='1.0' encoding='UTF-8'?> <saml:Conditions xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" NotBefore="2021-06-02T16:02:45.573Z" NotOnOrAfter="2021-06-02T17:02:45.573Z"><saml:AudienceRestriction><saml:Audience /></saml:AudienceRestriction></saml:Conditions> [02/Jun/2021 09:02:45 -0700] client_base ERROR XML parse error: 'NoneType' object has no attribute 'strip' [02/Jun/2021 09:02:45 -0700] middleware INFO Processing exception: 'NoneType' object has no attribute 'strip': Traceback (most recent call last): File "/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/core/handlers/base.py", line 185, in _get_response This looks like a breaking change from CDH 5.15 HUE saml to CDP 7.1.6 as we never set the Audience value in CDH 5.x in IDP before.
... View more
05-27-2021
09:39 AM
Some more progress: It appears CDP 7.1.6 we need to create the unencrypted dummy key file as below. To create an unencrypted private key file from an encrypted key we have to run: openssl rsa -in ssl_certificate.key -out ssl_certificate-nocrypt.key The output file (ssl_certificate-nocrypt.key) is an unencrypted PEM-formatted key that is used for the parameter key_file=/opt/cloudera/security/saml/ssl_certificate-nocrypt.key Now this error is gone: Could not deserialize key data. But we are getting different error below: AttributeError at /saml2/acs/ 'NoneType' object has no attribute 'strip' Request Method: POST Request URL: http://xxxx.com:8889/saml2/acs/ Django Version: 1.11.29 Exception Type: AttributeError Exception Value: 'NoneType' object has no attribute 'strip' Exception Location: /opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hue/build/env/lib/python2.7/site-packages/pysaml2-4.9.0-py2.7.egg/saml2/response.py in for_me, line 212 Python Executable: /opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hue/build/env/bin/python2.7 Python Version: 2.7.5 Python Path: ['/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hue/desktop/libs/libsaml/attribute-maps', ------------------------------------------------- Below is the python code in line 212 which errors out: 202 def for_me(conditions, myself): 203 """ Am I among the intended audiences """ 204 205 if not conditions.audience_restriction: # No audience restriction 206 return True 207 208 for restriction in conditions.audience_restriction: 209 if not restriction.audience: 210 continue 211 for audience in restriction.audience: 212 if audience.text.strip() == myself: 213 return True 214 else: 215 # print("Not for me: %s != %s" % (audience.text.strip(), 216 # myself)) 217 pass 218 219 return False
... View more
05-26-2021
04:26 PM
Hello, After we upgraded from CDH 5.15 to CDP 7.1.6 runtime. The HUE SAML login got broken. It gives an error below. Any ideas? ValueError at /saml2/login/ Could not deserialize key data. Request Method: GET Request URL: http://xxxxx.com:8889/saml2/login/?next=/ Django Version: 1.11.29 Exception Type: ValueError Exception Value: Could not deserialize key data. Exception Location: /opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hue/build/env/lib/python2.7/site-packages/cryptography-2.9-py2.7-linux-x86_64.egg/cryptography/hazmat/backends/openssl/backend.py in _handle_key_loading_error, line 1382 Python Executable: /opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hue/build/env/bin/python2.7 Python Version: 2.7.5 Python Path: ['/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hue/desktop/libs/libsaml/attribute-maps',
... View more
Labels:
- Labels:
-
Cloudera Data Explorer
-
Security
01-13-2021
01:56 PM
1 Kudo
This was resolved by manually upgrading the agents in the other nodes which were still at CM5.16 by running the commands below: First update the /etc/yum.repos.d/cloudera-manager.repo on the nodes with the proper repo for CM7.2.4. After that run command on each agent to be upgraded: $yum upgrade cloudera-manager-daemons cloudera-manager-agent After that restart the cloudera manager agents on all nodes: $ systemctl restart cloudera-scm-agent Next go to Cloudera Manager GUI and restart the Cloudera Management Service. After that restart the CDH Cluster all services. This should resolve the issues.
... View more
01-12-2021
03:25 PM
Hello Experts, I have upgraded CDH 5.16 to CDP 7.2.4 Cloudera Manager only( not the runtime yet). After upgrade the Cloudera Manager server is running good including agent. But the Cloudera Agents on the rest of the nodes wont start when running $ sudo systemctl restart cloudera-scm-agent The agents are still at 5.16 as the final step for upgrade does show the upgrade button for the agents. Before the upgrade the agents were running fine. Also the agents where shut down during the Cloudera Manager upgrade. How to start the agents and upgrade them to CDP 7.2.4 now. When trying to restart agents It just shows: $ sudo systemctl status cloudera-scm-agent ● cloudera-scm-agent.service - LSB: Cloudera SCM Agent Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-agent; bad; vendor preset: disabled) Active: active (exited) since Tue 2021-01-12 17:52:10 EST; 7s ago Docs: man:systemd-sysv-generator(8) Process: 14093 ExecStop=/etc/rc.d/init.d/cloudera-scm-agent stop (code=exited, status=0/SUCCESS) Process: 14210 ExecStart=/etc/rc.d/init.d/cloudera-scm-agent start (code=exited, status=0/SUCCESS) In the /var/log/cloudera-scm-agent/cloudera-scm-agent.log I see some below messages: u'KS_INDEXER', u'ZOOKEEPER-SERVER', u'SERVER', u'HIVE_ON_TEZ', u'HIVE_ON_TEZ', u'HIVE_LLAP', u'HIVE_LLAP', u'KEYTRUSTEE_SERVER', u'KEYTRUSTEE_SERVER', u'SCHEMAREGISTRY-SCHEMA_REGISTRY_SERVER', u'SCHEMA_REGISTRY_SERVER', u'OZONE', u'OZONE', u'MAPREDUCE-JOBTRACKER', u'JOBTRACKER', u'THALES_KMS-HSMKP_THALES', u'HSMKP_THALES'], u'flood_seed_timeout': 100, u'eventserver_port': 7185} Traceback (most recent call last): File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/agent.py", line 1566, in handle_heartbeat_response self._handle_heartbeat_response(response) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/agent.py", line 1581, in _handle_heartbeat_response self.java_home_config = self.extra_configs['JAVA_HOME'] KeyError: 'JAVA_HOME' Any thoughts? Thanks!
... View more
Labels:
- Labels:
-
Cloudera Manager
-
Cloudera on premises
01-09-2021
09:19 AM
@GangWar you are a genius! After this java parameter change all CDH services started smoothly and everything running fine with Active Directory kerberos. Thanks so much!
... View more
01-08-2021
12:36 PM
Hello Experts, After changing MIT Kerberos to AD Kerberos and Regenerating all the Kerberos credentials in CM the zookeeper, YARN etc. is not starting. There is an error about the Active Directory samaccount not able to login as the zookeeper principal. I checked that the principals are created in the AD OrgUnit for Cloudera. And the $ kinit -kt zookeeper.keytab zookeeper/redacted@ADREALM on the linux servers works fine. Any thoughts how to fix? SERVICE_TYPEZOOKEEPER SEVERITYCRITICAL STACKTRACE javax.security.sasl.SaslException: Problem with callback handler [Caused by javax.security.sasl.SaslException: redacted@ADREALM.COM is not authorized to connect as zookeeper/redacted@ADREALM.COM] at com.sun.security.sasl.gsskerb.GssKrb5Server.doHandshake2(GssKrb5Server.java:333) at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:161) at org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer.authenticate(SaslQuorumAuthServer.java:98) at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:449) at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReceiverThread.run(QuorumCnxManager.java:423) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: javax.security.sasl.SaslException: Thanks!
... View more
Labels:
- « Previous
-
- 1
- 2
- Next »