Member since
04-22-2014
1218
Posts
341
Kudos Received
157
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 26302 | 03-03-2020 08:12 AM | |
| 16441 | 02-28-2020 10:43 AM | |
| 4739 | 12-16-2019 12:59 PM | |
| 4490 | 11-12-2019 03:28 PM | |
| 6694 | 11-01-2019 09:01 AM |
10-01-2018
08:52 AM
1 Kudo
@Nishikant, yes, when testing on a limited-resource VM, I shut down anything I don't need. Still, with limited resources, don't expect awesome performance. If you are using YARN, that spins up containers that will require memory as well. If you have Cloudera Manager installed, it is what you use to start and stop all hadoop roles. You may also need to configure heap space in Cloudera Manager to reduce the heap to ensure you don't overcommit.
... View more
09-24-2018
09:31 AM
@salve08, It looks as if you did not show us the actual curl command that is not working. Is there something specific you were trying to do with curl? Cloudera Manager is a graphical user interface. To connect to it use a browser to navigate to: http://cn_host:7180 CM runs without TLS on port 7180 by default.
... View more
09-24-2018
09:22 AM
@alvaro_soares, If the Oracle Jar is in the right place and you ran scm_prepare_database as root successfully, permissions on the oracle jdbc jar may be implicated. Make sure the Cloudera Manager user (cloudera-scm) has read access to the jar files in /usr/share/java.
... View more
09-21-2018
12:38 PM
@rseidler, I think that "skipped" happens because the logging uses throttled logging (only 1 of many such lines are printed. Are all your agents having trouble heartbeating or just one or two? Maybe take a screen shot of your Hosts tab in CM to give us an idea. That log snippet doesn't tell me much other than the fact that some clients have sent a heartbeat at some point since CM was started.
... View more
09-21-2018
12:29 PM
@xBigDatax, I'm sorry but can't quite understand what you are saying/asking. Can you clarify what you mean by: "edge node cloudera version showing none which is hold me to apply any role on edge node" Any agent should have 'tmpfs' listed in monitored_nodev_filesystem_types. This should be in your config.ini unless you have a reason to remove it... I don't know of one. monitored_nodev_filesystem_types=nfs,nfs4,tmpfs
... View more
09-21-2018
12:23 PM
@AadD, If you can post the command(s) you used to create the CM private key and cert that would help. Also, if you can share the output for keytool -list -v -keystore <keystore> that could help. Generally, if the server won't start at all, there is a problem with a password or permissions on the JKS file you are using for your Key file or truststore. When implementing for the first time, only enable TLS for the CM admin console (UI). If there is a problem using a certificate, it should start up using non-tls and warn you that something's wrong. Check /var/log/cloudera-scm-server/cloudera-scm-server.log for exceptions or messages pertaining to SSL/TLS. Cheers, Ben
... View more
09-20-2018
10:31 AM
@Paulina, I agree that this is pretty odd, but we'll figure it out We see that you have LDAP Groups Mapping configured for HDFS I assume, then, that HDFS stuff appears to be working correctly. Let's confirm by running: # hdfs groups maslova # hdfs groups hdfs If Sentry is using the same lookup as HDFS, then we would assume the results would be the same and you would see that maslova is a member of ploymatica group. However, if the "hdfs groups" command results differ, this indicates that Sentry is not using the same configuration for group lookups. Let's start by getting those results.
... View more
09-20-2018
09:55 AM
1 Kudo
@xBigDatax, Sadly my previous didn't get published for some reason, so I'll repeat in short. The following is not fatal and is not sign of a real problem: [20/Sep/2018 18:18:42 +0000] 123357 Monitor-HostMonitor throttling_logger ERROR Could not find local file system for /var/run/cloudera-scm-agent/process Notice that the "Monitor-HostMonitor" thread is what returns the error. Your agent appears to be in good health and it should be heartbeating. If /var/run/cloudera-scm-agent/process exists and shows up when running the "df" command, there is likely no big problem: cm_processes 8133940 48444 8085496 1% /run/cloudera-scm-agent/process I remember seeing that error years ago and it was caused by a mismatch between a Modern agent code base and an older /etc/cloudera-scm-agent/config.ini. Check to make sure you see this in your agent's config.ini: monitored_nodev_filesystem_types=nfs,nfs4,tmpfs If it is not, then add it and restart the agent with "service cloudera-scm-agent restart" The monitored_nodev_filesystem_types option lists the file systems that are considered 'nodev'. If "tmpfs" is not listed there, the file system will be considered 'local'. As we can see, cm_process is 'tmpfs': cm_processes on /run/cloudera-scm-agent/process type tmpfs (rw,relatime,mode=751) The config.ini configuration I mentioned should included tmpfs so certain evaluation is skipped (and you won't get that ERROR message).
... View more
09-20-2018
09:44 AM
@rseidler, Since the basics are covered, I'll say that the stack trace you provided looks pretty odd and indicates that the agent was reading a reply from Cloudera Manager but before it could complete, the connection went away... The Error-Log of the agent is: >>[20/Sep/2018 10:25:06 +0000] 9377 MainThread agent ERROR Heartbeating to node001:7182 failed. >>Traceback (most recent call last): >> File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1371, in _send_heartbeat >> response = self.requestor.request('heartbeat', heartbeat_data) >> File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request >> return self.issue_request(call_request, message_name, request_datum) >> File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request >> call_response = self.transceiver.transceive(call_request) >> File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive >> result = self.read_framed_message() >> File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 491, in read_framed_message >> framed_message = response_reader.read_framed_message() >> File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 411, in read_framed_message >> buffer_length = self._read_buffer_length() >> File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 424, in _read_buffer_length >> raise ConnectionClosedException("Reader read 0 bytes.") >>ConnectionClosedException: Reader read 0 bytes. We see in the code that this means that no bytes were received: 421 def _read_buffer_length(self): 422 read = self.reader.read(BUFFER_HEADER_LENGTH) 423 if read == '': 424 raise ConnectionClosedException("Reader read 0 bytes.") This does not appear to be a TCP problem, so I would assert that we will likely find some more information on the Cloudera Manager side. Please check: /var/log/cloudera-scm-server/cloudera-scm-server.log See if you find messages regarding that host or regarding a problem processing heartbeats. Since this is a "clean" problem, I suspect that Cloudera Manager may not be accepting the heartbeat and should hopefully tell you why. *** NOTE: If Cloudera doesn't show any information, check to see what server is listening on port 7182 just to make sure it is really CM: # netstat -nap |grep 7182 |grep LISTEN (note the pid) # ps aux |grep <pid> For example: # netstat -nap |grep 7182|grep LISTEN tcp 0 0 0.0.0.0:7182 0.0.0.0:* LISTEN 28669/java # ps aux |grep 28669 |grep "cmf.Main" This should return one result which is the CM process.
... View more
09-20-2018
08:26 AM
1 Kudo
@AadD, Thanks for trying out Cloudera; sorry that it has been a rocky start, but you came to the right place for help. We'll need some information in order to assist: - What steps did you follow to intall? What documentation? (list specific URLs if you can) - What steps did you follow to enable TLS? (list specific URLs if you can) - Did Cloudera Manager ever start and allow you to access it on port 7180? (http://cm_host:7180?) - How do you know that Cloudera Manager running or responding to HTTP requests? What did you do to test? - If you ssh to the host where Cloudera Manager should be running, is it listening on port 7180? 7183? 7182? Run: # netstat -nap |grep 7180;netstat -nap |grep 7183; netstat -nap |grep 7182 - If the problem is with TLS, then the following steps will turn off the TLS port for the Cloudera Manager User Interface: (1) Review your Cloudera Manager database configuration: # cat /etc/cloudera-scm-server/db.properties (2) Use the information in db.properties to run the psql command: For example, this is how I would connect on my host: - I see the following in db.properties: com.cloudera.cmf.db.type=postgresql com.cloudera.cmf.db.host=localhost:7432 com.cloudera.cmf.db.name=scm com.cloudera.cmf.db.user=scm com.cloudera.cmf.db.password=FpZ3wh9LFT com.cloudera.cmf.db.setupType=EMBEDDED (3) Use the db.properties information to connect to your database For example: - Based on the information in my environment, that would be: # psql -U scm -h localhost -p 7432 Password for user scm: FpZ3wh9LFT psql (9.2.18) Type "help" for help. scm=> (4) Check the configuration that governs TLS for Cloudera Manager and Cloudera Manager agent communication: scm=> select * from configs where attr = 'web_tls'; scm=> select * from configs where attr = 'agent_tls'; (5) If any rows are returned from the above commands, delete them: scm=> delete from configs where attr = 'web_tls'; scm=> delete from configs wehre attr = 'agent_tls'; Search again to verify that no rows are returned: scm=> select * from configs where attr = 'web_tls'; scm=> select * from configs where attr = 'agent_tls'; (6) If there are no rows returned, restart Cloudera Manager with: # service cloudera-scm-server restart (7) CM can take minutes to start, so wait about 5 to be sure and then run the netstats again. If you see that CM is listening on port 7182 (agent communication) and 7180 (non-tls web ui) then try accessing it via your browser. If CM still fails to start, review the Cloudera Manager log. On the Cloudera Manager host it is by default: /var/log/cloudera-scm-server/cloudera-scm-server.log Let us know how it goes. Cheers, Ben
... View more