Member since
01-25-2016
5
Posts
0
Kudos Received
0
Solutions
01-28-2016
06:00 PM
I checked the log files on the node running hive and in /var/log/hive/hadoop-cmf-CD-HIVE-zzISkbiD-HIVEMETASTORE-ip-10-0-1-145.eu-central-1.compute.internal.log.out I get this ERROR message ERROR org.apache.hadoop.hdfs.KeyProviderCache: [pool-6-thread-2]: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !! and in the Hue log file I get : [28/Jan/2016 17:22:02 -0800] models ERROR error syncing beeswax
Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/desktop/core/src/desktop/models.py", line 297, in sync
for job in find_jobs_with_no_doc(SavedQuery):
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/db/models/query.py", line 96, in __iter__
self._fetch_all()
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/db/models/query.py", line 857, in _fetch_all
self._result_cache = list(self.iterator())
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/db/models/query.py", line 220, in iterator
for row in compiler.results_iter():
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/db/models/sql/compiler.py", line 713, in results_iter
for rows in self.execute_sql(MULTI):
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/db/models/sql/compiler.py", line 786, in execute_sql
cursor.execute(sql, params)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/db/backends/sqlite3/base.py", line 452, in execute
return Database.Cursor.execute(self, query, params)
OperationalError: no such column: beeswax_savedquery.is_redacted
[28/Jan/2016 17:22:02 -0800] models ERROR error syncing search
Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/desktop/core/src/desktop/models.py", line 315, in sync
from search.models import Collection
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/apps/search/src/search/models.py", line 34, in <module>
from libsolr.api import SolrApi
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/desktop/libs/libsolr/src/libsolr/api.py", line 46, in <module>
class SolrApi(object):
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/desktop/libs/libsolr/src/libsolr/api.py", line 50, in SolrApi
def __init__(self, solr_url, user, security_enabled=SECURITY_ENABLED.get()):
AttributeError: 'Config' object has no attribute 'get'
[28/Jan/2016 17:33:52 -0800] views ERROR Error in config validation by liboozie: <html><head><title>Apache Tomcat/6.0.44 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:w
hite;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#
525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sa
ns-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - </h1><HR size="1" noshade="noshade"><p><b>type</
b> Exception report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>java.lang.Uns
upportedOperationException
org.apache.oozie.util.MetricsInstrumentation.getVariables(MetricsInstrumentation.java:333)
org.apache.oozie.servlet.BaseAdminServlet.instrToJson(BaseAdminServlet.java:339)
org.apache.oozie.servlet.BaseAdminServlet.sendInstrumentationResponse(BaseAdminServlet.java:396)
org.apache.oozie.servlet.BaseAdminServlet.doGet(BaseAdminServlet.java:127)
javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:289)
javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:589)
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:552)
org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/6.0.44 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.44</h3></body></html> (error 500)
Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/desktop/core/src/desktop/views.py", line 445, in _get_config_errors
for confvar, error in validator(request.user):
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/desktop/libs/liboozie/src/liboozie/conf.py", line 86, in config_validator
intrumentation = api.get_instrumentation()
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/desktop/libs/liboozie/src/liboozie/oozie_api.py", line 304, in get_instrumentation
resp = self._root.get('admin/instrumentation', params)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/desktop/core/src/desktop/lib/rest/resource.py", line 97, in get
return self.invoke("GET", relpath, params, headers=headers, allow_redirects=True)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/desktop/core/src/desktop/lib/rest/resource.py", line 78, in invoke
urlencode=self._urlencode)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/desktop/core/src/desktop/lib/rest/http_client.py", line 161, in execute
raise self._exc_class(ex)
RestException: <html><head><title>Apache Tomcat/6.0.44 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>java.lang.UnsupportedOperationException
org.apache.oozie.util.MetricsInstrumentation.getVariables(MetricsInstrumentation.java:333)
org.apache.oozie.servlet.BaseAdminServlet.instrToJson(BaseAdminServlet.java:339)
org.apache.oozie.servlet.BaseAdminServlet.sendInstrumentationResponse(BaseAdminServlet.java:396)
org.apache.oozie.servlet.BaseAdminServlet.doGet(BaseAdminServlet.java:127)
javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:289)
javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:589)
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:552)
org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/6.0.44 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.44</h3></body></html> (error 500)
[28/Jan/2016 17:33:53 -0800] conf ERROR The application won't work without a running HiveServer2.
Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/apps/beeswax/src/beeswax/conf.py", line 185, in config_validator
server.get_databases()
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/apps/beeswax/src/beeswax/server/dbms.py", line 151, in get_databases
handle = self.execute_and_wait(query, timeout_sec=timeout)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/apps/beeswax/src/beeswax/server/dbms.py", line 548, in execute_and_wait
handle = self.client.query(query)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 923, in query
return self._client.execute_async_query(query, statement)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 703, in execute_async_query
return self.execute_async_statement(statement=query_statement, confOverlay=configuration)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 721, in execute_async_statement
res = self.call(self._client.ExecuteStatement, req)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 597, in call
raise QueryServerException(Exception('Bad status for request %s:\n%s' % (req, res)), message=message)
QueryServerException: Bad status for request TExecuteStatementReq(confOverlay={}, sessionHandle=TSessionHandle(sessionId=THandleIdentifier(secret='r\x94N\xc0\xac\x99D\xd9\xad\xe1\xb7\x15?N\xd4B', guid='o\x1c\xe77\x06\x7fJ\x19\xaa\xd9N\xf0.\xcd[\xc2')), runAsync=True, statement="SHOW DATABASES LIKE '*'"):
TExecuteStatementResp(status=TStatus(errorCode=40000, errorMessage="Error while compiling statement: FAILED: InvalidConfigurationException hive.server2.authentication can't be none in non-testing mode", sqlState='42000', infoMessages=["*org.apache.hive.service.cli.HiveSQLException:Error while compiling statement: FAILED: InvalidConfigurationException hive.server2.authentication can't be none in non-testing mode:17:16", 'org.apache.hive.service.cli.operation.Operation:toSQLException:Operation.java:326', 'org.apache.hive.service.cli.operation.SQLOperation:prepare:SQLOperation.java:102', 'org.apache.hive.service.cli.operation.SQLOperation:runInternal:SQLOperation.java:171', 'org.apache.hive.service.cli.operation.Operation:run:Operation.java:268', 'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementInternal:HiveSessionImpl.java:410', 'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementAsync:HiveSessionImpl.java:397', 'org.apache.hive.service.cli.CLIService:executeStatementAsync:CLIService.java:258', 'org.apache.hive.service.cli.thrift.ThriftCLIService:ExecuteStatement:ThriftCLIService.java:509', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1313', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1298', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56', 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:285', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615', 'java.lang.Thread:run:Thread.java:745', "*org.apache.sentry.binding.hive.conf.InvalidConfigurationException:hive.server2.authentication can't be none in non-testing mode:32:16", 'org.apache.sentry.binding.hive.authz.HiveAuthzBinding:validateHiveServer2Config:HiveAuthzBinding.java:167', 'org.apache.sentry.binding.hive.authz.HiveAuthzBinding:validateHiveConfig:HiveAuthzBinding.java:135', 'org.apache.sentry.binding.hive.authz.HiveAuthzBinding:<init>:HiveAuthzBinding.java:83', 'org.apache.sentry.binding.hive.authz.HiveAuthzBinding:<init>:HiveAuthzBinding.java:79', 'org.apache.sentry.binding.hive.HiveAuthzBindingHook:<init>:HiveAuthzBindingHook.java:109', 'sun.reflect.NativeConstructorAccessorImpl:newInstance0:NativeConstructorAccessorImpl.java:-2', 'sun.reflect.NativeConstructorAccessorImpl:newInstance:NativeConstructorAccessorImpl.java:57', 'sun.reflect.DelegatingConstructorAccessorImpl:newInstance:DelegatingConstructorAccessorImpl.java:45', 'java.lang.reflect.Constructor:newInstance:Constructor.java:526', 'java.lang.Class:newInstance:Class.java:374', 'org.apache.hadoop.hive.ql.hooks.HookUtils:getHooks:HookUtils.java:60', 'org.apache.hadoop.hive.ql.Driver:getHooks:Driver.java:1294', 'org.apache.hadoop.hive.ql.Driver:compile:Driver.java:407', 'org.apache.hadoop.hive.ql.Driver:compile:Driver.java:305', 'org.apache.hadoop.hive.ql.Driver:compileInternal:Driver.java:1110', 'org.apache.hadoop.hive.ql.Driver:compileAndRespond:Driver.java:1104', 'org.apache.hive.service.cli.operation.SQLOperation:prepare:SQLOperation.java:100'], statusCode=3), operationHandle=None) Does this help?
... View more
01-28-2016
05:44 PM
Hi, It is listed as running
... View more
01-28-2016
03:11 AM
I deployed a Cloudera EDH cluster on AWS using Quickstart and the cloudera-director executable. I took the aws.reference.conf file that comes with the Cloudera launcher. Everything works fine during the deployment, but I am having problems with HUE/Hive editor. When I go to the HUE UI I see the screen below : And when I try to open the Hive Editor I get : Can someone help?
... View more
01-27-2016
02:12 AM
I also tried deploying the cluster using the command line (cloudera-director) and using the aws.reference.conf file, the job stops with the following output on the terminal : * Creating Sentry Database ... done
* Waiting for firstRun on cluster C5-Reference-AWS ... done
* Cloudera Manager 'First Run' command execution failed: Failed to perform First Run of services. ... then I tried to check the status : [ec2-user@ip-10-0-2-205 setup-default]$ cloudera-director status aws.reference.conf
Process logs can be found at /home/ec2-user/.cloudera-director/logs/application.log
Plugins will be loaded from /var/lib/cloudera-director-plugins
Cloudera Director 2.0.0 initializing ...
Unexpected internal error (see logs): Cluster C5-Reference-AWS is in stage BOOTSTRAP_FAILED. See cluster status and server logs for details. Any help is appreciated!
... View more
01-27-2016
12:47 AM
Hello, I have followed the instructions described here https://s3.amazonaws.com/quickstart-reference/cloudera/hadoop/latest/doc/Cloudera_EDH_on_AWS.pdf for deploying EDH on AWS. Things seem fine all the way until boostrapping the cluster. I have the following cluster : AMI : ami-30d9e02d Cloudera Manager on d2.xlarge 2 masters (m4.2xlarge) 2 workers (d2.xlarge) 1 gateway ( m4.2xlarge) I used the AWS CloudFormation template and was able to connect to Cloudera Manager via the web console w/o problems. I deployed the cluster and all EC2 nodes are running with Status Checks ok (2/2), but the cluster fails at b ootstrap. I logged to one of the masters and I see the following at the bottom of /var/log/cloudera-scm-agent/cloudera-scm-agent.log : [27/Jan/2016 03:16:44 +0000] 3069 MonitorDaemon-Reporter throttling_logger ERROR Error sending messages to firehose: MGMT-HOSTMONITOR-244944378552b77b5c898d702d752f7f
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/src/cmf/monitor/firehose.py", line 116, in _send
self._port)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 469, in __init__
self.conn.connect()
File "/usr/lib64/python2.6/httplib.py", line 720, in connect
self.timeout)
File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
raise error, msg
error: [Errno 111] Connection refused
[27/Jan/2016 03:19:46 +0000] 3069 DnsResolutionMonitor throttling_logger ERROR Timeout with args ['/usr/java/jdk1.7.0_67-cloudera/bin/java', '-classpath', '/usr/share/cmf/lib/agent-5.5.1.jar', 'com.cloudera.cmon.agent.DnsTest']
None
[27/Jan/2016 03:19:46 +0000] 3069 DnsResolutionMonitor throttling_logger ERROR Failed to run DnsTest.
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/src/cmf/monitor/host/dns_names.py", line 83, in collect_dns_metrics
self._subprocess_with_timeout(args, self._poll_timeout)
File "/usr/lib64/cmf/agent/src/cmf/monitor/host/dns_names.py", line 55, in _subprocess_with_timeout
return subprocess_with_timeout(args, timeout)
File "/usr/lib64/cmf/agent/src/cmf/subprocess_timeout.py", line 94, in subprocess_with_timeout
raise Exception("timeout with args %s" % args)
Exception: timeout with args ['/usr/java/jdk1.7.0_67-cloudera/bin/java', '-classpath', '/usr/share/cmf/lib/agent-5.5.1.jar', 'com.cloudera.cmon.agent.DnsTest'] and in the same file for the gateway : [27/Jan/2016 03:20:27 +0000] 3072 DnsResolutionMonitor throttling_logger ERROR Timeout with args ['/usr/java/jdk1.7.0_67-cloudera/bin/java', '-classpath', '/usr/share/cmf/lib/agent-5.5.1.jar', 'com.cloudera.cmon.agent.DnsTest']
None
[27/Jan/2016 03:20:27 +0000] 3072 DnsResolutionMonitor throttling_logger ERROR Failed to run DnsTest.
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/src/cmf/monitor/host/dns_names.py", line 83, in collect_dns_metrics
self._subprocess_with_timeout(args, self._poll_timeout)
File "/usr/lib64/cmf/agent/src/cmf/monitor/host/dns_names.py", line 55, in _subprocess_with_timeout
return subprocess_with_timeout(args, timeout)
File "/usr/lib64/cmf/agent/src/cmf/subprocess_timeout.py", line 94, in subprocess_with_timeout
raise Exception("timeout with args %s" % args)
Exception: timeout with args ['/usr/java/jdk1.7.0_67-cloudera/bin/java', '-classpath', '/usr/share/cmf/lib/agent-5.5.1.jar', 'com.cloudera.cmon.agent.DnsTest'] and the same log file in the worker node has the following : [27/Jan/2016 03:08:52 +0000] 3012 MainThread agent ERROR Failed to connect to previous supervisor.
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 1635, in find_or_start_supervisor
self.configure_supervisor_clients()
File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 1882, in configure_supervisor_clients
supervisor_options.realize(args=["-c", os.path.join(self.supervisor_dir, "supervisord.conf")])
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/supervisor-3.0-py2.6.egg/supervisor/options.py", line 1564, in realize
Options.realize(self, *arg, **kw)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/supervisor-3.0-py2.6.egg/supervisor/options.py", line 311, in realize
self.process_config()
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/supervisor-3.0-py2.6.egg/supervisor/options.py", line 319, in process_config
self.process_config_file(do_usage)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/supervisor-3.0-py2.6.egg/supervisor/options.py", line 354, in process_config_file
self.usage(str(msg))
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/supervisor-3.0-py2.6.egg/supervisor/options.py", line 142, in usage
self.exit(2)
SystemExit: 2 I'm pretty new to Cloudera and AWS so any insight is appreciated!
... View more
