All Questions, Answers, and Comments in Cloudera Community

Re: Failed to upload YARN MapReduce Framework JARs

fpang — Sat, 23 Jun 2018 00:25:47 GMT

Hi Kelonsen,

Were you adding a new Yarn service to existing cluster?

The YarnInstallMrFrameworkJars always creates the framework jar directory in hdfs before the upload.

Can you grab the earlier part of that screen, or find that command in the Previous Commands?

It would show whether the framework jar directory creation was attempted and whether is was successful.

Also, do you have oozie and hive services?

Did you add and start services one at a time?, do you know the order that services were added and started?

Did the step previous to the failing step have the title " Create YARN MapReduce Framework Dir" ?

which the exact impala version of cdh 6

teng — Wed, 20 Jun 2018 04:14:44 GMT

i see

https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_60_packaging.html

it said impala version impala-3.9.0+cdh6.0.0-beta1

but in CDH 6

impalad version 2.11.0-cdh6.0.0-beta1 RELEASE (build ded6764d7c4ac82cb9a0173e668b120d67fe2a6f)

Is it the same version of CDH 5.14?

Re: Failed to upload YARN MapReduce Framework JARs

kelonsen — Tue, 19 Jun 2018 05:35:47 GMT

I have solved it, because the user yarn have not permission to upload jar to hdfs. i should look the stderr log carefully. thanks !

18/06/19 11:16:10 INFO uploader.FrameworkUploader: Target hdfs://hadoop10/user/yarn/mapreduce/mr-framework/3.0.0-cdh6.0.0-mr-framework.tar.gz
18/06/19 11:16:11 INFO uploader.FrameworkUploader: Set replication to 3 for path: hdfs://hadoop10/user/yarn/mapreduce/mr-framework/3.0.0-cdh6.0.0-mr-framework.tar.gz
18/06/19 11:16:11 INFO uploader.FrameworkUploader: Disabling Erasure Coding for path: hdfs://hadoop10/user/yarn/mapreduce/mr-framework/3.0.0-cdh6.0.0-mr-framework.tar.gz
18/06/19 11:16:11 ERROR uploader.FrameworkUploader: Error in execution Permission denied: user=yarn, access=EXECUTE, inode="/user":hdfs:supergroup:drwx------
	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:400)
	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:316)
	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:243)

Failed to upload YARN MapReduce Framework JARs

kelonsen — Tue, 19 Jun 2018 05:12:24 GMT

version: cloudera manager 6.0-beta1

problem: Completed only 1/2 steps. First failure: Failed to execute command Install YARN MapReduce Framework JARs on service YARN (MR2 Included)

Re: Question on "[ANNOUNCE] Cloudera Enterprise 6 Beta Re...

Icarus — Fri, 15 Jun 2018 20:41:06 GMT

Hi...

Thanks for the question. We have customers in different versions of RHEL and they need to know what is the minimum OS version supported so they can upgrade their current version of CDH to CDH 6.0. Knowing what are the minimal versions provides clarity in that respect.

Re: CM/CDH 6 fails to activate csd spark2.3.0

Ana — Fri, 15 Jun 2018 19:55:15 GMT

Spark 2.x is now part of CDH 6. You don't need a separate parcel.

Re: Installation fails ,Could not initialize database ser...

Ana — Fri, 15 Jun 2018 19:45:56 GMT

Can you try the manual installation procedure to debug the installer?

Question on "[ANNOUNCE] Cloudera Enterprise 6 Beta Released"

saranvisa — Fri, 15 Jun 2018 09:04:45 GMT

In the below announcement for c6 beta, Is there any specific reason for specifying OS for RHEL as follows? I mean why don't you simply mention only the first line. Pls help me to understand -or- it is just a typo?

Operating Systems

RHEL 6.8 and above
RHEL 7.2 and above

https://community.cloudera.com/t5/Community-News-Release/ANNOUNCE-Cloudera-Enterprise-6-Beta-Released/m-p/67312#M226

Re: CM/CDH 6 fails to activate csd spark2.3.0

Michalis — Thu, 14 Jun 2018 18:41:02 GMT

Ah yes, that make sense!

Re: CM/CDH 6 fails to activate csd spark2.3.0

MMR — Thu, 14 Jun 2018 17:15:53 GMT

Thanks for your inputs. Spark2 is available as an add on service, I got confused with the name as it is 'SPARK' instead of 'SPARK2'

Re: CM/CDH 6 fails to activate csd spark2.3.0

Michalis — Thu, 14 Jun 2018 12:19:18 GMT

com.cloudera.parcel.ParcelRelationsException: CDH 6.0.0-1.cdh6.0.0.p0.339140 replaces SPARK2 2.3.0.cloudera2-1.cdh5.13.3.p0.316101.

That seems to suggest that CDH6 replaces SPARK2 2.3.0.cloudera2-1.cdh5.13.3.p0.316101 (see [0])

SPARK2 is already present [1] in the CDH6 parcel

[0]

...
{
"name": "spark",
"pkg_release": "beta1.339140",
"pkg_version": "2.2.0+cdh6.0.0",
"version": "2.2.0-cdh6.0.0-beta1"
},
...
https://archive.cloudera.com/cdh6/6.0.0-beta1/parcels/manifest.json

[1] notice cdh5.13.3 use the link below for brief explanation of what the version denotes.

https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball.html

Re: CM/CDH 6 fails to activate csd spark2.3.0

saranvisa — Thu, 14 Jun 2018 12:26:22 GMT

According to the below link, Cloudera Enterprise 6 beta supports Spark 2.2. May be you can try Spark 2.2 instead of 2.3, it may help you

https://community.cloudera.com/t5/Community-News-Release/ANNOUNCE-Cloudera-Enterprise-6-Beta-Released/m-p/67312#M226

Re: Getting An internal error while Linking Altus to our...

surajacharya — Thu, 14 Jun 2018 02:12:52 GMT

So if you go to your subscription and then look at access control IAM. You
should see a Cloudera altus application. Also, if your environment is
created, you can go ahead and create the cluster and see if that succeeds.

Hope this helps.

Suraj

Re: Getting An internal error while Linking Altus to our...

VamsiMN — Thu, 14 Jun 2018 02:08:28 GMT

Hi Suraj, the roles are all set. All required network setup on azure is done with the base template . Is there any other way to figure out the exact problem?

CM/CDH 6 fails to activate csd spark2.3.0

MMR — Thu, 14 Jun 2018 01:41:16 GMT

Hello,

I'm trying to install spark 2.3.0 on CM/CDH 6 beta, after downloading the CDS Powered By Apache Spark CSD file to /opt/cloudera/parcels/ and adding the Remote Parcel Repository URLs in Parcel Settings 'http://archive.cloudera.com/spark2/parcels/2.3.0.cloudera2/'.

I'm getting the following error while activating the parcel

spark jar file on my cloudera server

ll /opt/cloudera/csd/
total 20
-rw-r--r-- 1 cloudera-scm cloudera-scm 19010 Jun 13 17:08 SPARK2_ON_YARN-2.3.0.cloudera2.jar

logs from cloudera server

2018-06-13 18:20:00,970 WARN scm-web-225:com.cloudera.server.web.cmf.WebController: Failed to activate parcel SPARK2:2.3.0.cloudera2-1.cdh5.13.3.p0.316101 for cluster DbCluster{id=1, name=cluster}
2018-06-13 18:20:00,972 ERROR scm-web-225:com.cloudera.server.web.common.JsonResponse: JsonResponse created with throwable:
com.cloudera.parcel.ParcelRelationsException: CDH 6.0.0-1.cdh6.0.0.p0.339140 replaces SPARK2 2.3.0.cloudera2-1.cdh5.13.3.p0.316101.
at com.cloudera.parcel.components.ParcelDependencyManagerImpl.validateDependencies(ParcelDependencyManagerImpl.java:418)
at com.cloudera.parcel.components.ParcelDependencyManagerImpl.validateDependenciesForActivation(ParcelDependencyManagerImpl.java:375)
at com.cloudera.parcel.components.ParcelManagerImpl.activateParcel(ParcelManagerImpl.java:295)
at com.cloudera.parcel.components.ParcelManagerImpl.activateParcel(ParcelManagerImpl.java:246)
at com.cloudera.server.web.cmf.parcel.ParcelActivationController.activateParcel(ParcelActivationController.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:158)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:121)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:134)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
at java.lang.Thread.run(Thread.java:748)

Re: Installation fails ,Could not initialize database ser...

stanton009 — Wed, 13 Jun 2018 18:17:43 GMT

I am getting a simiilar problem on CentOS 7.

From the Cloudera 6 bin installer:
Failed to start Embedded Service and Configuration Database, See /var/log/cloudera-manager-installer/7.start-cloudera-scm-server-db.log for details. Click OK to to revert this installation.

When checking that log file we find:
Job for cloudera-scm-server-db.service failed because the control process exited with error code. See "systemctl status cloudera-scm-server-db.service" and "journalctl -xe" for details.

when running systemctl status cloudera-scm-server-db.service
Failed to get properties: Access denied

when checking journalctl -xe [server info removed]
...
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit cloudera-scm-server-db.service has begun starting up. The files belonging to this database system will be owned by user "cloudera-scm".  This user must also own the server process. The database cluster will be initialized with locale "en_US.UTF8". The default text search configuration will be set to "english". fixing permissions on existing directory /var/lib/cloudera-scm-server-db/data ... initdb: could not change permissions Could not initialize database server. This usually means that your PostgreSQL installation failed or isn't working properly. Please ensure that PostgreSQL can be installed. Please also uninstall any other instances of ... 
I confirmed that postgresql is installed:
# rpm -qa | grep postgresql postgresql-9.2.23-3.el7_4.x86_64 postgresql-libs-9.2.23-3.el7_4.x86_64 postgresql-server-9.2.23-3.el7_4.x86_64
...
The /var/lib/cloudera-scm-server-db/data directory (and parent) are not present on the system. Even if you create them yourself which ownership to cloudera-scm:cloudera-scm, it disappears during installation and you get the same error.

Re: Getting An internal error while Linking Altus to our...

surajacharya — Wed, 13 Jun 2018 15:33:28 GMT

Hi ,

Did you do the role assignment of Cloudera Service principle to your subscription?

Here are the document on how to do it : https://www.cloudera.com/documentation/altus/topics/alt_env_setup_azure.html#role_based_access

Regarding the error you see, I am sorry that we dont show a much more explicit error message. We will work on improving the error message so others dont run into the same issue again.

Let me know if you have any more questions.

Thanks

Suraj

Getting An internal error while Linking Altus to our Azure Account.

VamsiMN — Wed, 13 Jun 2018 14:46:58 GMT

We started the process to setup Azure Environment in Altus.

As a part of it, I launched quick start template menioned in the page to get required network resources. Following are the created resources.

a Resource Group
a Virtual Network
a Network Security Group
a VM running bind DNS server
a User Assigned Managed Service Identity (MSI)

I entered all required details in evironment setup wizard and clicked on create. I see the following error. I got this error multiple times, not sure what's the problem.

An internal error has occurred. Retry your request, but if the problem persists, contact us with details by posting a message on the Cloudera Community forums.

Request ID: cdf78a0a-58e6-4879-b11d-a7f8a62f1e41

Can someone help me understand what went wrong.

Re: Installation fails ,Could not initialize database ser...

mostrows — Mon, 11 Jun 2018 16:43:08 GMT

Please identify the OS being used.

Was postgres correctly installed?

Is postgres running?

Please post all of the log files from the installer directory.

Installation fails ,Could not initialize database server.

datasir — Mon, 11 Jun 2018 03:38:11 GMT

vi 6.start-cloudera-scm-server-db.log

Could not initialize database server.
This usually means that your PostgreSQL installation failed or isn't working properly.
PostgreSQL is installed using the set of repositories found on this machine. Please
ensure that PostgreSQL can be installed. Please also uninstall any other instances of
PostgreSQL and then try again., giving up