Member since
10-02-2017
116
Posts
3
Kudos Received
8
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1000 | 07-18-2020 12:04 PM | |
1728 | 09-11-2019 01:14 PM | |
2501 | 08-16-2019 08:17 AM | |
6348 | 08-15-2019 12:23 PM | |
4205 | 05-14-2019 08:48 AM |
07-15-2019
09:50 AM
Thank you for the clarification. D.
... View more
07-13-2019
02:50 PM
I'm using Altus Director 6.2.
Is the following declaration in your latest Aluts Director documentation still accurate?
Changing the Instance Type is Not Supported in Azure
Changing the instance type of an already-deployed VM is not supported in Azure.
Changing the instance type of a VM through a tool external to Altus Director is not supported in Azure. You cannot, for example, use the Azure Portal to change the instance type. Altus Director is not updated when instance type changes are made with external tools, and your cluster will show errors in Altus Director.
I have performed both of the actions described in the bullet points above after we decided our existing workers were not sized properly for the amount of data being ingested / processed. Cloudera Director still shows the cluster in a "green" state. No complaints / errors / warnings thus far. Is this really not supported?
Reference: https://www.cloudera.com/documentation/director/latest/topics/director_get_started_azure_important_notes.html
... View more
07-11-2019
08:55 AM
Thank you for the response. To your point about bootstrapping/terminating - I have noticed during testing when I'm cycling through many bootstrapping / terminations that Director can get into a state where it seems to no longer respond to bootstrapping requests (from command line). I'll typically let Director "cool down" for a few minutes and retry.
... View more
07-10-2019
11:31 AM
Assuming the latest version of Altus Director... are there any documentated recommendations for the number of deployments/clusters that Director can reasonbly be expected to manage?
Our clusters are less than 21 nodes each and our Director server is running on an EC2 instance with 8 cores and 60GB of RAM and I have a dedicated RDS MariaDB DB sized at 2 cores and 16GB of RAM with 200GB of storage. Thus far I'm managing around 14 clusters from ranging in size from 9 to 20 nodes. No issues thus far. I'm just curious if there is general guidance around this subject.
... View more
07-10-2019
08:49 AM
We have a need to enable public IPs on the master nodes due to an requirement in Azure when adding instances to a load balancer pool. Our cluster is accessed via private IPs only, outside of this requirement.
Within the cluster bootstrap file, we enable public IPs for the master instance groups. During deployment, director will throw an error similar to the following for some but not all master nodes:
', errorInfo=ErrorInfo{code=INSTANCE_SSH_PORT_UNAVAILABLE, properties={sshServiceEndpoints=[BaseServiceEndpoint{hostEndpoint=HostEndpoint{hostAddressString='10.0.14.15', hostAddress=Optional.of(/10.0.14.15)}, port=Optional.absent(), url=Optional.absent()}, BaseServiceEndpoint{hostEndpoint=HostEndpoint{hostAddressString='52.232.245.168', hostAddress=Optional.of(/52.232.245.168)}, port=Optional.absent(), url=Optional.absent()}]}, causes=[]}}
[2019-07-10 02:49:45.507 +0000] INFO [p-d4ed238466ce-WaitForSshToSucceed] 5f79b1c9-b7d6-41d0-a933-0cfff86cf6e4 POST /api/d6.2/environments/azuresb/deployments/azuresb-1/clusters com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - com.cloudera.launchpad.bootstrap.WaitForServersUntilTime: Waiting until 2019-07-10T03:09:45.038Z for an accessible port on endpoints [10.0.14.15:22, 40.79.57.122:22]
From the logs, it look like Director should try both IP's. I confirmed that I can ssh to the instance's private IP from Director - which leads me to believe it tried to SSH via the public IP, failed, and never tried to connect via the private IP. As I mentioned, this does not happen for all masters, however all masters are configured identically - with a private and public IP.
Can someone confirm if both IP's would need to be accessible from the Director server for the installation to continue?
... View more
07-09-2019
10:47 AM
Cleaned up the redundant common-instanceTemplate section per the latest spec. Looks like Director is now accepting my custom image - however now I'm running into the other limitation that's been a thorn in my side, which is the fact that the custom image can't have data disks defined like they can in AWS. I believe the workaround for that is to define disk count as 0, but I'll need to rebuild my worker packer image to include the additional data disks for hdfs. Anyway, different subject. Thank you Bill for the guidance. I'll mark this issue as resolved.
... View more
07-09-2019
08:31 AM
Bill, I belived I misinterpreted your initial suggestion. I was focused on the common-instanceTemplate section entirely, and didn't check the instances section. My initial interpretation of "instances" section was the more instance specific parameter definitions within the common-instanceTemplate section. Now I see what you are getting at. Let me try another test.....
... View more
07-09-2019
08:19 AM
Also tried updating hocon file to reflect the new formatting approach you referenced: common-instanceTemplate {
# Core config fields that are common to all node types
base {
type: STANDARD_DS13_V2
image: /subscriptions/f5300b4d-bebc-4d95-8b6c-4ea77d9b3423/resourceGroups/Packer/providers/Microsoft.Compute/images/cloudera-packerbake-36.0.55
useCustomManagedImage: Yes
customImagePlan: ""
...
...
...
}
}
# Core config fields that are common to all master nodes
master-base: ${common-instanceTemplate.base} {
computeResourceGroup: ${customer-params.resourceGroup}
availabilitySet: ${customer-params.masterAvailabilitySet}
storageAccountType: "Premium_LRS"
dataDiskSize: 512
managedDisks: Yes
publicIP: Yes
}
# Config fields for master-1 nodes
master-1: ${common-instanceTemplate.master-base} {
dataDiskCount: 1
instanceNamePrefix: master-1
publicIP: ${?common-instanceTemplate.master-base.publicIP}
}
...
...
...
Hitting same exception. Including full trace... ERROR [qtp187120217-21] 14c8453a-4e63-4fec-8eef-e7b587b0af52 POST /api/d6.2/environments/azuresb/deployments - - com.cloudera.launchpad.api.d6_0.response.ExceptionTranslatorD6_0: Found validation issues com.cloudera.launchpad.api.common.DeploymentsResource$InvalidDeploymentTemplateException: Validation errors: ErrorInfo{code=PROVIDER_EXCEPTION, properties={message=Image '/subscriptions/f5300b4d-bebc-4d95-8b6c-4ea77d9b3423/resourceGroups/Packer/providers/Microsoft.Compute/images/cloudera-packerbake-36.0.55' is referencing a Custom Image but not all required fields are set. To use Custom Images set the 'useCustomManagedImage' and 'customImagePlan' fields.}, causes=[]}Conditions: ValidationExceptionCondition{scope='azure.com.cloudera.director.azure.compute.provider.AzureComputeProvider.template', key='Optional.of(image)', exceptionInfo='ErrorInfo{code=PROVIDER_EXCEPTION, properties={message=Image '/subscriptions/f5300b4d-bebc-4d95-8b6c-4ea77d9b3423/resourceGroups/Packer/providers/Microsoft.Compute/images/cloudera-packerbake-36.0.55' is referencing a Custom Image but not all required fields are set. To use Custom Images set the 'useCustomManagedImage' and 'customImagePlan' fields.}, causes=[]}'} at com.cloudera.launchpad.api.common.DeploymentsResource.validate(DeploymentsResource.java:835) at com.cloudera.launchpad.api.common.DeploymentsResource.lambda$validateWithMarketplaceMode$0(DeploymentsResource.java:855) at com.cloudera.launchpad.pluggable.util.ConfigurationPropertiesUtil.validateWithMarketplaceMode(ConfigurationPropertiesUtil.java:106) at com.cloudera.launchpad.api.common.DeploymentsResource.validateWithMarketplaceMode(DeploymentsResource.java:850) at com.cloudera.launchpad.api.common.DeploymentsResource.create(DeploymentsResource.java:270) at com.cloudera.launchpad.api.common.DeploymentsResource$$FastClassBySpringCGLIB$$d7365ebd.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88) at com.cloudera.launchpad.api.common.ResourceMDCEnhancer.enhanceMDCForDeploymentApiCall(ResourceMDCEnhancer.java:179) at sun.reflect.GeneratedMethodAccessor458.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) at com.cloudera.launchpad.api.common.DeploymentsResource$$EnhancerBySpringCGLIB$$5f181472.create(<generated>) at com.cloudera.launchpad.api.d6_1.DeploymentsResourceD6_1.create(DeploymentsResourceD6_1.java:98) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:877) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:215) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at com.cloudera.launchpad.DenyTraceTrackRequestFilter.doFilter(DenyTraceTrackRequestFilter.java:32) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:158) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:126) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:111) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at net.rakugakibox.spring.boot.logback.access.LogbackAccessSecurityAttributesSaveFilter.doFilter(LogbackAccessSecurityAttributesSaveFilter.java:28) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:90) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:155) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:531) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748)
... View more
07-09-2019
07:49 AM
Still failing. Here is what I tried... common-instanceTemplate {
# Core config fields that are common to all node types
base {
type: STANDARD_DS13_V2
#image: cloudera-centos-75-latest
image: /subscriptions/f5300b4d-bebc-4d95-8b6c-4ea77d9b3423/resourceGroups/Packer/providers/Microsoft.Compute/images/cloudera-packerbake-36.0.55
useCustomManagedImage: Yes
customImagePlan: ""
...
...
...
master-base {
computeResourceGroup: ${customer-params.resourceGroup}
availabilitySet: ${customer-params.masterAvailabilitySet}
storageAccountType: "Premium_LRS"
dataDiskSize: 512
managedDisks: Yes
useCustomManagedImage: ${?common-instanceTemplate.base.useCustomManagedImage}
customImagePlan: ${?common-instanceTemplate.base.customImagePlan}
publicIP: Yes
...
...
...
# Config fields for master-1 nodes
master-1 {
dataDiskCount: 1
instanceNamePrefix: master-1
useCustomManagedImage: ${?common-instanceTemplate.base.useCustomManagedImage}
customImagePlan: ${?common-instanceTemplate.base.customImagePlan}
publicIP: ${?common-instanceTemplate.master-base.publicIP}
} I repeated this for each master instance, worker base, worker instance, and manager sections. Director error: ValidationExceptionCondition{scope='azure.com.cloudera.director.azure.compute.provider.AzureComputeProvider.template', key='Optional.of(image)', exceptionInfo='ErrorInfo{code=PROVIDER_EXCEPTION, properties={message=Image '/subscriptions/f5300b4d-bebc-4d95-8b6c-4ea77d9b3423/resourceGroups/Packer/providers/Microsoft.Compute/images/cloudera-packerbake-36.0.55' is referencing a Custom Image but not all required fields are set. To use Custom Images set the 'useCustomManagedImage' and 'customImagePlan' fields.}, causes=[]}'}
... View more
07-09-2019
07:39 AM
Thank you for the reply Bill. You have confirmed my hunch. I'm giving this a try now. Will let you know.
... View more