Member since
11-04-2016
74
Posts
16
Kudos Received
7
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3326 | 02-28-2019 03:22 AM | |
2944 | 02-01-2019 01:15 AM | |
4168 | 04-16-2018 03:38 AM | |
32649 | 09-16-2017 04:36 AM | |
9119 | 09-11-2017 02:43 PM |
11-19-2018
04:05 AM
Hi @Harsh J You mentioning the Safe Valve gave me an idea! I thought maybe the UI in CM is not setting one or both of those key/values. So I did this manually and it worked! Now every container asked by Spark Pipe() has the same owner as the Spark application itself (no more nobody or yarn! - there must be something with the UI that won't map one of those two configs back to yarn-site.xml):
... View more
11-19-2018
02:19 AM
Hi @Harsh J Thanks for your response. Yes, these are the two configs you mentioned, and also I checked all the "safety valves" there is nothing related to any Linux or cgroups: I have even remove the "nobody" user from allowed and left the "nonsecure-mode.lcaol-user" empty, but still says "nobody". If I revert all the changes, it says "yarn". So these configs impact something somewhere. Cloudera Express: 5.15.1 Java Version: 1.8.0_181 CDH: 5.15.1-1.cdh5.15.1.p0.4 UPDATE: one more thing that might be useful, when I download client configuration from CM, I can't find these two configs in any of the configs. Not sure if that is normal. Best, Maziyar
... View more
11-18-2018
04:58 PM
Hi, I have followed the instruction to allow Linux user to run the container as the user which lunch the application, but still, I can see the actual user is nobody. Even though I can see this: main : run as user is nobody
main : requested yarn user is panahi I have OpenLDAP sync users and groups in all the nodes across the cluster. My only problem is the yarn containers are launched either by yarn in a non-secure cluster with default values, or nobody when you change "Limit Nonsecure Container Executor Users" to false, and "yarn.nodemanager.container-executor.class" to true. Despite the fact Spark runs with the user which runs it, the result of this snippet which Spark calls another application is always either yarn or nobody in any situation: val test = sc.parallelize(Seq("test user")).repartition(1)
val piped = test.pipe(Seq("whoami"))
val c = piped.collect()
... View more
11-18-2018
04:48 PM
Have you found any way to run YARN container as the user who'is launched it? I also have set these two and have all the nodes sync with LDAP, still runs as nobody despite the fact I can see it says the yarn user request is ...
... View more
11-04-2018
03:51 AM
2 Kudos
Hi, Is there any plan or roadmap for releasing Cloudera Express 6.0 or CM 6.x and CDH 6.x are only for the enterprise edition? https://www.cloudera.com/documentation/enterprise/latest/topics/cm_faqs.html#topic_1_0__section_nrr_wwv_ls Many thanks,
... View more
Labels:
- Labels:
-
Cloudera Manager
06-20-2018
01:55 PM
As you mentioned correctly Apache Spark is offering MlLib (or ML) which it comes with a set of features for some basic NLP, most popular algorithms for clustering and classifications, etc. But that is not all! You can use many libraries which are released to complete Spark in a domain of Machine Learning and Deep Learning. Basically, these libraries are using Spark APIs and Engine. You can have a look here (or other lists): https://github.com/awesome-spark/awesome-spark#machine-learning-extension
... View more
04-16-2018
03:38 AM
I followed the instruction of downgrading back to 5.14.1 on Ubuntu from here: https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cm_ag_downgrade_to_cm4.html The server started without a problem, I upgraded CM to 5.14.2 on all the masters again and on the Web console, I upgraded agents to 5.14.2 on all the machines. I am guessing an interruption caused by the unexpected shutdown or a leftover of hard-coded apt pointing to 5.14.1 may have caused this. But now everything has been resolved. Thank you.
... View more
04-15-2018
06:42 AM
As an update I checked `/etc/cloudera-scm-server/db.properties` and it is correct and I can connect to the scm database without a problem with username/password.
... View more
04-15-2018
03:31 AM
Hi, I upgraded my CDH from 5.14.1 to 5.14.2 and now I can't start cloudera-scm-server. It gives me this error: 2018-04-15 12:23:44,501 INFO main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: MYSQL database engine and table mapping: {InnoDB=[AUDITS, CLIENT_CONFIGS, EXTERNAL_ACCOUNTS_AUD, PROCESSES_DETAIL, RELEASES, RELEASES_AUD, SERVICES, USER_SETTINGS, EXTERNAL_ACCOUNTS, HOSTS_AUD, HOST_TEMPLATE_TO_ROLE_CONF_GRP, DIAGNOSTICS_EVENTS, CM_PEERS, CLUSTER_ACTIVATED_RELEASES, SCHEMA_VERSION, CREDENTIALS, ROLES_AUD, PARCEL_COMPONENTS, ROLE_CONFIG_GROUPS, ROLE_STALENESS_STATUS, CLUSTERS_AUD, SERVICES_AUD, COMMAND_SCHEDULES, CONFIGS_AUD, SNAPSHOT_POLICIES, REVISIONS, CLUSTER_MANAGED_RELEASES, CONFIG_CONTAINERS, GLOBAL_SETTINGS, METRICS, HOST_TEMPLATES, PARCELS, PROCESSES, COMMANDS_DETAIL, CLUSTER_UNDISTRIBUTED_RELEASES, CONFIGS, USER_ROLES, ROLE_CONFIG_GROUPS_AUD, HOSTS, CLIENT_CONFIGS_TO_HOSTS, COMMANDS, CLUSTERS, ROLES, CLUSTER_ACTIVATED_RELEASES_AUD, USERS, CM_VERSION, PROCESS_ACTIVE_RELEASES]}
2018-04-15 12:23:44,591 ERROR main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean:
********************************************************************************
* Cloudera Manager cannot be safely upgraded while there are pending *
* active commands. Please downgrade to the prior version 5.14.1 *
* and either let the commands run to completion or abort them. *
* There are 6 active commands of type Start *
* *
* If you must proceed without downgrade, active commands can be hard *
* stopped if you start cloudera-scm-server with 'force_start' argument, *
* e.g. 'service cloudera-scm-server force_start' *
********************************************************************************
2018-04-15 12:23:44,593 INFO main:org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@7dc0f706: defining beans [commandLineConfigurationBean,entityManagerFactoryBean,com.cloudera.server.cmf.TrialState,com.cloudera.server.cmf.TrialManager,com.cloudera.cmf.crypto.LicenseLoader]; root of factory hierarchy
2018-04-15 12:23:44,594 ERROR main:com.cloudera.server.cmf.Main: Server failed.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState': Cannot resolve reference to bean 'entityManagerFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Cloudera Manager cannot be upgraded with active commands
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)
at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:616)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:148)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1003)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:907)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
at com.cloudera.server.cmf.Main.bootstrapSpringContext(Main.java:393)
at com.cloudera.server.cmf.Main.<init>(Main.java:243)
at com.cloudera.server.cmf.Main.main(Main.java:216)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Cloudera Manager cannot be upgraded with active commands
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:149)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:102)
at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1440)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:247)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
... 17 more
Caused by: java.lang.RuntimeException: Cloudera Manager cannot be upgraded with active commands
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.failIfActiveCommandsExist(EntityManagerFactoryBean.java:229)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.ensureNoActiveCommandsDuringUpgrade(EntityManagerFactoryBean.java:190)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:124)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:65)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:142)
... 22 more I have Ubuntu 16.4 and the external database MySQL. Many thanks.
... View more
Labels:
- Labels:
-
Cloudera Manager
03-06-2018
03:11 PM
This is a good point to keep in mind when someone chooses this method since this is the only workaround for saving tables inside Hive in the older version of Cloudera. NOTE: This problem only happened in Cloudera and has been solved in a newer version (Hive/Spark). As of CDH 5.14.0 this has been completely resolved.
... View more