Cloudera Community
spin0
user rank
Contributor
My Accepted Solutions
Show All

Re: sqoop2 Server failed to start

by Contributor in Support Questions
‎02-02-2017 03:32 PM
‎02-02-2017 03:32 PM
I finally found the issue and resolved it. Note: This is with CDH-5.9.0-1 and installation was executed from Cloudera Manager (fresh installation).   Issue: After adding Sqoop2 service from Cloudera Manager, Sqoop service cannot be started.   Cause: Base on th error in Role log, there is an error that was bugging me the most. "Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. " Basically during the installation of Sqoop2 service, the script that does the installation somehow did not create the DB repository (in this case it was PostgreSQL). As you can see the error above it is obvious that was something relating to back-end.   Looking into Sqoop configuration from Clouder Manager, the PostgreSQL database name was sqoop. The DB account was sa and DB Account password was empty. It is very strange.   Solution:   1- You need to access to PostgreSQL to find out whether sqoop account and database are present which they are not. In the case, sa account. And sure enough it was not present. It rings the bell right the way. Changing Embedded PostgreSQL Database Passwords https://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_ig_embed_pstgrs.html   2- Create account (role) and database for sqoop. Configuring Sqoop 2 to Use PostgreSQLing Embedded https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cdh_ig_sqoop2_configure.html In this case  I created account: sqoop and its password. And database name is sqoop.   3- You need to upgrade sqoop database to make sure that all structures are correct. Use Cloudera Manager to do Sqoop database upgrade. Note: When you creat a new database, there is not structure for Sqoop. By doing the upgrade, the system adjusts all Sqoop tabales needed.   4- Configure new account, database, host and port using Cloudera Manager. Save it. Use Cloudera Manager to configure Sqoop database info. - Sqoop repository DB server with port (Default 7432 for PostgreSQL) Sqoop Repository Database Host: - Sqoop database name Sqoop Repository Database Name: - DB user account Sqoop Repository Database User: - DB password Sqoop Repository Database Password:   5- Start Sqoop service                   ... View more

Re: Fail to start HDFS FAILOVERCONTROLLER (FC) - s...

by Contributor in Support Questions
‎12-22-2016 02:25 PM
1 Kudo
‎12-22-2016 02:25 PM
1 Kudo
All,   The resolution to this error is to enable the HDFS HA Enabling. Thank everyone for helping it.   1- You need to pay attention to Failover Controller (FC) already exist on the nodes that you assign to be active and standby for HDFS HA. Basically remove FC from these nodes before doing the HDFS HA Enabling.   2- Have your JournalNodes Edits Directory set up. Usually it is in /var/lib/jn   Once the HDFS HA is enable, you can verify it by doing from Cloudera Manager - HDFS - Instances - Federation and High Availability <- click on it to see the setup   or   -HDFS - Configuration -<Do a search on nameservice> In filed NameNodes Nameservice, you should see all nodes that you assign in HDFS HA.       ... View more

Re: sqoop2 Server failed to start

by Contributor in Support Questions
‎12-22-2016 12:25 PM
‎12-22-2016 12:25 PM
I have additional info. regrading the error message in red (Role log). Note: This is a new installation of sqoop2 using Cloudera Manager and it uses postgresql (Instead of Derby database).   I'm not sure why this error after the installation. I take a look in the following directires and they have files for postgresql, it seems to be OK for postgresql.   r--r-- sqoop2 sqoop2 /var/lib/sqoop2/postgresql-9.0-801.jdbc4.jar   --r--  root root /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/jars/postgresql-9.0-801.jdbc4.jar -xr-x root root /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/jars/postgresql-9.1-901.jdbc4.jar --r--  root root /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/jars/post.jar   lrwxrwxrwx root root /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/sqoop2/webapps/sqoop/WEB-INF/lib/postgresql-9.1-901/../../../jars/postgresql-9.1-901.jdbc4.jar   Any helps will be appraciated.     Role log Sqoop server failed to start java.lang.RuntimeException: Failure in server initialization          at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:63)          at org.apache.sqoop.server.ServerInitializer.contextInitialized(ServerInitializer.java:41)          at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4236)          at org.apache.catalina.core.StandardContext.start(StandardContext.java:4739)          at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:803)          at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:780)          at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)          at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944)          at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779)          at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505)          at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322)          at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325)          at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)          at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1069)          at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)          at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)          at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)          at org.apache.catalina.core.StandardService.start(StandardService.java:525)          at org.apache.catalina.core.StandardServer.start(StandardServer.java:759)          at org.apache.catalina.startup.Catalina.start(Catalina.java:595)          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)          at java.lang.reflect.Method.invoke(Method.java:606)          at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)          at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: org.apache.sqoop.common.SqoopException: JDBCREPO_0007:Unable to lease link          at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:64)          at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:87)          at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:63)          at org.apache.sqoop.repository.JdbcRepository.isRepositorySuitableForUse(JdbcRepository.java:144)          at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:129)          at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:78)          at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:56)          ... 25 more Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.          at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136)          at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)          at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)          at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)          at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)          at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)          at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)          at org.postgresql.Driver.makeConnection(Driver.java:393)          at org.postgresql.Driver.connect(Driver.java:267)          at java.sql.DriverManager.getConnection(DriverManager.java:571)          at java.sql.DriverManager.getConnection(DriverManager.java:187)          at org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:78)          at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)          at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148)          at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)          at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:62)          ... 31 more Caused by: java.net.ConnectException: Connection refused          at java.net.PlainSocketImpl.socketConnect(Native Method)          at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)          at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)          at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)          at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)          at java.net.Socket.connect(Socket.java:579)          at java.net.Socket.connect(Socket.java:528)          at java.net.Socket.<init>(Socket.java:425)          at java.net.Socket.<init>(Socket.java:208)          at org.postgresql.core.PGStream.<init>(PGStream.java:62)          at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:76)         ... View more

sqoop2 Server failed to start

by Contributor in Support Questions
‎12-21-2016 03:16 PM
‎12-21-2016 03:16 PM
  I just install sqoop2 server on a node (Via Cloudera Manager) and the service could not start (failed to start). Not sure the issue. I see some key word in the stderr and stdout logs. I use CDH-5.9.0-1.    Need helps. Thank you.   stdout log Wed Dec 21 14:52:05 PST 2016 JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera using 5 as CDH_VERSION log4j:ERROR A "org.apache.log4j.RollingFileAppender" object is not assignable to a "org.apache.log4j.Appender" variable. log4j:ERROR The class "org.apache.log4j.Appender" was loaded by log4j:ERROR [org.apache.catalina.loader.StandardClassLoader@2d33d68a] whereas object of type log4j:ERROR "org.apache.log4j.RollingFileAppender" was loaded by [WebappClassLoader context: /sqoop delegate: false repositories: /WEB-INF/classes/ ----------> Parent Classloader: org.apache.catalina.loader.StandardClassLoader@2d33d68a ]. log4j:ERROR Could not instantiate appender named "RFA". log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Wed Dec 21 14:52:09 PST 2016 JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera using 5 as CDH_VERSION log4j:ERROR A "org.apache.log4j.RollingFileAppender" object is not assignable to a "org.apache.log4j.Appender" variable. log4j:ERROR The class "org.apache.log4j.Appender" was loaded by log4j:ERROR [org.apache.catalina.loader.StandardClassLoader@3e4523d6] whereas object of type log4j:ERROR "org.apache.log4j.RollingFileAppender" was loaded by [WebappClassLoader context: /sqoop delegate: false repositories: /WEB-INF/classes/ ----------> Parent Classloader: org.apache.catalina.loader.StandardClassLoader@3e4523d6 ]. log4j:ERROR Could not instantiate appender named "RFA". log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Wed Dec 21 14:52:15 PST 2016 JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera using 5 as CDH_VERSION log4j:ERROR A "org.apache.log4j.RollingFileAppender" object is not assignable to a "org.apache.log4j.Appender" variable. log4j:ERROR The class "org.apache.log4j.Appender" was loaded by log4j:ERROR [org.apache.catalina.loader.StandardClassLoader@3e4523d6] whereas object of type log4j:ERROR "org.apache.log4j.RollingFileAppender" was loaded by [WebappClassLoader context: /sqoop delegate: false repositories: /WEB-INF/classes/ ----------> Parent Classloader: org.apache.catalina.loader.StandardClassLoader@3e4523d6 ]. log4j:ERROR Could not instantiate appender named "RFA". log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Wed Dec 21 14:52:22 PST 2016 JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera using 5 as CDH_VERSION log4j:ERROR A "org.apache.log4j.RollingFileAppender" object is not assignable to a "org.apache.log4j.Appender" variable. log4j:ERROR The class "org.apache.log4j.Appender" was loaded by log4j:ERROR [org.apache.catalina.loader.StandardClassLoader@264a1535] whereas object of type log4j:ERROR "org.apache.log4j.RollingFileAppender" was loaded by [WebappClassLoader context: /sqoop delegate: false repositories: /WEB-INF/classes/ ----------> Parent Classloader: org.apache.catalina.loader.StandardClassLoader@264a1535 ]. log4j:ERROR Could not instantiate appender named "RFA". log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.     stderr log Wed Dec 21 14:52:22 PST 2016 + source_parcel_environment + '[' '!' -z /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/meta/cdh_env.sh ']' + OLD_IFS=' ' + IFS=: + SCRIPT_ARRAY=($SCM_DEFINES_SCRIPTS) + DIRNAME_ARRAY=($PARCEL_DIRNAMES) + IFS=' ' + COUNT=1 ++ seq 1 1 + for i in '`seq 1 $COUNT`' + SCRIPT=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/meta/cdh_env.sh + PARCEL_DIRNAME=CDH-5.9.0-1.cdh5.9.0.p0.23 + . /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/meta/cdh_env.sh ++ CDH_DIRNAME=CDH-5.9.0-1.cdh5.9.0.p0.23 ++ export CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop ++ CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop ++ export CDH_MR1_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-0.20-mapreduce ++ CDH_MR1_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-0.20-mapreduce ++ export CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-hdfs ++ CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-hdfs ++ export CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-httpfs ++ CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-httpfs ++ export CDH_MR2_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-mapreduce ++ CDH_MR2_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-mapreduce ++ export CDH_YARN_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-yarn ++ CDH_YARN_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-yarn ++ export CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hbase ++ CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hbase ++ export CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/zookeeper ++ CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/zookeeper ++ export CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hive ++ CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hive ++ export CDH_HUE_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hue ++ CDH_HUE_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hue ++ export CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/oozie ++ CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/oozie ++ export CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop ++ CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop ++ export CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/flume-ng ++ CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/flume-ng ++ export CDH_PIG_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/pig ++ CDH_PIG_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/pig ++ export CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hive-hcatalog ++ CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hive-hcatalog ++ export CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/sqoop2 ++ CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/sqoop2 ++ export CDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/llama ++ CDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/llama ++ export CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/sentry ++ CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/sentry ++ export TOMCAT_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-tomcat ++ TOMCAT_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-tomcat ++ export JSVC_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-utils ++ JSVC_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-utils ++ export CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop/bin/hadoop ++ CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop/bin/hadoop ++ export CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/impala ++ CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/impala ++ export CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/solr ++ CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/solr ++ export CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hbase-solr ++ CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hbase-solr ++ export SEARCH_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/search ++ SEARCH_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/search ++ export CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/spark ++ CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/spark ++ export WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/etc/hive-webhcat/conf.dist/webhcat-default.xml ++ WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/etc/hive-webhcat/conf.dist/webhcat-default.xml ++ export CDH_KMS_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-kms ++ CDH_KMS_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-kms ++ export CDH_PARQUET_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/parquet ++ CDH_PARQUET_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/parquet ++ export CDH_AVRO_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/avro ++ CDH_AVRO_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/avro + locate_cdh_java_home + '[' -z '' ']' + '[' -z /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-utils ']' + local BIGTOP_DETECT_JAVAHOME= + for candidate in '"${JSVC_HOME}"' '"${JSVC_HOME}/.."' '"/usr/lib/bigtop-utils"' '"/usr/libexec"' + '[' -e /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-utils/bigtop-detect-javahome ']' + BIGTOP_DETECT_JAVAHOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-utils/bigtop-detect-javahome + break + '[' -z /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-utils/bigtop-detect-javahome ']' + . /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-utils/bigtop-detect-javahome ++ BIGTOP_DEFAULTS_DIR=/etc/default ++ '[' -n /etc/default -a -r /etc/default/bigtop-utils ']' ++ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6') ++ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk') ++ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle') ++ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk') ++ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle') ++ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk') ++ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk') ++ case ${BIGTOP_JAVA_MAJOR} in ++ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]}) ++ '[' -z '' ']' ++ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}' +++ ls -rvd /usr/java/jdk1.7.0_67-cloudera ++ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`' ++ '[' -e /usr/java/jdk1.7.0_67-cloudera/bin/java ']' ++ export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera ++ JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera ++ break 2 + verify_java_home + '[' -z /usr/java/jdk1.7.0_67-cloudera ']' + echo JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera + set -x + echo 'using 5 as CDH_VERSION' + replace_conf_dir + find /run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER -type f '!' -path '/run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/logs/*' '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER#g' '{}' ';' Can't open /run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/supervisor.conf: Permission denied. + perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER#g' /run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/catalina_logging.properties /run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/cloudera-monitor.properties /run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/cloudera-stack-monitor.properties /run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/log4j.properties /run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/sqoop_bootstrap.properties /run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/sqoop.properties + '[' 5 -ge 5 ']' + '[' -z /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/meta/cdh_env.sh ']' + TOMCAT_CONF=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/sqoop2/../../etc/sqoop2/tomcat-conf.dist + export CATALINA_BASE=/var/lib/sqoop2/tomcat-deployment + CATALINA_BASE=/var/lib/sqoop2/tomcat-deployment + env TOMCAT_CONF=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/sqoop2/../../etc/sqoop2/tomcat-conf.dist TOMCAT_DEPLOYMENT=/var/lib/sqoop2/tomcat-deployment SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/sqoop2 bash /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/sqoop2/tomcat-deployment.sh + export CATALINA_TMPDIR=/run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/temp + CATALINA_TMPDIR=/run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/temp + export CATALINA_OUT=/var/log/sqoop2/sqoop.out + CATALINA_OUT=/var/log/sqoop2/sqoop.out + export CATALINA_PID=/run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/sqoop.pid + CATALINA_PID=/run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/sqoop.pid ++ replace_pid -Xms318767104 -Xmx318767104 -XX:+HeapDumpOnOutOfMemoryError '-XX:HeapDumpPath=/tmp/sqoop_sqoop-SQOOP_SERVER-7fbdaee63bd87efee4217fb2ae176aef_pid{{PID}}.hprof' -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh ++ echo -Xms318767104 -Xmx318767104 -XX:+HeapDumpOnOutOfMemoryError '-XX:HeapDumpPath=/tmp/sqoop_sqoop-SQOOP_SERVER-7fbdaee63bd87efee4217fb2ae176aef_pid{{PID}}.hprof' -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh ++ sed 's#{{PID}}#12585#g' + SQOOP_SERVER_OPTS='-Xms318767104 -Xmx318767104 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/sqoop_sqoop-SQOOP_SERVER-7fbdaee63bd87efee4217fb2ae176aef_pid12585.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh' + OPTIONS='-Xms318767104 -Xmx318767104 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/sqoop_sqoop-SQOOP_SERVER-7fbdaee63bd87efee4217fb2ae176aef_pid12585.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh' + OPTIONS='-Xms318767104 -Xmx318767104 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/sqoop_sqoop-SQOOP_SERVER-7fbdaee63bd87efee4217fb2ae176aef_pid12585.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -Dlog4j.configuration=file:////run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/log4j.properties' + OPTIONS='-Xms318767104 -Xmx318767104 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/sqoop_sqoop-SQOOP_SERVER-7fbdaee63bd87efee4217fb2ae176aef_pid12585.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -Dlog4j.configuration=file:////run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/log4j.properties -Dsqoop.config.dir=/run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER' + OPTIONS='-Xms318767104 -Xmx318767104 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/sqoop_sqoop-SQOOP_SERVER-7fbdaee63bd87efee4217fb2ae176aef_pid12585.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -Dlog4j.configuration=file:////run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/log4j.properties -Dsqoop.config.dir=/run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER -Djava.net.preferIPv4Stack=true' + OPTIONS='-Xms318767104 -Xmx318767104 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/sqoop_sqoop-SQOOP_SERVER-7fbdaee63bd87efee4217fb2ae176aef_pid12585.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -Dlog4j.configuration=file:////run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/log4j.properties -Dsqoop.config.dir=/run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER -Djava.net.preferIPv4Stack=true -Dsqoop.http.port=12000' + OPTIONS='-Xms318767104 -Xmx318767104 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/sqoop_sqoop-SQOOP_SERVER-7fbdaee63bd87efee4217fb2ae176aef_pid12585.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -Dlog4j.configuration=file:////run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/log4j.properties -Dsqoop.config.dir=/run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER -Djava.net.preferIPv4Stack=true -Dsqoop.http.port=12000 -Dsqoop.admin.port=8005' ++ replace_pid ++ echo ++ sed 's#{{PID}}#12585#g' + CATALINA_OPTS= + export 'CATALINA_OPTS=-Xms318767104 -Xmx318767104 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/sqoop_sqoop-SQOOP_SERVER-7fbdaee63bd87efee4217fb2ae176aef_pid12585.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -Dlog4j.configuration=file:////run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/log4j.properties -Dsqoop.config.dir=/run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER -Djava.net.preferIPv4Stack=true -Dsqoop.http.port=12000 -Dsqoop.admin.port=8005 ' + CATALINA_OPTS='-Xms318767104 -Xmx318767104 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/sqoop_sqoop-SQOOP_SERVER-7fbdaee63bd87efee4217fb2ae176aef_pid12585.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -Dlog4j.configuration=file:////run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/log4j.properties -Dsqoop.config.dir=/run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER -Djava.net.preferIPv4Stack=true -Dsqoop.http.port=12000 -Dsqoop.admin.port=8005 ' + export LOGGING_CONFIG=-Djava.util.logging.config.file=/run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/catalina_logging.properties + LOGGING_CONFIG=-Djava.util.logging.config.file=/run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/catalina_logging.properties + mkdir -p /run/cloudera-scm-agent/process/227-sqoop-SQOOP_SERVER/temp + EXT_LIBS_DIR=/var/lib/sqoop2 + cp /usr/share/java/mysql-connector-java.jar /var/lib/sqoop2 + cp /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar /var/lib/sqoop2 + cp /usr/share/java/oracle-connector-java.jar /var/lib/sqoop2 + '[' '' == upgrade -o '' == create-database ']' + exec /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-tomcat/bin/catalina.sh run Dec 21, 2016 2:52:23 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/java], exists: [false], isDirectory: [false], canRead: [false] Dec 21, 2016 2:52:24 PM org.apache.catalina.core.AprLifecycleListener init INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib Dec 21, 2016 2:52:24 PM org.apache.coyote.http11.Http11Protocol init INFO: Initializing Coyote HTTP/1.1 on http-12000 Dec 21, 2016 2:52:24 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 795 ms Dec 21, 2016 2:52:24 PM org.apache.catalina.core.StandardService start INFO: Starting service Catalina Dec 21, 2016 2:52:24 PM org.apache.catalina.core.StandardEngine start INFO: Starting Servlet Engine: Apache Tomcat/6.0.45 Dec 21, 2016 2:52:24 PM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive sqoop.war   ... View more
Labels:

Re: Fail to start HDFS FAILOVERCONTROLLER (FC) - s...

by Contributor in Support Questions
‎12-21-2016 02:23 PM
‎12-21-2016 02:23 PM
Darren,   It sounds complicated with HDFS HA.   If you have a role called "SecondaryNameNode", then that's incorrect. This is a very confusing name for the role in Hadoop. The SecondaryNameNode is only used in non-HA scenario. In HA, you have multiple (regular) NameNode roles defined.   With the comment above, it rings my bell now the concept Primary and Secondary for non-HA and it is was in old version. Thank for reminding me this concept.   We verified the current setup in HDFS - Configuration - History and Rollback, we did not see anything or changes related to HDFS HA. Only one thing is JournalNodes Edits Directory: /var/lib/jn   We think it is Ok now to enable HDFS HA Enabling option.   We will follow the link to enable HDFS HA using Cloudera Manager  http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_hag_hdfs_ha_enabling.html  ... View more

Re: Fail to start HDFS FAILOVERCONTROLLER (FC) - s...

by Contributor in Support Questions
‎12-21-2016 01:38 PM
‎12-21-2016 01:38 PM
  We did the HDFS HA Enabling before and it leads to this troubleshooting.   We did verify the following and its currenyt roles: NameNode (supposed to be Active Node) -> Roles: FC, JN, NN etc. NameNode Secondary (supposed to be Standby Node) -> Roles: FC, JN, NM etc. Third Node -> JN   And JournalNode Edits Directory: /var/lib/jn (drwx------ hdfs hadoop)   In term of requirements, it is OK I beleive.   When we verify HDFS - Configuration - Section nameservice Here is what we found: NameNode Nameservice: <No Value> Mount Points:           / SecondaryNameNode Nameservice: <No Value>   Questions 1- Does it mean that HDFS HA was not set up correctly by system (Scripts during the HDFS HA enabling)? 2- Does it mean that we do not have any HDFS HA set up since there is no name service? 3- By doing again HDFS HA Enabling, will it cause any duplication of nameservice?   Thanks.       ... View more

Re: Fail to start HDFS FAILOVERCONTROLLER (FC) - s...

by Contributor in Support Questions
‎12-20-2016 03:41 PM
‎12-20-2016 03:41 PM
Note : Error log is from the NameNode (Primary) where Failover Controller fails to start.  We install Fialover Controller on NameNode (Primary) and NamNode(Secondary).   What does the end of the stderr log say?   The end of stderr log: + '[' refresh-datanode = zkfc ']' + '[' mkdir = zkfc ']' + '[' nfs3 = zkfc ']' + '[' namenode = zkfc -o secondarynamenode = zkfc -o datanode = zkfc ']' + exec /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-hdfs/bin/hdfs --config /run/cloudera-scm-agent/process/184-hdfs-FAILOVERCONTROLLER zkfc Exception in thread "main" org.apache.hadoop.HadoopIllegalArgumentException: HA is not enabled for this namenode. at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.create(DFSZKFailoverController.java:130) at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:186)   Did you check the role logs for your FC for a relevant error message? I do not see anything particular. You probably are familiar with it. It is new to me so I could miss something. Here is the whole stderr log.   Tue Dec 20 14:41:57 PST 2016 + source_parcel_environment + '[' '!' -z /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/meta/cdh_env.sh ']' + OLD_IFS=' ' + IFS=: + SCRIPT_ARRAY=($SCM_DEFINES_SCRIPTS) + DIRNAME_ARRAY=($PARCEL_DIRNAMES) + IFS=' ' + COUNT=1 ++ seq 1 1 + for i in '`seq 1 $COUNT`' + SCRIPT=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/meta/cdh_env.sh + PARCEL_DIRNAME=CDH-5.9.0-1.cdh5.9.0.p0.23 + . /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/meta/cdh_env.sh ++ CDH_DIRNAME=CDH-5.9.0-1.cdh5.9.0.p0.23 ++ export CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop ++ CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop ++ export CDH_MR1_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-0.20-mapreduce ++ CDH_MR1_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-0.20-mapreduce ++ export CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-hdfs ++ CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-hdfs ++ export CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-httpfs ++ CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-httpfs ++ export CDH_MR2_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-mapreduce ++ CDH_MR2_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-mapreduce ++ export CDH_YARN_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-yarn ++ CDH_YARN_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-yarn ++ export CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hbase ++ CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hbase ++ export CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/zookeeper ++ CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/zookeeper ++ export CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hive ++ CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hive ++ export CDH_HUE_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hue ++ CDH_HUE_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hue ++ export CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/oozie ++ CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/oozie ++ export CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop ++ CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop ++ export CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/flume-ng ++ CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/flume-ng ++ export CDH_PIG_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/pig ++ CDH_PIG_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/pig ++ export CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hive-hcatalog ++ CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hive-hcatalog ++ export CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/sqoop2 ++ CDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/sqoop2 ++ export CDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/llama ++ CDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/llama ++ export CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/sentry ++ CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/sentry ++ export TOMCAT_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-tomcat ++ TOMCAT_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-tomcat ++ export JSVC_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-utils ++ JSVC_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-utils ++ export CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop/bin/hadoop ++ CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop/bin/hadoop ++ export CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/impala ++ CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/impala ++ export CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/solr ++ CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/solr ++ export CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hbase-solr ++ CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hbase-solr ++ export SEARCH_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/search ++ SEARCH_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/search ++ export CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/spark ++ CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/spark ++ export WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/etc/hive-webhcat/conf.dist/webhcat-default.xml ++ WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/etc/hive-webhcat/conf.dist/webhcat-default.xml ++ export CDH_KMS_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-kms ++ CDH_KMS_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-kms ++ export CDH_PARQUET_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/parquet ++ CDH_PARQUET_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/parquet ++ export CDH_AVRO_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/avro ++ CDH_AVRO_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/avro + locate_cdh_java_home + '[' -z '' ']' + '[' -z /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-utils ']' + local BIGTOP_DETECT_JAVAHOME= + for candidate in '"${JSVC_HOME}"' '"${JSVC_HOME}/.."' '"/usr/lib/bigtop-utils"' '"/usr/libexec"' + '[' -e /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-utils/bigtop-detect-javahome ']' + BIGTOP_DETECT_JAVAHOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-utils/bigtop-detect-javahome + break + '[' -z /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-utils/bigtop-detect-javahome ']' + . /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/bigtop-utils/bigtop-detect-javahome ++ BIGTOP_DEFAULTS_DIR=/etc/default ++ '[' -n /etc/default -a -r /etc/default/bigtop-utils ']' ++ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6') ++ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk') ++ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle') ++ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk') ++ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle') ++ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk') ++ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk') ++ case ${BIGTOP_JAVA_MAJOR} in ++ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]}) ++ '[' -z '' ']' ++ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}' +++ ls -rvd /usr/java/jdk1.7.0_67-cloudera ++ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`' ++ '[' -e /usr/java/jdk1.7.0_67-cloudera/bin/java ']' ++ export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera ++ JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera ++ break 2 + verify_java_home + '[' -z /usr/java/jdk1.7.0_67-cloudera ']' + echo JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera + . /usr/lib64/cmf/service/common/cdh-default-hadoop ++ [[ -z 5 ]] ++ '[' 5 = 3 ']' ++ '[' 5 = -3 ']' ++ '[' 5 -ge 4 ']' ++ export HADOOP_HOME_WARN_SUPPRESS=true ++ HADOOP_HOME_WARN_SUPPRESS=true ++ export HADOOP_PREFIX=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop ++ HADOOP_PREFIX=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop ++ export HADOOP_LIBEXEC_DIR=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop/libexec ++ HADOOP_LIBEXEC_DIR=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop/libexec ++ export HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/184-hdfs-FAILOVERCONTROLLER ++ HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/184-hdfs-FAILOVERCONTROLLER ++ export HADOOP_COMMON_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop ++ HADOOP_COMMON_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop ++ export HADOOP_HDFS_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-hdfs ++ HADOOP_HDFS_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-hdfs ++ export HADOOP_MAPRED_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-mapreduce ++ HADOOP_MAPRED_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-mapreduce ++ '[' 5 = 4 ']' ++ '[' 5 = 5 ']' ++ export HADOOP_YARN_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-yarn ++ HADOOP_YARN_HOME=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-yarn ++ replace_pid ++ echo ++ sed 's#{{PID}}#64392#g' + export HADOOP_NAMENODE_OPTS= + HADOOP_NAMENODE_OPTS= ++ replace_pid ++ echo ++ sed 's#{{PID}}#64392#g' + export HADOOP_DATANODE_OPTS= + HADOOP_DATANODE_OPTS= ++ replace_pid ++ echo ++ sed 's#{{PID}}#64392#g' + export HADOOP_SECONDARYNAMENODE_OPTS= + HADOOP_SECONDARYNAMENODE_OPTS= ++ replace_pid ++ echo ++ sed 's#{{PID}}#64392#g' + export HADOOP_NFS3_OPTS= + HADOOP_NFS3_OPTS= ++ replace_pid ++ echo ++ sed 's#{{PID}}#64392#g' + export HADOOP_JOURNALNODE_OPTS= + HADOOP_JOURNALNODE_OPTS= + '[' 5 -ge 4 ']' + HDFS_BIN=/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-hdfs/bin/hdfs + export 'HADOOP_OPTS=-Djava.net.preferIPv4Stack=true ' + HADOOP_OPTS='-Djava.net.preferIPv4Stack=true ' + echo 'using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME' + echo 'using 5 as CDH_VERSION' + echo 'using /run/cloudera-scm-agent/process/184-hdfs-FAILOVERCONTROLLER as CONF_DIR' + echo 'using as SECURE_USER' + echo 'using as SECURE_GROUP' + set_hadoop_classpath + set_classpath_in_var HADOOP_CLASSPATH + '[' -z HADOOP_CLASSPATH ']' + [[ -n /usr/share/cmf ]] ++ find /usr/share/cmf/lib/plugins -maxdepth 1 -name '*.jar' ++ tr '\n' : + ADD_TO_CP=/usr/share/cmf/lib/plugins/event-publish-5.9.0-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-5.9.0.jar: + [[ -n navigator/cdh57 ]] + for DIR in '$CM_ADD_TO_CP_DIRS' ++ find /usr/share/cmf/lib/plugins/navigator/cdh57 -maxdepth 1 -name '*.jar' ++ tr '\n' : + PLUGIN=/usr/share/cmf/lib/plugins/navigator/cdh57/audit-plugin-cdh57-2.8.0-shaded.jar: + ADD_TO_CP=/usr/share/cmf/lib/plugins/event-publish-5.9.0-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-5.9.0.jar:/usr/share/cmf/lib/plugins/navigator/cdh57/audit-plugin-cdh57-2.8.0-shaded.jar: + eval 'OLD_VALUE=$HADOOP_CLASSPATH' ++ OLD_VALUE= + NEW_VALUE=/usr/share/cmf/lib/plugins/event-publish-5.9.0-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-5.9.0.jar:/usr/share/cmf/lib/plugins/navigator/cdh57/audit-plugin-cdh57-2.8.0-shaded.jar: + export HADOOP_CLASSPATH=/usr/share/cmf/lib/plugins/event-publish-5.9.0-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-5.9.0.jar:/usr/share/cmf/lib/plugins/navigator/cdh57/audit-plugin-cdh57-2.8.0-shaded.jar + HADOOP_CLASSPATH=/usr/share/cmf/lib/plugins/event-publish-5.9.0-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-5.9.0.jar:/usr/share/cmf/lib/plugins/navigator/cdh57/audit-plugin-cdh57-2.8.0-shaded.jar + set -x + replace_conf_dir + find /run/cloudera-scm-agent/process/184-hdfs-FAILOVERCONTROLLER -type f '!' -path '/run/cloudera-scm-agent/process/184-hdfs-FAILOVERCONTROLLER/logs/*' '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/184-hdfs-FAILOVERCONTROLLER#g' '{}' ';' Can't open /run/cloudera-scm-agent/process/184-hdfs-FAILOVERCONTROLLER/supervisor.conf: Permission denied. + make_scripts_executable + find /run/cloudera-scm-agent/process/184-hdfs-FAILOVERCONTROLLER -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';' + '[' DATANODE_MAX_LOCKED_MEMORY '!=' '' ']' + ulimit -l + export HADOOP_IDENT_STRING=hdfs + HADOOP_IDENT_STRING=hdfs + '[' -n '' ']' + '[' mkdir '!=' zkfc ']' + acquire_kerberos_tgt hdfs.keytab + '[' -z hdfs.keytab ']' + '[' -n '' ']' + '[' validate-writable-empty-dirs = zkfc ']' + '[' file-operation = zkfc ']' + '[' bootstrap = zkfc ']' + '[' failover = zkfc ']' + '[' transition-to-active = zkfc ']' + '[' initializeSharedEdits = zkfc ']' + '[' initialize-znode = zkfc ']' + '[' format-namenode = zkfc ']' + '[' monitor-decommission = zkfc ']' + '[' jnSyncWait = zkfc ']' + '[' nnRpcWait = zkfc ']' + '[' -safemode = '' -a get = '' ']' + '[' monitor-upgrade = zkfc ']' + '[' finalize-upgrade = zkfc ']' + '[' rolling-upgrade-prepare = zkfc ']' + '[' rolling-upgrade-finalize = zkfc ']' + '[' nnDnLiveWait = zkfc ']' + '[' refresh-datanode = zkfc ']' + '[' mkdir = zkfc ']' + '[' nfs3 = zkfc ']' + '[' namenode = zkfc -o secondarynamenode = zkfc -o datanode = zkfc ']' + exec /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/hadoop-hdfs/bin/hdfs --config /run/cloudera-scm-agent/process/184-hdfs-FAILOVERCONTROLLER zkfc Exception in thread "main" org.apache.hadoop.HadoopIllegalArgumentException: HA is not enabled for this namenode. at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.create(DFSZKFailoverController.java:130) at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:186)   ... View more

Re: Fail to start HDFS FAILOVERCONTROLLER (FC) - s...

by Contributor in Support Questions
‎12-20-2016 02:16 PM
‎12-20-2016 02:16 PM
bgoogley,   Have you configured the agent to run as a user other than root?   When installation took place,we created a Hadoop account that we use (with sudo) to install Cloudera Manager. So I'm not sure where to check it.   The account used to install Cloudera Manager hduser is belong to hduser and wheel groups.     What are the file permissions listed in your FAILOVERCONTROLLER dir?   For this location /run/cloudera-scm-agent/process/152-hdfs-FAILOVERCONTROLLER/supervisor.conf Here is the permissions: drwxr-xr-x root root run drwxr-xr-x cloudera-scm cloudera-scm cloudera-scm-agent drwxr-x--x root root process drwxr-x--x hdfs hdfs 152-hdfs-FAILOVERCONTROLLER -rw-------- root root supervisor.conf   other files in that directory -rw-r------ hdfs hdfs <Any files>     Note: I also check Cloudera SCM Agent on all nodes (NameNode [Pramary, Secondary], DataNode etc.) and it is active on all nodes. #sudo service cloudera-scm-agent status.         ... View more

Fail to start HDFS FAILOVERCONTROLLER (FC) - super...

by Contributor in Support Questions
‎12-20-2016 10:26 AM
‎12-20-2016 10:26 AM
I have CDH 5.9.0-1 set up and it works well (Done through Cloudera Manager). I try to enable Failover Controller (FC) in HDFS. After the installation I got error (Error Log) on the NameNode (Primary) with   + replace_conf_dir + find /run/cloudera-scm-agent/process/152-hdfs-FAILOVERCONTROLLER -type f '!' -path '/run/cloudera-scm-agent/process/152-hdfs-FAILOVERCONTROLLER/logs/*' '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/run/cloudera-scm-agent/process/152-hdfs-FAILOVERCONTROLLER#g' '{}' ';' Can't open /run/cloudera-scm-agent/process/152-hdfs-FAILOVERCONTROLLER/supervisor.conf: Permission denied. + make_scripts_executable + find /run/cloudera-scm-agent/process/152-hdfs-FAILOVERCONTROLLER -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';' + '[' DATANODE_MAX_LOCKED_MEMORY '!=' '' ']'   I'm not sure what is the issue other than the permissions. This supervisor.conf is owned by root. Each time I restart the FC, a new process number is introduced. So even I change permmissions to allow everyone to Read, the new process is owned by root only.   I consult this link but not sure how to proceed. http://community.cloudera.com/t5/Cloudera-Manager-Installation/Problem-with-cloudera-scm-agent-and-supervisord/m-p/47045#M8572   Any help will be appraciated.     ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎03-13-2017 07:33 PM
Community Statistics
Member Since ‎04-29-2014 08:59 PM
Last Visited ‎03-13-2017 07:33 PM
Posts 12
Kudos received 1