Hi,   I'm trying to reindex some PDF and Word documents using MapReduceIndexerTool but getting just "connection refused" (see complete error message below)   This used to work earlier, so the Morphline and the collection are fine. Using CDH 5.0.0 Quickstart VM. Used dry-run and verbose options, but no additional info available. SolrCloud logs do not show anything suspicious either. Maybe zookeeper got stuck?   Any ideas?   Regards, Slavo     $ hadoop jar /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/solr/contrib/mr/search-mr-1.0.0-cdh5.0.eIndexerTool -Dmapred.child.java.opts=-Xmx2G --log4j log4j.properties --morphline-file morphline.tamapdox.conf --output-dir hdfs://tamapdox.data.output --mappers 1 --reducers 1 --go-live --update-conflict-resolver org.apache.solr.hadoop.dedup.RetainMostRecentU03:2181/solr --verbose --collection tamapdox hdfs://mfhadoopt03:8020/user/mftamap/solr/tamap/tamapdox.data2 0    [main] INFO  org.apache.solr.common.cloud.ConnectionManager  – Waiting for client to connect to ZooKeeper 16   [main-EventThread] INFO  org.apache.solr.common.cloud.ConnectionManager  – Watcher org.apache.solr.common.cloud.ConnectionMantcher:mfhadoopt03:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None 17   [main] INFO  org.apache.solr.common.cloud.ConnectionManager  – Client is connected to ZooKeeper 29   [main] INFO  org.apache.solr.common.cloud.ZkStateReader  – Updating cluster state from ZooKeeper... 1210745 [main] WARN  org.apache.hadoop.security.UserGroupInformation  – PriviledgedActionException as:mftamap (auth:SIMPLE) cause:java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to localhost.localdomain:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused Exception in thread "main" java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to localhost.localdomain:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused         at sun.reflect.GeneratedConstructorAccessor3.newInstance(Unknown Source)         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)         at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)         at org.apache.hadoop.ipc.Client.call(Client.java:1413)         at org.apache.hadoop.ipc.Client.call(Client.java:1362)         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)         at com.sun.proxy.$Proxy10.getClusterMetrics(Unknown Source)         at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:157)         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:606)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)         at com.sun.proxy.$Proxy11.getClusterMetrics(Unknown Source)         at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:295)         at org.apache.hadoop.mapred.ResourceMgrDelegate.getClusterMetrics(ResourceMgrDelegate.java:139)         at org.apache.hadoop.mapred.YARNRunner.getClusterMetrics(YARNRunner.java:178)         at org.apache.hadoop.mapreduce.Cluster.getClusterStatus(Cluster.java:246)         at org.apache.hadoop.mapred.JobClient$3.run(JobClient.java:719)         at org.apache.hadoop.mapred.JobClient$3.run(JobClient.java:717)         at java.security.AccessController.doPrivileged(Native Method)         at javax.security.auth.Subject.doAs(Subject.java:415)         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)         at org.apache.hadoop.mapred.JobClient.getClusterStatus(JobClient.java:717)         at org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:670)         at org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:633)         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)         at org.apache.solr.hadoop.MapReduceIndexerTool.main(MapReduceIndexerTool.java:620)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:606)         at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: java.net.ConnectException: Connection refused         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)         at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)         at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)         at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)         at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:604)         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:699)         at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)         at org.apache.hadoop.ipc.Client.getConnection(Client.java:1461)         at org.apache.hadoop.ipc.Client.call(Client.java:1380)         ... 29 more ... View more

Hi, We are running CDH 5.4.5 with Kerberos and OpenLDAP. While integration of OpenLDAP into Hue went smoothly, I'm struggling with configuring external authentication in Cloudera Manager: /var/log/cloudera-scm-server/cloudera-scm-server.log shows issues in LDAP/AD authentication (we have a fallback to database active) scm-web-107:com.cloudera.server.web.cmf.CmfLdapAuthenticationProvider: LDAP/AD authentication failure for USERNAME The CM external auth config is like follows:   Authentication Backend Order - External then Database External Authentication Type - LDAP LDAP URL - ldap://hostname:389 (the same as in Hue) LDAP Bind User Distinguished Name - cn=ourproxyuser,ou=users,dc=mycompany,dc=com LDAP Bind Password - (*****) LDAP User Search Filter - (uid={0}) LDAP User Search Base - OU=users,dc=mycompany,dc=com LDAP Group Search Filter - (memberUID={1}) LDAP Group Search Base - OU=groups,dc=mycompany,dc=com LDAP Distinguished Name Pattern  - cn={0},OU=users,DC=mycompany,dc=com   all other attributes are empty (or default)   From what I can see in slapd.log   # tail -f /var/log/slapd.log Sep 21 18:09:36 hostname slapd[37748]: slap_listener_activate(7): Sep 21 18:09:36 hostname slapd[37748]: >>> slap_listener(ldap:///) Sep 21 18:09:36 hostname slapd[37748]: connection_get(50): got connid=1036 Sep 21 18:09:36 hostname slapd[37748]: connection_read(50): checking for input on id=1036 Sep 21 18:09:36 hostname slapd[37748]: op tag 0x60, time 1442848176 Sep 21 18:09:36 hostname slapd[37748]: conn=1036 op=0 do_bind Sep 21 18:09:36 hostname slapd[37748]: >>> dnPrettyNormal: <cn=USERNAME,ou=users,dc=mycompany,dc=com> Sep 21 18:09:36 hostname slapd[37748]: <<< dnPrettyNormal: <cn=USERNAME,ou=users,dc=mycompany,dc=com>, <cn=USERNAME,ou=users,dc=mycompany,dc=com> Sep 21 18:09:36 hostname slapd[37748]: do_bind: version=3 dn="cn=USERNAME,ou=users,dc=mycompany,dc=com" method=128 Sep 21 18:09:36 hostname slapd[37748]: bdb_dn2entry("cn=USERNAME,ou=users,dc=mycompany,dc=com") Sep 21 18:09:36 hostname slapd[37748]: do_bind: v3 bind: "cn=USERNAME,ou=users,dc=mycompany,dc=com" to "cn=USERNAME,ou=users,dc=mycompany,dc=com" Sep 21 18:09:36 hostname slapd[37748]: send_ldap_result: conn=1036 op=0 p=3 Sep 21 18:09:36 hostname slapd[37748]: send_ldap_response: msgid=1 tag=97 err=0 Sep 21 18:09:36 hostname slapd[37748]: connection_get(50): got connid=1036 Sep 21 18:09:36 hostname slapd[37748]: connection_read(50): checking for input on id=1036 Sep 21 18:09:36 hostname slapd[37748]: op tag 0x63, time 1442848176 Sep 21 18:09:36 hostname slapd[37748]: conn=1036 op=1 do_search Sep 21 18:09:36 hostname slapd[37748]: >>> dnPrettyNormal: <cn=USERNAME,ou=users,dc=mycompany,dc=com> Sep 21 18:09:36 hostname slapd[37748]: <<< dnPrettyNormal: <cn=USERNAME,ou=users,dc=mycompany,dc=com>, <cn=USERNAME,ou=users,dc=mycompany,dc=com> Sep 21 18:09:36 hostname slapd[37748]: => get_ctrls Sep 21 18:09:36 hostname slapd[37748]: => get_ctrls: oid="2.16.840.1.113730.3.4.2" (noncritical) Sep 21 18:09:36 hostname slapd[37748]: <= get_ctrls: n=1 rc=0 err="" Sep 21 18:09:36 hostname slapd[37748]: ==> limits_get: conn=1036 op=1 self="cn=USERNAME,ou=users,dc=mycompany,dc=com" this="cn=USERNAME,ou=users,dc=mycompany,dc=com" Sep 21 18:09:36 hostname slapd[37748]: => bdb_search Sep 21 18:09:36 hostname slapd[37748]: bdb_dn2entry("cn=USERNAME,ou=users,dc=mycompany,dc=com") Sep 21 18:09:36 hostname slapd[37748]: send_ldap_result: conn=1036 op=1 p=3 Sep 21 18:09:36 hostname slapd[37748]: send_ldap_response: msgid=2 tag=101 err=32 Sep 21 18:09:36 hostname slapd[37748]: connection_get(50): got connid=1036 Sep 21 18:09:36 hostname slapd[37748]: connection_read(50): checking for input on id=1036 Sep 21 18:09:36 hostname slapd[37748]: op tag 0x42, time 1442848176 Sep 21 18:09:36 hostname slapd[37748]: ber_get_next on fd 50 failed errno=0 (Success) Sep 21 18:09:36 hostname slapd[37748]: conn=1036 op=2 do_unbind Sep 21 18:09:36 hostname slapd[37748]: connection_close: conn=1036 sd=50   there is an error "send_ldap_response: msgid=2 tag=101 err=32", but I just don't know why. How can I investigate this further?   Thanks in advance,   Regards, Slavo ... View more

Sqoop2 was installed, up and running before CMS upgrade to 5.4.6 (from 5.4.4)   Now it doesn't even start:   2015-09-14 13:54:11,480 INFO org.apache.sqoop.core.SqoopServer: Booting up Sqoop server 2015-09-14 13:54:11,489 INFO org.apache.sqoop.core.PropertiesConfigurationProvider: Starting config file poller thread 2015-09-14 13:54:11,498 ERROR org.apache.sqoop.core.SqoopServer: Server startup failure java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.sqoop.security.authentication.KerberosAuthenticationHandler         at org.apache.sqoop.utils.ClassUtils.loadClass(ClassUtils.java:65)         at org.apache.sqoop.security.SecurityFactory.getAuthenticationHandler(SecurityFactory.java:31)         at org.apache.sqoop.security.AuthenticationManager.initialize(AuthenticationManager.java:97)         at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:53)         at org.apache.sqoop.server.ServerInitializer.contextInitialized(ServerInitializer.java:41)         at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210)         at org.apache.catalina.core.StandardContext.start(StandardContext.java:4709)         at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:802)         at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)         at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)         at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944)         at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779)         at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505)         at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322)         at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325)         at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)         at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1068)         at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)         at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1060)         at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)         at org.apache.catalina.core.StandardService.start(StandardService.java:525)         at org.apache.catalina.core.StandardServer.start(StandardServer.java:759)         at org.apache.catalina.startup.Catalina.start(Catalina.java:595)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:497)         at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)         at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: java.lang.ClassNotFoundException: org.apache.sqoop.security.authentication.KerberosAuthenticationHandler         at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)         at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)         at org.apache.sqoop.utils.ClassUtils.loadClass(ClassUtils.java:61)         ... 28 more 2015-09-14 13:54:11,503 ERROR org.apache.sqoop.server.ServerInitializer: Sqoop server failed to start java.lang.RuntimeException: Failure in server initialization         at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:63)         at org.apache.sqoop.server.ServerInitializer.contextInitialized(ServerInitializer.java:41)         at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210)         at org.apache.catalina.core.StandardContext.start(StandardContext.java:4709)         at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:802)         at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)         at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)         at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944)         at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779)         at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505)         at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322)         at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325)         at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)         at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1068)         at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)         at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1060)         at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)         at org.apache.catalina.core.StandardService.start(StandardService.java:525)         at org.apache.catalina.core.StandardServer.start(StandardServer.java:759)         at org.apache.catalina.startup.Catalina.start(Catalina.java:595)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:497)         at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)         at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.sqoop.security.authentication.KerberosAuthenticationHandler         at org.apache.sqoop.utils.ClassUtils.loadClass(ClassUtils.java:65)         at org.apache.sqoop.security.SecurityFactory.getAuthenticationHandler(SecurityFactory.java:31)         at org.apache.sqoop.security.AuthenticationManager.initialize(AuthenticationManager.java:97)         at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:53)         ... 25 more Caused by: java.lang.ClassNotFoundException: org.apache.sqoop.security.authentication.KerberosAuthenticationHandler         at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)         at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)         at org.apache.sqoop.utils.ClassUtils.loadClass(ClassUtils.java:61)         ... 28 more   Any ideas? ... View more

Hi,    I want to migrate the Hue database from sqlite3 to MySQL and dump the data from sqlite to a JSON file:   logged on as user "hue" (who is the "owner" of Hue)   $ /opt/cloudera/parcels/CDH-5.4.4-1.cdh5.4.4.p0.4/lib/hue/build/env/bin/hue dumpdata >hue_dumpdata.2015-07-20.json CommandError: Unable to serialize database: no such table: desktop_document2   When I use sqlite to display the list of tables, I can see it there:   $ sqlite3 /var/hadoop/data/hue/desktop.db SQLite version 3.6.20 Enter ".help" for instructions Enter SQL statements terminated with a ";" sqlite> .tables '%'; auth_group                      oozie_datainput auth_group_permissions          oozie_dataoutput ... desktop_document                oozie_generic desktop_document2               oozie_history desktop_document2_dependencies  oozie_hive ...   What's wrong with this?   Thanks in advance.   Regards, Slavo ... View more

I get the following error in Solr log:   org.apache.solr.core.CoreContainer Unable to create core: pdfcollection_shard1_replica1 org.apache.solr.common.SolrException: Error loading class 'solr.clustering.ClusteringComponent' at org.apache.solr.core.SolrCore.<init>(SolrCore.java:841) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:631) at org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:273) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:671) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:368) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:360) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.apache.solr.common.SolrException: Error loading class 'solr.clustering.ClusteringComponent' at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:449) at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:381) at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:532) at org.apache.solr.core.SolrCore.createInitInstance(SolrCore.java:588) at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2215) at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2209) at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2242) at org.apache.solr.core.SolrCore.loadSearchComponents(SolrCore.java:1261) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:768) ... 11 more Caused by: java.lang.ClassNotFoundException: solr.clustering.ClusteringComponent at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:789) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:270) at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:433) ... 19 more   I changed the Solr configuration to use -Dsolr.clustering.enabled=true in Java Configuration Options forSolrServer (this gets applied to $CATALINA_OPT)   and copied the clustering jar file plus contribs from the parcels to my local lib cp/opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/solr/solr-clustering-4.4.0-cdh5.0.0.jar lib cp -R /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/share/doc/solr-doc-4.4.0+cdh5.0.0+178/solr-clustering/org/apache/solr/handler/clustering/* lib   then used full path in solrconfig.xml to point to the jar file. <libpath="/home/cloudera/solr/lib/solr-clustering-4.4.0-cdh5.0.0.jar" />   and restarted solr using CM.     ... View more