Cloudera Community
sam_001
user rank
Explorer
My Accepted Solutions
Show All

Logs on primary node getting filled by Kafka warni...

by Explorer in Support Questions
‎01-20-2022 03:54 AM
‎01-20-2022 03:54 AM
Hi All   We have 5 node cluster and post upgrade from 1.7.1 to 1.14.0. We observed that below warnings are continuously writing on primary node logs.  2022-01-20 05:32:49,820 WARN [kafka-producer-network-thread | producer-1] o.a.k.common.network.SslTransportLayer Failed to send SSL Close message java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:65) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) at org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:213) at org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:176) at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:703) at org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:61) at org.apache.kafka.common.network.Selector.doClose(Selector.java:741) at org.apache.kafka.common.network.Selector.close(Selector.java:729) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:522) at org.apache.kafka.common.network.Selector.poll(Selector.java:412) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460) at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71) at org.apache.kafka.clients.producer.internals.Sender.awaitLeastLoadedNodeReady(Sender.java:415) at org.apache.kafka.clients.producer.internals.Sender.maybeSendTransactionalRequest(Sender.java:347) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:214) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163) at java.lang.Thread.run(Thread.java:748) 2022-01-20 05:32:49,857 WARN [kafka-producer-network-thread | producer-2] org.apache.kafka.clients.NetworkClient [Producer clientId=producer-2, transactionalId=d801193a-15ab-4823-be82-caddbcae65b0] Connection to node -9 could not be established. Broker may not be available. 2022-01-20 05:32:49,906 WARN [kafka-producer-network-thread | producer-83] org.apache.kafka.clients.NetworkClient [Producer clientId=producer-83, transactionalId=6fe4555f-5f6e-4fb3-82c5-a7ddaa1e8788] Connection to node -9 could not be established. Broker may not be available. 2022-01-20 05:32:49,949 WARN [kafka-producer-network-thread | producer-92] org.apache.kafka.clients.NetworkClient [Producer clientId=producer-92, transactionalId=3fe7b7c8-336b-4c29-addb-ab5cdb8425dc] Connection to node -9 could not be established. Broker may not be available. 2022-01-20 05:32:49,973 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0 bytes) from archive 2022-01-20 05:32:50,073 WARN [kafka-producer-network-thread | producer-2] o.a.k.common.network.SslTransportLayer Failed to send SSL Close message java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:65) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) at org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:213) at org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:176) at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:703) at org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:61) at org.apache.kafka.common.network.Selector.doClose(Selector.java:741) at org.apache.kafka.common.network.Selector.close(Selector.java:729) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:522) at org.apache.kafka.common.network.Selector.poll(Selector.java:412) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460) at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71) at org.apache.kafka.clients.producer.internals.Sender.awaitLeastLoadedNodeReady(Sender.java:415) at org.apache.kafka.clients.producer.internals.Sender.maybeSendTransactionalRequest(Sender.java:347) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:214) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163) at java.lang.Thread.run(Thread.java:748) 2022-01-20 05:32:50,236 WARN [kafka-producer-network-thread | producer-83] o.a.k.common.network.SslTransportLayer Failed to send SSL Close message java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:65) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) at org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:213) at org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:176) at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:703) at org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:61) at org.apache.kafka.common.network.Selector.doClose(Selector.java:741) at org.apache.kafka.common.network.Selector.close(Selector.java:729) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:522) at org.apache.kafka.common.network.Selector.poll(Selector.java:412) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460) at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71) at org.apache.kafka.clients.producer.internals.Sender.awaitLeastLoadedNodeReady(Sender.java:415) at org.apache.kafka.clients.producer.internals.Sender.maybeSendTransactionalRequest(Sender.java:347) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:214) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163) at java.lang.Thread.run(Thread.java:748) 2022-01-20 05:32:50,304 WARN [kafka-producer-network-thread | producer-37] org.apache.kafka.clients.NetworkClient [Producer clientId=producer-37, transactionalId=d229f1f0-31f2-4b58-9099-29b6df1db2dc] Connection to node -9 could not be established. Broker may not be available. 2022-01-20 05:32:50,398 WARN [kafka-producer-network-thread | producer-2] o.a.k.common.network.SslTransportLayer Failed to send SSL Close message java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:65) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) at org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:213) at org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:176) at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:703) at org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:61) at org.apache.kafka.common.network.Selector.doClose(Selector.java:741) at org.apache.kafka.common.network.Selector.close(Selector.java:729) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:522) at org.apache.kafka.common.network.Selector.poll(Selector.java:412) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460) at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71) at org.apache.kafka.clients.producer.internals.Sender.awaitLeastLoadedNodeReady(Sender.java:415) at org.apache.kafka.clients.producer.internals.Sender.maybeSendTransactionalRequest(Sender.java:347) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:214) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163) at java.lang.Thread.run(Thread.java:748)   @MattWho ... View more
Labels:

Re: Post upgrade of apache nifi cluster getting F...

by Explorer in Support Questions
‎12-16-2021 10:12 PM
‎12-16-2021 10:12 PM
@MattWho  -Thanks for your reply.   We have changed below property    Before “nifi.provenance.repository.compress.on.rollover=true”   After “nifi.provenance.repository.compress.on.rollover=false”   We have the enough space for provenance /dev/mapper/vg04-provenrepo 3.7T 27G 3.7T 1% /opt/nifi/provenance_repository ... View more

Post upgrade of apache nifi cluster getting Faile...

by Explorer in Support Questions
‎12-08-2021 05:02 AM
‎12-08-2021 05:02 AM
Hi All   We have upgraded the 3 node nifi cluster from 1.7.1 to 1.12.1 after that we are getting below error:   2021-12-07 01:22:12,822 ERROR [Compress Provenance Logs-1-thread-1] o.a.n.p.s.EventFileCompressor Failed to compress /opt/nifi/provenance_repository/uat1/8781241.prov on rollover   When we saw in the provenance repository its compressed 10 mins before. -rw-r-----. 1 bidanifiu1 bidanifiu1 791251 Dec 7 01:12 8781241.prov.gz   ... View more
Labels:

Re: Cloudera scm server start failed

by Explorer in Support Questions
‎01-24-2019 12:34 PM
‎01-24-2019 12:34 PM
Hello Ihebert,   I found the solution I forgot to give permissions to Mysql java connector jar file ... View more

Cloudera scm server start failed

by Explorer in Support Questions
‎01-23-2019 08:37 AM
‎01-23-2019 08:37 AM
  when I run the command service cloudera-scm-server start Shows failed  Urgent help needed  as I have stuck with issue since last 2 days   org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)         at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)         at com.cloudera.server.cmf.Main.bootstrapSpringContext(Main.java:392)         at com.cloudera.server.cmf.Main.<init>(Main.java:242)         at com.cloudera.server.cmf.Main.main(Main.java:216) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: cmf.server] Unable to build EntityManagerFactory         at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:149)         at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:102)         at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1440)         at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:247)         at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)         at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)         ... 17 more Caused by: javax.persistence.PersistenceException: [PersistenceUnit: cmf.server] Unable to build EntityManagerFactory         at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:924)         at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:899)         at org.hibernate.ejb.HibernatePersistence.createEntityManagerFactory(HibernatePersistence.java:59)         at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:63)         at com.cloudera.enterprise.CommonMain.setupHibernate(CommonMain.java:183)         at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:114)         at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:65)         at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:142)         ... 22 more             tail -n50 /var/log/cloudera-scm-server/cloudera-scm-server.out         at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)         at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)         at com.cloudera.server.cmf.Main.bootstrapSpringContext(Main.java:392)         at com.cloudera.server.cmf.Main.<init>(Main.java:242)         at com.cloudera.server.cmf.Main.main(Main.java:216) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: cmf.server] Unable to build EntityManagerFactory         at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:149)         at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:102)         at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1440)         at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:247)         at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)         at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)         ... 17 more Caused by: javax.persistence.PersistenceException: [PersistenceUnit: cmf.server] Unable to build EntityManagerFactory         at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:924)         at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:899)         at org.hibernate.ejb.HibernatePersistence.createEntityManagerFactory(HibernatePersistence.java:59)         at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:63)         at com.cloudera.enterprise.CommonMain.setupHibernate(CommonMain.java:183)         at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:114)         at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:65)         at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:142)         ... 22 more Caused by: org.hibernate.service.classloading.spi.ClassLoadingException: HHH010003: JDBC Driver class not found: com.mysql.jdbc.Driver         at org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider.configure(C3P0ConnectionProvider.java:142)         at org.hibernate.service.internal.StandardServiceRegistryImpl.configureService(StandardServiceRegistryImpl.java:75)         at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:159)         at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:131)         at org.hibernate.engine.jdbc.internal.JdbcServicesImpl.buildJdbcConnectionAccess(JdbcServicesImpl.java:223)         at org.hibernate.engine.jdbc.internal.JdbcServicesImpl.configure(JdbcServicesImpl.java:89)         at org.hibernate.service.internal.StandardServiceRegistryImpl.configureService(StandardServiceRegistryImpl.java:75)         at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:159)         at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:131)         at org.hibernate.cfg.Configuration.buildTypeRegistrations(Configuration.java:1797)         at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1755)         at org.hibernate.ejb.EntityManagerFactoryImpl.<init>(EntityManagerFactoryImpl.java:96)         at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:914)         ... 29 more Caused by: org.hibernate.service.classloading.spi.ClassLoadingException: Unable to load class [com.mysql.jdbc.Driver]         at org.hibernate.service.classloading.internal.ClassLoaderServiceImpl.classForName(ClassLoaderServiceImpl.java:149)         at org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider.configure(C3P0ConnectionProvider.java:139)         ... 41 more Caused by: java.lang.ClassNotFoundException: Could not load requested class : com.mysql.jdbc.Driver         at org.hibernate.service.classloading.internal.ClassLoaderServiceImpl$AggregatedClassLoader.findClass(ClassLoaderServiceImpl.java:296)         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)         at java.lang.Class.forName0(Native Method)         at java.lang.Class.forName(Class.java:348)         at org.hibernate.service.classloading.internal.ClassLoaderServiceImpl.classForName(ClassLoaderServiceImpl.java:146)         ... 42 more ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎09-19-2024 03:54 AM
Community Statistics
Member Since ‎01-22-2019 06:00 AM
Last Visited ‎09-19-2024 03:54 AM
Posts 11