About Koffi

Koffi · ‎09-04-2020

Using: HDP 3.0.1 HDFS 3.1.0 NAMENODE HEAP: 84.8% 3.3 GB / 4.0 GB DISK USAGE (DFS USED): 71.20% 63.9 TB / 89.7 TB DISK USAGE (NON DFS USED): 0.74% 676.2 GB / 89.7 TB DISK REMAINING: 28.06% 25.2 TB / 89.7 TB Block Size : 128 MB Any idea how to solve to reduce reduce the name node heap size usage? Thank you

Koffi · ‎08-19-2020

Hi, I am currently facing issue with NiFi 1.7.0 (2 nodes), My flows had been running fine since months. Couple of weeks ago we did a downgrading of the nifi machines. We went from CPU 16 RAM 128G to CPU 4 RAM 32G. After the dowgrading we changed parameters to make sure that nifi take into consideration the new size of the machines Initial values: Initial memory allocation: 80G Max memory allocation: 112g New values: Initial memory allocation: 20G Max memory allocation: 26G within nifi we did also change in NIFI Genaral Settings from Maximum timer driven thread count: 64 Maximum event driven thread count: 16 to Maximum timer driven thread count: 2 Maximum event driven thread count: 4 Now some time when we try to load we get an enexpected error occured with 2020-08-18 10:57:51,942 ERROR [NiFi logging handler] org.apache.nifi.StdErr OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000780000000, 218103808, 0) failed; error='Cannot allocate memory' (errno=12) 2020-08-18 10:57:51,956 INFO [NiFi logging handler] org.apache.nifi.StdOut # There is insufficient memory for the Java Runtime Environment to continue. 2020-08-18 10:57:51,957 INFO [NiFi logging handler] org.apache.nifi.StdOut # Native memory allocation (mmap) failed to map 218103808 byt Could you please help me with this issue.

Koffi · ‎07-09-2020

I get below error sometimes while running a query in zeppelin through JDBC connection: java.sql.SQLException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection closed by remote host at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:334) at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:265) at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:291) at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:291) at org.apache.zeppelin.jdbc.JDBCInterpreter.executeSql(JDBCInterpreter.java:718) at org.apache.zeppelin.jdbc.JDBCInterpreter.interpret(JDBCInterpreter.java:801) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:103) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:633) at org.apache.zeppelin.scheduler.Job.run(Job.java:188) at org.apache.zeppelin.scheduler.ParallelScheduler$JobRunner.run(ParallelScheduler.java:162) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection closed by remote host at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) at org.apache.thrift.transport.TSaslTransport.flush(TSaslTransport.java:501) at org.apache.thrift.transport.TSaslClientTransport.flush(TSaslClientTransport.java:37) at org.apache.hadoop.hive.metastore.security.TFilterTransport.flush(TFilterTransport.java:77) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) at org.apache.hive.service.rpc.thrift.TCLIService$Client.send_ExecuteStatement(TCLIService.java:239) at org.apache.hive.service.rpc.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:231) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1572) at com.sun.proxy.$Proxy8.ExecuteStatement(Unknown Source) at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:323) ... 16 more Caused by: java.net.SocketException: Connection closed by remote host at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1547) at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159) ... 30 more How can we resolved this ? note:

Koffi · ‎06-05-2020

Hi, We are currently using HDP 3.0.1 on a multi nodes environment. We are not able to start the Metrics Collector, and the following message appears on the logs /var/log/ambari-metrics-collector/ambari-metrics-collector.log FATAL Error starting AMSApplicationServer org.apache.ambari.metrics.core.timeline.MetricsSystemInitializationException: Error creating Metrics Metadata Schema in HBase using Phoenix. at org.apache.ambari.metrics.core.timeline.discovery.TimelineMetricMetadataManager.initializeMetadata(TimelineMetricMetadataManager.java:169) at org.apache.ambari.metrics.core.timeline.discovery.TimelineMetricMetadataManager.initializeMetadata(TimelineMetricMetadataManager.java:134) at org.apache.ambari.metrics.core.timeline.HBaseTimelineMetricsService.initializeSubsystem(HBaseTimelineMetricsService.java:121) at org.apache.ambari.metrics.core.timeline.HBaseTimelineMetricsService.serviceInit(HBaseTimelineMetricsService.java:102) We did try to: Restart Ambari agents Restart Metrics Monitors (roll back) restart Metrics Collector (rool back) Restart Ambari metric Restart HBASE Try to follow the steps on https://community.cloudera.com/t5/Community-Articles/How-to-solve-Ambari-Metrics-corrupted-data/ta-p/245261 Each try was not able to fix the error Is there a way to resolve this problem? Thank you

Koffi · ‎04-23-2020

Hi, I am getting the following alert on Ambari for the Yarn service: There are 2 stale alerts from 2 host(s): node-m03.host.com [NodeManager Health Summary (6h 34m)], node-wn01.host.com [NodeManager Health (7h)] However, i don't see any nodemanagers with the unhealthy status on ambari... And i did a check on the YARN service with success. I am not sure what exactly i should be doing to correct this alert. Could you please help me troubleshoot this issue? Thank you HDP 3.0.1 YARN 3.1.0 Ambari 2.7.1

Koffi · ‎03-30-2020

Hi, We are currenly using HDP 3.0.1 with ambari 2.7.1. I was trying to solve a connection issue on the zeppelin ui that give to some of the users a disconnected status while they are using the chrome browser...(curiously it seems to be working just fine with firefox ). I am able to see on the zeppelin notebook logs the following message: Exception in login: org.apache.shiro.authc.AuthenticationException: Authentication token of type [class org.apache.zeppelin.realm.jwt.JWTAuthenticationToken] could not be authenticated by any configured realms. Please ensure that at least one realm can authenticate these tokens. Also on the knox logs i have the following error message not sure if it is linked: Shiro unable to login: javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C090400, comment: AcceptSecurityContext error, data 52e, v1db1 Could you please help us. Thank you

Koffi · ‎03-17-2020

Hi, I have a hadoop environment that has 2 nifi nodes running. I am not able to connect to the nifi UI since yesterday and i try do restart one of the node without any success.... In in nifi-app.log, file seeing this as only Error level error: 2020-03-17 13:42:37,974 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for ***-nf02.****.ca:9091 -- Node disconnected from cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow. Could please help?

Koffi · ‎01-27-2020

Hi, I am currently using HDP 3.0.1 with ambari 2.7.1. What is the best way to set up a automatic snapshot of important folder on HDFS in a multi node environment? Thank you

Koffi · ‎12-30-2019

Hi, I Have a hadoop cluster 3.0.1 with 3 journalnodes, 1 nfsgateways node and 6 workernodes. I connected by ssh to the worker nodes today and realised by doing a "df -h" that one a the one local disk (/data/4) is around 94% used on every worker nodes whereas the others disk are between 50% and 65%... The HDFS status on the another hand is the following: Disk Usage (DFS Used) 44.77% 28.1 TB / 62.8 TB Disk Usage (Non DFS Used) 14.97% 9.4 TB / 62.8 TB Disk Remaining 40.26% 25.3 TB / 62.8 TB What are the the elements i should check to make sure that a full local disk won't create any issue?

Koffi · ‎11-28-2019

Hi, A large amount of users and groups were imported in ranger after a ldap usersync and now i need to find a way to delete more than 200 of users and groups. I know that ranger has a api... i would like to know how to remove a list of external groups or users using that api or maybe another method that allow me to do it faster? Thank you

Koffi · ‎11-22-2019

Hi @MattWho , In the case where knox is used for the authentification what type of changes need to be done in order to take into account the ad group? I know that in knox there is the topology where we can define the ldap... Thank you

Koffi · ‎11-21-2019

Hi @Shelton when you said authorizations.xml are you talking about authorizers.xml? the hadoop environment use ranger for the securioty and also is connected to a ldap server for the users and groups. I don't see any users.xml in the conf directory.

Koffi · ‎11-21-2019

Hi, We are currently using HDP 3.0 with ambari and we installed 2 nifi nodes. We made some config changes on nifi node01 without restarting both nodes (i only restarted the node01 and not node02). The changes were not working properly so we decided to roll back to the previous configs but whenever i try to start node01 i am getting the following error: Failed to connect node to cluster because local flow is different than cluster flow. My guess would be that both nodes are out of synch.... How can we fix this issue? Thank you for your help.

Koffi · ‎11-18-2019

Hi @MattWho, Thank you for your answer, the identification is done through knox and i was wondering which config need to be changed in my case?

Koffi · ‎11-15-2019

HI, We have a multi host hadoop environment that use HDP-3.0.1 with Nifi 1.7.0. The environment is Kerberised and we use ranger for the security, it is also connected to the LDAP/AD server for the user and groups. Whenever i try to connect to NIFI using a u ser_test i get the message : The u ser_test is part of a group who has full access to nifi in ranger... -Policy Name: All-nifi-Resource -NIfi Resource Identifier: * -Allow Conditions: -Select Group: group_test -Permissions: Read, Write It seems that the AD group is not used when trying to connect in nifi (If i put the username u ser_test in the Select User section in Ranger it has access to nifi...). I have the following msg in the nifi-user.log: 2019-11-13 12:18:29,368 INFO [NiFi Web Server-65454] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=xx-xx-xx-xxXX.XxxXx.xx, O=XxxXx Inc., L=XXXX, ST=XXXX, C=XX 2019-11-13 12:18:33,097 INFO [NiFi Web Server-65454] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.. Returning Conflict response. 2019-11-13 12:18:33,136 INFO [NiFi Web Server-65286] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: OpenId Connect is not configured.. Returning Conflict response. 2019-11-13 12:18:33,151 INFO [NiFi Web Server-65436] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<Knox JWT token>) GET https://xx-xx-xx-xxXX.XxxXx.xx:xxxx/nifi-api/flow/current-user (source ip: XX.XXX.XXX.XXX) 2019-11-13 12:18:33,151 INFO [NiFi Web Server-65436] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for user_test 2019-11-13 12:12:32,122 INFO [NiFi Web Server-65436] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[user_test], groups[none] does not have permission to access the requested resource. Unable to view the user interface. Returning Forbidden response. I am not sure why none appears and not group_test. Thank you in advance for your help on that subject.

Koffi · ‎11-15-2019

Hi @Shelton, Thank you for your answer. From the link you provided i saw the following command that is used to stop every components using the ambari api. curl -i -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Stop All Host Components","operation_level":{"level":"HOST","cluster_name":"Sandbox","host_names":"sandbox.hortonworks.com"},"query":"HostRoles/component_name.in(APP_TIMELINE_SERVER,DATANODE,HISTORYSERVER,METRICS_COLLECTOR,METRICS_GRAFANA,METRICS_MONITOR,NAMENODE,NFS_GATEWAY,NODEMANAGER,RANGER_ADMIN,RANGER_TAGSYNC,RANGER_USERSYNC,RESOURCEMANAGER,SECONDARY_NAMENODE,ZOOKEEPER_SERVER)"},"Body":{"HostRoles":{"state":"INSTALLED"}}}' http://localhost:8080/api/v1/clusters/Sandbox/hosts/sandbox.hortonworks.com/host_components I guess it gives me the order in which they will be executed... my question would be does it waits for the previous component to complete its stopping process before starting is own stopping process? Exemple: APP_TIMELINE_SERVER, DATANODE Does DATANODE starts its stopping process after the APP_TIMELINE_SERVER complete its process? Thank you

Koffi · ‎11-15-2019

Hi, I am currently using Hortonworks HDP 3.01 with ambari 2.7.1.0 but it seems that restarting all services can be a challenge because starting services/components must be done in a certain order to avoid timeouts and errors.... Is there a way to script the process? There is a section "Service Auto Start" in ambari but it doesn't seems that Ambari is smart enough to start services/componants while taking into consideration the dependencies... Could you please help? Thank you

Koffi · ‎10-02-2019

Hi, Is there a way in NIFI to compress a simple file exemple.txt to a specific compresssed gzip file like archive.gzip or archive.gz? When the compressed file (archive.gzip or archive.gz) is decompressed the filename should be exemple.txt and not archive. Thank you

Koffi · ‎09-13-2019

Hello, I am having an issue running a query on my host and i getting the following message: Any idea how to solve this issue?

Koffi · ‎09-03-2019

hi @nshawa, I am having the following error on PutHiveStreaming processor after running the template you provided: Any idea how to fix this?

Koffi · ‎09-03-2019

Hi @DennisJaheruddi, Thank you very much for your answer. I try to follow the steps you have provided: 1.Please check for an error, if it does not show in the ui, check the nifi app log I checked both logs and i was not able to find anything meaningful... 2. Try using the example flow exactly as given, without the changes I downloaded and uploaded the original pattern in my nifi. I get the error message 3. If this succeeds, only change the puthive3streaming processor it does not work... 4. If the original template fails, it is likely there is a problem outside nifi, perhaps in how you prepared the hive table. In that case double check the details such as: Table is stored as ORC Transactional Property is set to “True” The Table is Bucketed i have used the following script to generate the hive table CREATE TABLE OLYMPICS(CITY STRING,EDITION INT,SPORT STRING,SUB_SPORT STRING,ATHLETE STRING,COUNTRY STRING,GENDER STRING,EVENT STRING,EVENT_GENDER STRING,MEDAL STRING) CLUSTERED BY (EDITION)INTO 3 BUCKETS ROW FORMAT DELIMITED STORED AS ORC LOCATION '/user/hive/olympics' TBLPROPERTIES('transactional'='true'); 5. If the original template still fails, try using a normal PutHive processor could you please give me a example? Thank you.

Koffi · ‎08-26-2019

HI, I am trying to import data from a csv file into hive using Apache NiFi - Version 1.7.0.3.2.0.0-520. following the steps described on : https://community.cloudera.com/t5/Community-Articles/Stream-data-into-HIVE-like-a-Boss-using-NiFi-HiveStreaming/ta-p/247130 i have changed few things - The table olympics on hive is not external - I am using PutHive3Streaming My current workflow: Getfile: InferAvroSchema: ConvertCSVToAvro: PutHive3Streaming: PutFile: AvroReader: I am not getting any errors so far but when i run the processors the data is not stored in hive and i have 1 failure queued on PutHive3Streaming --> PutFile. Could you please give me some guidance? Thank you

Koffi · ‎08-19-2019

@jsensharma I have realized that this host has a different java by default then the ambari host host: shell> java -version openjdk version "1.8.0_222" OpenJDK Runtime Environment (build 1.8.0_222-b10) OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode) ambari host: shell> java -version java version "1.8.0_112" Java(TM) SE Runtime Environment (build 1.8.0_112-b15) Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode) Could it be the root cause of this issue?

Koffi · ‎08-19-2019

Hi @jsensharma Please find below the complete error trace including Caused By section: 2019-08-19 13:37:49,643 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:run(997)) - Public cache exiting 2019-08-19 13:37:49,643 WARN nodemanager.NodeResourceMonitorImpl (NodeResourceMonitorImpl.java:run(167)) - org.apache.hadoop.yarn.server.nodemanager.NodeResourceMonitorImpl is interrupted. Exiting. 2019-08-19 13:37:49,645 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(210)) - Stopping NodeManager metrics system... 2019-08-19 13:37:49,646 INFO impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted. 2019-08-19 13:37:49,647 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(216)) - NodeManager metrics system stopped. 2019-08-19 13:37:49,647 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(607)) - NodeManager metrics system shutdown complete. 2019-08-19 13:37:49,647 ERROR nodemanager.NodeManager (NodeManager.java:initAndStartNodeManager(936)) - Error starting NodeManager org.apache.hadoop.yarn.exceptions.YarnRuntimeException: The per-node collector webapp failed to start. at org.apache.hadoop.yarn.server.timelineservice.collector.NodeTimelineCollectorManager.startWebApp(NodeTimelineCollectorManager.java:315) at org.apache.hadoop.yarn.server.timelineservice.collector.NodeTimelineCollectorManager.serviceStart(NodeTimelineCollectorManager.java:132) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) at org.apache.hadoop.yarn.server.timelineservice.collector.PerNodeTimelineCollectorsAuxService.serviceStart(PerNodeTimelineCollectorsAuxService.java:101) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStart(AuxServices.java:313) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:643) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:934) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1013) Caused by: java.io.IOException: Problem starting http server at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1165) at org.apache.hadoop.yarn.server.timelineservice.collector.NodeTimelineCollectorManager.startWebApp(NodeTimelineCollectorManager.java:311) ... 13 more Caused by: java.security.UnrecoverableKeyException: Get Key failed: Given final block not properly padded at sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:410) at sun.security.provider.KeyStoreDelegator.engineGetKey(KeyStoreDelegator.java:96) at sun.security.provider.JavaKeyStore$DualFormatJKS.engineGetKey(JavaKeyStore.java:70) at java.security.KeyStore.getKey(KeyStore.java:1023) at sun.security.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:133) at sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:70) at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:256) at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1087) at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:301) at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:221) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:72) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:268) at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:235) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.server.Server.doStart(Server.java:401) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1134) ... 14 more Caused by: javax.crypto.BadPaddingException: Given final block not properly padded at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:989) at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:845) at com.sun.crypto.provider.PKCS12PBECipherCore.implDoFinal(PKCS12PBECipherCore.java:399) at com.sun.crypto.provider.PKCS12PBECipherCore$PBEWithSHA1AndDESede.engineDoFinal(PKCS12PBECipherCore.java:431) at javax.crypto.Cipher.doFinal(Cipher.java:2165) at sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:348) ... 37 more 2019-08-19 13:37:49,651 INFO nodemanager.NodeManager (LogAdapter.java:info(51)) - SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NodeManager at hd-prd-ll01.hadoop.com/**.*.**.** ************************************************************/

Koffi · ‎08-15-2019

Hi, I am getting the following message on ambari on every worker nodes: 1/1 local-dirs usable space is below configured utilization percentage/no more usable space [ /data/2/hadoop/yarn/local : used space above threshold of 90.0% ] ; I have also checked on the ResourceManager UI > Nodes. 6 nodes are currently on a unhealthy state. Could you please help me get them back to a healthy state? Thank you

Koffi · ‎08-14-2019

Hello, I have currently a kerberized hadoop HDP 3.0.1 cluster where HIVESERVER2 INTERACTIVE has been deployed. It was working just fine but today... HIVESERVER2 INTERACTIVE reported the following alert on ambari: Hive LLAP Application The application reported a 'NOT RUNNING' state. Check took 7.116s. Any idea what could be the issue? Thank you for your help!

Koffi · ‎08-14-2019

The HDP Version is 3.0.1.0 with 16 nodes 4 are worker nodes and the environment is kerberized. The node wn03 needed to be rebooted by the infra for maintenance purpose. I stopped all the services on that worker node and restarted all the services after rebooting the worker machine wn03. This morning i get the following warning in ambari: Connection failed to https://hd-dev-c1-wn03.hadoop.com:8044 (Execution of 'curl --location-trusted -k --negotiate -u : -b /var/lib/ambari-agent/tmp/cookies/8d14b5f3-5456-4599-9510-0036effff91d -c /var/lib/ambari-agent/tmp/cookies/8d14b5f3-5456-4599-9510-0036effff91d -w '%{http_code}' https://hd-dev-c1-wn03.hadoop.com:8044 --connect-timeout 5 --max-time 7 -o /dev/null 1>/tmp/tmpGaZLZ4 2>/tmp/tmpag2rxg' returned 7. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to hd-dev-c1-wn03.hadoop.com:8044; Connection refused 000)

Koffi · ‎08-13-2019

HI, I am new with YARN, I am not able to restart the nodemanagers on some of nodes. Could you please help me out?

Koffi · ‎08-12-2019

Hi I am trying to start live the nodemanager previously decommisionned on a node but it stops soon after it starts... ERROR collector.NodeTimelineCollectorManager (NodeTimelineCollectorManager.java:startWebApp(314)) - The per-node collector webapp failed to start. java.io.IOException: Problem starting http server any idea what could be the problem?

Koffi · ‎08-09-2019

Hi , I have added new worker nodes to my Hadoop cluster. i get the following error on the yarn logs less hadoop-yarn-nodemanager-hd-c2-wn09.log: 2019-08-09 08:44:44,898 ERROR ssl.SSLFactory (SSLHostnameVerifier.java:check(357)) - Host check error {} javax.net.ssl.SSLException: hostname in certificate did't match: != OR Any idea how to fix this?

Online	Offline
Last Visited	‎10-07-2020 03:05 PM

Date Registered	‎05-30-2019 11:54 AM
Last Visited	‎10-07-2020 03:05 PM
Total Messages Posted	45
Total Kudos Received	1

Re: Failed to connect node to cluster because loca...

Namenode Heap Size usage is > 80% on daily basis

Nifi There is insufficient memory for the java Run...

java.sql.SQLException: org.apache.thrift.transport...

Ambari Metrics Collector not starting

Ambari Alert on Yarn: NodeManager Health Summary

Zeppelin disconnected error HDP3.0.1

Can't connect to the nifi UI

Automatic snapshot of important folders in HDFS

One disk of the datanode is at 99% used...

How to Remove a large list of LDAP USERS/GROUPS in...

Re: AD group not use while trying to connect to ni...

Re: Failed to connect node to cluster because loca...

Failed to connect node to cluster because local fl...

Re: AD group not use while trying to connect to ni...

AD group not use while trying to connect to nifi

Re: Restart services in Ambari 2.7.1.0

Restart services in Ambari 2.7.1.0

Nifi compress Filename

Token can't be found in cache

Re: Stream data into HIVE like a Boss using NiFi H...

Re: PutHive3Streaming not storing data in Hive

PutHive3Streaming not storing data in Hive

Re: Trying to start the nodemanager

Re: Trying to start the nodemanager

Worker node on a unhealthy state

The application reported a 'NOT RUNNING' state

Re: Can't restart the ressourcemanager on node

Can't restart the ressourcemanager on node

Trying to start the nodemanager

ERROR ssl.SSLFactory