Thanks @Karthik Palanisamy for sharing this article. I have done the same activity without initializing shared edits or stopping the entire HDFS service. Below are the steps which I have followed. Can you please elaborate why we need to re-initialize the shared edits and correct me if the steps I have followed are not appropriate one. 1. Change the settings in Ambari for journal node edits directory(dfs.journalnode.edits.dir) from /hadoop/hdfs/journal/ to /data/1/journal/ 2. Don't restart any services immediately. 3. Stop the journal node on NODE1. a. SSH to NODE1 b. sudo mkdir -p /data/1/journal c. sudo chown hdfs:hadoop /data/1/journal d. sudo rsync -ahvAX /hadoop/hdfs/journal/* /data/1/journal/ Start the journal node on NODE1 4. Repeat step 3 for remaining two journal nodes NODE2 and NODE3. 5. Restart the required services accordingly (Rolling or All at once) ... View more

Thanks @bkosaraju for the response and details. ... View more

Apologies @nkumar for the delay in response. The issue is related to Ambari which behaves differently after disabling and re-enabling the kerberos. Issue got fixed after making changes to ambari with the help of Hortonworks Support using below REST calls. curl -u test:test -H "X-Requested-By: ambari" -X POST http://ambari-server:8080/api/v1/clusters/MyClusterName/services/KERBEROS curl -u test:test -H "X-Requested-By: ambari" -X POST http://ambari-server:8080/api/v1/clusters/MyClusterName/services/KERBEROS/components/KERBEROS_CLIENT curl -s -u test:test http://ambari-server:8080/api/v1/hosts|grep host_name| sed -n 's/.*"host_name" : "\([^\"]*\)".*/\1/p'>hostcluster.txt for i in `cat hostcluster.txt`; do curl -u test:test -H "X-Requested-By: ambari" -X POST http://ambari-server:8080/api/v1/clusters/MyClusterName/hosts/$i/host_components/KERBEROS_CLIENT; done curl -u test:test -H 'X-Requested-By: ambari' -X PUT -d '{"HostRoles": {"state":"INSTALLED"}}' http://ambari-server:8080/api/v1/clusters/MyClusterName/host_components?HostRoles/state=INIT curl -H "X-Requested-By:ambari" -u test:test -i -X PUT -d @./payload.json http://ambari-server:8080/api/v1/clusters/MyClusterName ... View more

Update ! We created our own jar file for search_replace from apache flume 1.6 code with minor changes. Now its working as expected. ... View more

Hi @dthakkar, Im using below piece of config which works flawlessly in apache flume 1.7 but HDP flume throws error saying class not found for search_replace. Even HDP Flume documentation doesn't mentions search_replace https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/ds_flume/FlumeUserGuide.html mytest.sources.src.interceptors = s0 s1 s2 s3 mytest.sources.src.interceptors.s0.type = host mytest.sources.src.interceptors.so.useIP = false mytest.sources.src.interceptors.s1.type = search_replace mytest.sources.src.interceptors.s1.searchPattern = ^\\{ mytest.sources.src.interceptors.s1.replaceString = \\" mytest.sources.src.interceptors.s2.type = search_replace mytest.sources.src.interceptors.s2.searchPattern = \\} mytest.sources.src.interceptors.s2.replaceString = mytest.sources.src.interceptors.s3.type = search_replace mytest.sources.src.interceptors.s3.searchPattern = \\,\\{ <br> ... View more

Hi @Geoffrey Shelton Okot, I have checked the content and everything looks good and we are using same krb config files across different clusters. I dont see any discrepencies with the kerberos. ... View more

Thanks @Geoffrey Shelton Okot for the quick response. Ambari is running as an user which has got sudo privileges. And auto-start services is enabled but only metrics-collector is enabled. In the KDC i can see that it has created corresponding principals associated with the service and hostnames. Only issue i have observed is it stopped creating keytab files and distribute it to the designated system which ambari reported successful. I have carried out this activity some 9-10 times but all the time its ending up without creating keytab files. ... View more

Yes I have regenerated but it didn't help. What i have seen is it didn't create new keytabs. ... View more

Thanks @bkosaraju for the quick response. Somehow it didn't help me. Here is what i have done. 1. Remove LDAP users using below command curl --insecure -u admin:$PASSWORD -H 'X-Requested-By: ambari' -X DELETE http://$AMBARI_HOST:8080/api/v1/users/myUser 2. Update /etc/amabari-server/conf/ambari.properties with below parameter authentication.ldap.username.forceLowercase=false 3. Restart Ambari server. 4. Sync single ldap user using below command ambari-server sync-ldap --users users Still I'm getting same results, its creating lowercase account and lowercase HDFS user directories. Am I missing anything ? HDP : 2.6.1.0 Ambari : 2.5.1.0 ... View more

Thanks @Jay SenSharma for the quick response. As I mentioned earlier it goes down only when flume agents are running else it just works fine. Ambari metrics verion : 0.1.0 The /var/log/ambari-metrics-collector/ambari-metrics-collector.out file has only this log. Sep 15, 2017 11:59:51 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider as a provider class Sep 15, 2017 11:59:51 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.AHSWebServices as a root resource class Sep 15, 2017 11:59:51 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices as a root resource class Sep 15, 2017 11:59:51 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class Sep 15, 2017 11:59:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate INFO: Initiating Jersey application, version 'Jersey: 1.11 12/09/2011 10:27 AM' Sep 15, 2017 11:59:52 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton" Sep 15, 2017 11:59:52 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider to GuiceManagedComponentProvider with the scope "Singleton" Sep 15, 2017 11:59:52 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.AHSWebServices to GuiceManagedComponentProvider with the scope "Singleton" Sep 15, 2017 11:59:52 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices to GuiceManagedComponentProvider with the scope "Singleton" java.lang.NumberFormatException at java.math.BigDecimal.<init>(BigDecimal.java:494) at java.math.BigDecimal.<init>(BigDecimal.java:383) at java.math.BigDecimal.<init>(BigDecimal.java:806) at java.math.BigDecimal.valueOf(BigDecimal.java:1274) at org.apache.phoenix.schema.types.PDouble.getMaxLength(PDouble.java:70) at org.apache.phoenix.expression.LiteralExpression.newConstant(LiteralExpression.java:210) at org.apache.phoenix.expression.LiteralExpression.newConstant(LiteralExpression.java:173) at org.apache.phoenix.expression.LiteralExpression.newConstant(LiteralExpression.java:160) at org.apache.phoenix.compile.UpsertCompiler$UpsertValuesCompiler.visit(UpsertCompiler.java:1016) at org.apache.phoenix.compile.UpsertCompiler$UpsertValuesCompiler.visit(UpsertCompiler.java:1000) at org.apache.phoenix.parse.BindParseNode.accept(BindParseNode.java:47) at org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:856) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:593) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:581) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:336) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:331) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:329) at org.apache.phoenix.jdbc.PhoenixPreparedStatement.executeUpdate(PhoenixPreparedStatement.java:199) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor.commitMetrics(PhoenixHBaseAccessor.java:310) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor.commitMetricsFromCache(PhoenixHBaseAccessor.java:257) at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsCacheCommitterThread.run(MetricsCacheCommitterThread.java:35) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) ... View more

@dthakkar agent1.channels.kafkaChannel.parseAsFlumeEvent =false above configuration made the difference. Thanks a lot. ... View more

Thanks @dthakkar for your valuable response. Will give it a try from my end. ... View more

Update 1: When using only Kafka as sink with other types of source and channel, I'm able to push messages to Kafka Topic. It fails when kafkaSink is used with KafkaSource or KafkaChannel . Following have been tested: 1. kafkaSource, kafkaChannel, other Sinks -- Works 2. kafkaSource, kafkaChannel, kafkaSink -- Fails 3. anySource(except kafka), anychannel(except kafka), kafkaSink -- Works 4. kafkaSource, memoryChannel, kafkaSink -- Fails 5. anySource, kafkaChannel, kafkaSink -- Fails Has anyone tried using both kafkaSource and kafkaSink in same flume agent ? Thanks. ... View more

This issue got resolved for me after installing tez_client, hive_client and hcat on hive metastore servers. ... View more

I'm also getting the same error in hive metastore alerts. Has anyone faced this issue and fixed ? ... View more