About sgowda

sgowda · ‎12-27-2016

@rdoktorics thanks , this works !

sgowda · ‎12-26-2016

below shell command works for me for both ambari and HDP repo cluster create --version 2.X --stackRepoId HDP-2.X --stackBaseURL http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/2.x/BUILDS/2.X.X.0-154 --utilsRepoId HDP-UTILS-1.1.0.21 --utilsBaseURL http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7 --stack HDP --verify true --os redhat7 --ambariRepoGpgKey http://s3.amazonaws.com/dev.hortonworks.com/ambari/centos6/2.x/BUILDS/2.X.X.0-524/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins --ambariRepoBaseURL http://s3.amazonaws.com/dev.hortonworks.com/ambari/centos6/2.x/BUILDS/2.X.X.0-524 --ambariVersion 2.X.X.0-524 --enableSecurity true --kerberosMasterKey master --kerberosAdmin admin --kerberosPassword admin --wait true

sgowda · ‎12-22-2016

Ever come across below error while starting HDFS datanode in unsecure cluster ? 016-12-22 09:00:41,045 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2016-12-22 09:00:41,152 INFO datanode.DataNode (DataNode.java:shutdown(1915)) - Shutdown complete. 2016-12-22 09:00:41,153 ERROR datanode.DataNode (DataNode.java:secureMain(2630)) - Exception in secureMain java.net.SocketException: Permission denied at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:463) at sun.nio.ch.Net.bind(Net.java:455) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125) at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:475) at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1021) at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:455) at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:440) at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:844) at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:194) at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:340) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:745) 2016-12-22 09:00:41,157 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1 2016-12-22 09:00:41,159 INFO datanode.DataNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG: This error means DataNode is unable to bind to provided socket. Reason ? This is most like because the http port or other port of DataNode are listening on port < 1024. For e.x http port property is set to as below. dfs.datanode.http.address=1022 Solution: Change the port something as 50075 which is greater than 1024. DataNode should be able to start with this solution.

sgowda · ‎12-22-2016

@gvenkataramanan Please try the blueprint. small-default-hdp-corrected.txt . Issue was the NameNode default heapsize of 1024 is causing issue. Overriding it to 2048 resolves the issue. Regarding HDFS datanode configured to /hadoopfs by default ? that is how cloudbreak is designed i believe so that datanode have seperate dedicated disks.

gvenkataramanan · ‎12-22-2016

Please start it as root and from inside the cloudbreak-deployment directory where you have the Profile file. cd cloudbreak-deployment sudo su root cbd start

sgowda · ‎12-22-2016

@Yukti Agrawal On accessing the RM UI and click on the applicationId and corresponding logs link. It will take you to a screen something as below. Here you would see a file dag_*.dot, this should give you the query/MR graph that is being executed. Other option would be if the execution engine is TEZ, then you can leverage TEZ UI to view the actual query as well.

a_rajeshinfotec · ‎12-19-2016

2016-12-19 04:43:41,466 ERROR [main]: exec.Task (TezTask.java:execute(212)) - Failed to execute tez graph. org.apache.tez.dag.api.TezException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1481976368061_0017 to YARN : Application application_1481976368061_0017 submitted by user root to unknown queue: agent at org.apache.tez.client.TezClient.start(TezClient.java:413) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196) at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:271) at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:151) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1720) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1477) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1254) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1118) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:168) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:379) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:739) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1481976368061_0017 to YARN : Application application_1481976368061_0017 submitted by user root to unknown queue: agent at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:271) at org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:72) at org.apache.tez.client.TezClient.start(TezClient.java:408) ... 22 more 2016-12-19 04:43:41,466 INFO [main]: hooks.ATSHook (ATSHook.java:<init>(90)) - Created ATS Hook 2016-12-19 04:43:41,466 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver> 2016-12-19 04:43:41,470 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(162)) - </PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1482122621466 end=1482122621470 duration=4 from=org.apache.hadoop.hive.ql.Driver> 2016-12-19 04:43:41,471 ERROR [main]: ql.Driver (SessionState.java:printError(932)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask 2016-12-19 04:43:41,471 INFO [main]: ql.Driver (Driver.java:execute(1621)) - Resetting the caller context to 2016-12-19 04:43:41,471 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(162)) - </PERFLOG method=Driver.execute start=1482122620489 end=1482122621471 duration=982 from=org.apache.hadoop.hive.ql.Driver> 2016-12-19 04:43:41,471 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver> 2016-12-19 04:43:41,471 INFO [ATS Logger 0]: hooks.ATSHook (ATSHook.java:createPostHookEvent(193)) - Received post-hook notification for :root_20161219044340_094281c1-c0f2-4cc9-83a5-962b067937f7 2016-12-19 04:43:41,488 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(162)) - </PERFLOG method=releaseLocks start=1482122621471 end=1482122621488 duration=17 from=org.apache.hadoop.hive.ql.Driver> 2016-12-19 04:43:41,505 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver> 2016-12-19 04:43:41,505 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(162)) - </PERFLOG method=releaseLocks start=1482122621505 end=1482122621505 duration=0 from=org.apache.hadoop.hive.ql.Driver>

KuldeepK · ‎12-09-2016

@justlearning You can use standard apache oozie examples and modify them as per your requirement. This is easiest way to get started writing Oozie workflows. It has example workflow.xml for each supported action. You can find Oozie examples on HDP cluster at below location(Provided you have installed Oozie client) /usr/hdp/current/oozie-client/doc/oozie-examples.tar.gz Hope this information helps!

sshimpi · ‎11-24-2016

@Santhosh B Gowda Make sure you have added the property in custom hdfs-site <property> <name>dfs.namenode.acls.enabled</name> <value>true</value> </property>

sgowda · ‎11-06-2016

Followed the steps mentioned in this article and it started working !!

Online	Offline
Last Visited	‎10-08-2018 05:03 AM

Member Since	‎05-20-2016 05:56 PM
Last Visited	‎10-08-2018 05:03 AM
Posts	155
Kudos received	220

Cloudera Community

Re: Enabling JMX for Nifi

Re: Can Oozie invoke and RestFul service for notif...

Re: Ambari installation failed through ambari-blu...

Re: Hbase: Failed to become active master

Re: what's the spark history server 's name?

Re: cloudbreak shell process cmdfile output

Re: specifying hdp repo via cloudbreak shell

Datanode does not start due to permission denied

Re: cloudbreak default blueprint installation in o...

Re: not able to start cloudbreak

Re: Is there a way to find out the query used by a...

Re: execution error, return code 1 from org.apache...

Re: process to create the oozie script in hortonwo...

Re: how to provide hdfs umask to a specific direct...

Re: phoenix query execution using oozie java actio...