Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

core-site for GPFS

core-site for GPFS

Super Collaborator

Hi i can configurate HDP to work with GPFS instead HDFS??

3 REPLIES 3

Re: core-site for GPFS

Mentor

@Roberto Sancho

IBM Spectrum Scale provides integration with Hadoop applications that use the Hadoop connector

Hadoop support for GPFS

Re: core-site for GPFS

Super Collaborator

Hi:

After do everythin I have tihs error when i start the Yarn Process

File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 273, in wait_for_dfs_directory_created
    raise Fail("DFS directory '" + dir_path + "' does not exist !")
resource_management.core.exceptions.Fail: DFS directory '/ats/done/' does not exist !






2016-07-04 09:31:47,649 - call returned (1, 'java.io.IOException: GPFSC00023E: Unable to establish communication with file system \n\tat org.apache.hadoop.fs.gpfs.GeneralParallelFileSystem.lockNativeRootAction(GeneralParallelFileSystem.java:2877)\n\tat org.apache.hadoop.fs.gpfs.GeneralParallelFileSystem.getFileStatus(GeneralParallelFileSystem.java:855)\n\tat org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)\n\tat org.apache.hadoop.fs.Globber.glob(Globber.java:252)\n\tat org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1655)\n\tat org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326)\n\tat org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235)\n\tat org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218)\n\tat org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)\n\tat org.apache.hadoop.fs.shell.Command.run(Command.java:165)\n\tat org.apache.hadoop.fs.FsShell.run(FsShell.java:287)\n\tat org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)\n\tat org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)\n\tat org.apache.hadoop.fs.FsShell.main(FsShell.java:340)\ntest: GPFSC00023E: Unable to establish communication with file system ')
2016-07-04 09:31:47,649 - Will retry 1 time(s), caught exception: DFS directory '/ats/done/' does not exist !. Sleeping for 20 sec(s)
2016-07-04 09:32:07,670 - Verifying if DFS directory '/ats/done/' exists.
2016-07-04 09:32:07,670 - call['hdfs --config /usr/hdp/current/hadoop-client/conf dfs -test -d /ats/done/'] {'user': 'yarn'}
2016-07-04 09:32:09,695 - call returned (1, 'java.io.IOException: GPFSC00023E: Unable to establish communication with file system \n\tat org.apache.hadoop.fs.gpfs.GeneralParallelFileSystem.lockNativeRootAction(GeneralParallelFileSystem.java:2877)\n\tat org.apache.hadoop.fs.gpfs.GeneralParallelFileSystem.getFileStatus(GeneralParallelFileSystem.java:855)\n\tat org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)\n\tat org.apache.hadoop.fs.Globber.glob(Globber.java:252)\n\tat org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1655)\n\tat org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326)\n\tat org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235)\n\tat org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218)\n\tat org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)\n\tat org.apache.hadoop.fs.shell.Command.run(Command.java:165)\n\tat org.apache.hadoop.fs.FsShell.run(FsShell.java:287)\n\tat org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)\n\tat org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)\n\tat org.apache.hadoop.fs.FsShell.main(FsShell.java:340)\ntest: GPFSC00023E: Unable to establish communication with file system ')
Highlighted

Re: core-site for GPFS

Mentor

@Roberto Sancho

Have you tried recreating manually the below directory with the correct permissions and restarting the YARN process

/ats/done/
Don't have an account?
Coming from Hortonworks? Activate your account here