Member since
11-03-2017
94
Posts
13
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
5180 | 04-11-2018 09:48 AM | |
1838 | 12-08-2017 01:07 PM | |
2364 | 12-01-2017 04:23 PM | |
11683 | 11-06-2017 04:08 PM |
02-26-2018
12:29 PM
@Jay Kumar SenSharma I think that the telnet service is not installed in Muy cluster, could I try by ssh instead??
... View more
02-26-2018
12:21 PM
I have two clusters A(unsecure) and B(secure): I try to list directories of the cluster A from the cluster B: I did this: cluster B, master node machine# hadoop fs -ls hdfs://Cluster A IP address of master node:8020/tmp/ I got this error : 18/02/26 12:54:23 WARN ipc.Client: Failed to connect to server: 10.166.54.12/10.166.54.12:8020: try once and fail.
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:650)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:745)
at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618)
at org.apache.hadoop.ipc.Client.call(Client.java:1449)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:816)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2158)
at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1423)
at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1419)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1419)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:252)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1674)
at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326)
at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235)
at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218)
at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103)
at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:297)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:350)
18/02/26 12:54:23 WARN retry.RetryInvocationHandler: Exception while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over null. Not retrying because try once and fail.
java.net.ConnectException: Call From slzuyd5hmn03.yres.ytech/10.166.60.143 to 10.166.54.12:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
... View more
Labels:
- Labels:
-
Apache Falcon
-
Apache Hadoop
02-20-2018
11:39 AM
ok thanks @Jay Kumar SenSharma
... View more
02-20-2018
10:53 AM
@Jay Kumar SenSharma Do I need to run the both command to stop hive service, isn't there one line command that can do this??
... View more
02-20-2018
10:44 AM
@Jay Kumar SenSharma do I need to run the both command to stop hive, isn't there one command that can do this??
... View more
02-20-2018
10:40 AM
@Jay Kumar SenSharma is there some alternative to stop the whole service "hive service" on one command??
... View more
02-20-2018
10:11 AM
I need to create a script shell that stops the Hive Metastore and Hive Server2 from any node of cluster and I need to know where the hive is hosted in My cluster in order to run this command: ssh nodename:ps aux | awk '{print $1,$2}' | grep hive | awk '{print
$2}' | xargs kill >/dev/null 2>&1 how can I know where nodes hive is hosted?
... View more
Labels:
02-05-2018
09:33 AM
@Malay Sharma The HDFS is a distributed file system. hadoop is mainly designed for batch processing of large volume of data. The default data block size of HDFS is 128 MB. When file size is significantly smaller than the block size the efficiency degrades. Mainly there are two reasons for producing small files:
Files could be the piece of a larger logical file. Since HDFS has only recently supported appends, these unbounded files are saved by writing them in chunks into HDFS. Another reason is some files cannot be combined together into one larger file and are essentially small. e.g. – A large corpus of images where each image is a distinct file. To understand HDFS block size in more detail, I'd recommend reviewing a few good stackoverflow questions:http://stackoverflow.com/questions/13012924/large-block-size-in-hdfs-how-is-the-unused-space-accounted-for http://stackoverflow.com/questions/19473772/data-block-size-in-hdfs-why-64mb For your disk/filesystem recommendations take a look here: https://community.hortonworks.com/content/kbentry/14508/best-practices-linux-file-systems-for-hdfs.html Hope that all helps! duplicate topic.
... View more
01-22-2018
02:42 PM
I would know if there is an order to respect when we start services on HDP in Ambari
... View more
Labels:
01-18-2018
08:48 PM
that is not FREE I guess, I m struggling with this for 2 hours, I can find free solution four this
... View more