Member since
03-14-2016
4721
Posts
1111
Kudos Received
874
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2717 | 04-27-2020 03:48 AM | |
| 5276 | 04-26-2020 06:18 PM | |
| 4443 | 04-26-2020 06:05 PM | |
| 3567 | 04-13-2020 08:53 PM | |
| 5376 | 03-31-2020 02:10 AM |
06-12-2017
05:30 PM
@Sami Ahmad
Are you sure that your Proxy Host and Port is the same that you are using in your "/etc/yum.conf" ? (I just gave an example, but in your case the proxy host & port might be different). You need to check first if your browser is able to access that file or not and if yes then check if your browser is using any proxy? Following is just example host & port. # cat /etc/yum.conf | grep proxy
proxy=http://yourproxy.test.com:808
... View more
06-12-2017
05:21 PM
@Sami Ahmad
Are you accessing the mentioned URL using browser? Do you have any proxy_url (proxy host/port) configured on your browser? If yes, then you should make sure that from the host where you have that repo it is configured to be accessible via the same proxy. For example if you do a curl command from the host where you have that repo then what happens? # wget http://hadoop1.tolls.dot.state.fl.us/AMBARI-2.4.2.0/centos6/2.4.2.0-136/repodata/repomd.xml
OR
# curl http://hadoop1.tolls.dot.state.fl.us/AMBARI-2.4.2.0/centos6/2.4.2.0-136/repodata/repomd.xm . If it fails then you should check if you have any proxy configured in the file "/etc/yum.conf"
Example:
# cat /etc/yum.conf | grep proxy
proxy=http://yourproxy.test.com:8080 .
... View more
06-12-2017
05:00 PM
@Sami Ahmad Looks like you have another repo file present inside your "/etc/yum.repos.d" directory which has the following Host entry in it: http://hadoop1.tolls.dot.state.fl.us . Can you please try to grep that host name from where is it coming from? # cd /etc/yum.repos.d
# grep 'hadoop1.tolls.dot.state.fl.us' *.repo
(OR)
# find . -name "*.repo" | xargs grep -i "hadoop1.tolls.dot.state.fl.us" . In the output of the above command if you find any such hostname then move that repo file somewhere else. Because that seems to be pointing to an Incorrect Ambari repo.
... View more
06-08-2017
08:16 AM
@Bala Vignesh N V Edge nodes are not designed to store the data. The Clients running inside the Edge Nodes (like HDFS client) are actually responsible for performing the operations like copy/put of files to HDFS (metadata will be stored on NameNode and the DataNodes will actually store the data/content of the file. . Following is the content from an old doc: https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html A client request to create a file does not reach the NameNode immediately. In fact, initially the HDFS client caches the file data into a temporary local file. Application writes are transparently redirected to this temporary local file. When the local file accumulates data worth over one HDFS block size, the client contacts the NameNode. The NameNode inserts the file name into the file system hierarchy and allocates a data block for it. The NameNode responds to the client request with the identity of the DataNode and the destination data block. Then the client flushes the block of data from the local temporary file to the specified DataNode. When a file is closed, the remaining un-flushed data in the temporary local file is transferred to the DataNode. The client then tells the NameNode that the file is closed. At this point, the NameNode commits the file creation operation into a persistent store. If the NameNode dies before the file is closed, the file is lost.
. There are some good information available on edge nodes which can be found in the following links. http://www.dummies.com/programming/big-data/hadoop/edge-nodes-in-hadoop-clusters/ https://dwbi.org/etl/bigdata/187-set-up-client-node-gateway-node-in-hadoop-cluster See this post making the good point of installing hadoop binaries via Ambari so they are always up to date with the rest of the cluster.https://community.hortonworks.com/questions/39568/how-to-create-edge-node-for-kerberized-cluster.html
... View more
06-08-2017
07:24 AM
@jack jack
First you should open a new HCC thread that way each individual queries are tracked separately.
... View more
06-08-2017
07:04 AM
@Perry Huang The following error (java.lang.NoSuchMethodError) usually indicates ... Other version of JAR (Jar version mismatch) were being used in the classpath. java.lang.NoSuchMethodError: org.apache.hadoop.yarn.logaggregation.LogCLIHelpers.dumpAllContainersLog
... View more
06-08-2017
07:02 AM
2 Kudos
@Bala Vignesh N V Edge node refers to a dedicated node (machine) where no Hadoop services are running, and where you install only so-called Hadoop clients (hdfs, Hive, HBase etc. clients). In your case your BI tool will also play a role of a Hadoop client. A client means that only respective component client libraries and scripts will be installed, together with its config files. If you change config through Ambari, then Ambari will automatically refresh config files on the edge node as well. In a small, test cluster without an edge node you can select one node where Hadoop services are running (for example, a master node) to play a role of your edge node. (In a large cluster with many users there are usually multiple edge nodes.) As the "edge node folder" you can use any folder on the edge node you decide to use. Usually we execute Sqoop, hdfs, Oozie, ...etc commands from an edgenode.
Edgenode is a client-facing machine that has all client tools to
operate on a cluster. It is not a good idea to use NameNode or other HDP
components as your edgenode. Typically you'd want a separate node
designated just for client access. Worker nodes make up the majority of virtual machines and perform the job of storing the data and running computations. Worker nodes usually runs both a DataNode and NodeManager ..etc kind of services. https://community.hortonworks.com/questions/87884/which-node-to-use.html
... View more
06-08-2017
06:54 AM
@Rahul Gupta
You can set: NameNode new generation size = 768
NameNode maximum new generation size = 768 . Permanent Generation values you can leave default. Because from JDK 1.8 onwards there is no concept of Permanent Generation it is replaced with the MetaSpace. https://dzone.com/articles/java-8-permgen-metaspace
... View more
06-08-2017
05:57 AM
@Rahul Gupta You can refer to the following Doc to know the NameNode Heap recommendations based on number of files; https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/ref-80953924-1cbf-4655-9953-1e744290a6c3.1.html .
... View more
06-08-2017
05:30 AM
@hardik desai
As the issue is intermittent so this may not be script related issue. However based on the error pattern it mostly look the driver is trying to create a new connection and the DBMS breaking
the socket between it and the driver, at a very early stage of the
process.
This has to be either a network issue, or more likely a DBMS issue like
if there are too many connection requests at once or in a short burst,
the DBMS listener process gets overloaded and severs some of the
incoming
sockets. - So we will need to check why the connection was not established. - Due to Load on the DB (Check the DB logs of the same timestamp) - Due to Load on your machine. ( Check the SAR report to find the historical data of the OS of the mentioned timestamp) - N/W drops also might be a reason. Check the "/var/log/messages" to see if any thing unusual happened during the time of error.
... View more