Member since
04-13-2016
422
Posts
150
Kudos Received
55
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1934 | 05-23-2018 05:29 AM | |
| 4970 | 05-08-2018 03:06 AM | |
| 1685 | 02-09-2018 02:22 AM | |
| 2716 | 01-24-2018 08:37 PM | |
| 6172 | 01-24-2018 05:43 PM |
07-17-2016
07:43 PM
1 Kudo
@Ahmed Bilal
You can copy the data to edge node and from there you can do winscp hadoop fs -get: Copies/Downloads files from HDFS to the local file system <code># Usage:# hdfs dfs -get <hdfs_src> <localdst>
or
hdfs dfs [-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
or
hdfs dfs [-moveToLocal <src> <localdst>]
# Example: hdfs dfs -get/user/hadoop/dir1/popularNames.txt /home/
2. By giving this hdfs fsck we can know where are block are located. hdfs fsck /user/root/tableDirName/filename -files -locations -blocks -racks 3. Even you can use FileView UI to download data which is much user-friendly
... View more
07-16-2016
04:01 AM
3 Kudos
@ripunjay godhani
1. Edge nodes are the interface between the Hadoop cluster and the outside network. 2. They’re also often used as staging areas for data being transferred into the Hadoop cluster. As such, Oozie, Pig, Sqoop, and management tools such as Hue and Ambari run well there. 3. Yes, it's always better. 4. Yes, it doesn't store any hdfs data, it used for accessing cluster and processing/accessing the data. 5. Yes, it's always better to be outside of secured VLAN. Link might help you more.
... View more
07-15-2016
03:21 PM
2 Kudos
@ANSARI FAHEEM AHMED Sandeep Nemuri is correct, for best practice and in order to avoid confusing I request to change YARN,your resource manager to rm01,rm02 In yarn-site.xml <property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm01,rm02</value>
</property> In Ambari, YARN--> custom yarn site --> yarn.resourcemanager.ha.rm-ids = rm01,rm02 Hope this helps you.
... View more
07-15-2016
04:55 AM
@ANSARI FAHEEM AHMED HADOOP_HEAPSIZE sets the JVM heap size for all Hadoop project servers such as HDFS, YARN, and MapReduce. HADOOP_HEAPSIZE is an integer passed to the JVM as the maximum memory (Xmx) argument. For example: HADOOP_HEAPSIZE=256 HADOOP_NAMENODE_OPTS is specific to the NameNode and sets all JVM flags, which must be specified.HADOOP_NAMENODE_OPTS overrides the HADOOP_HEAPSIZE Xmx value for the NameNode. For example: HADOOP_NAMENODE_OPTS=SHARED_HADOOP_NAMENODE_OPTS="-server -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hadoop/$USER/hs_err_pid%p.log -XX:NewSize=50m -XX:MaxNewSize=100m -XX:PermSize=128m -XX:MaxPermSize=256m -Xloggc:/var/log/hadoop/$USER/gc.log-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xms250m -Xmx250m -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT" Both HADOOP_NAMENODE_OPTS and HADOOP_HEAPSIZE are stored in /etc/hadoop/conf/hadoop-env.sh.
... View more
07-15-2016
04:13 AM
2 Kudos
@ANSARI FAHEEM AHMED NameNode heap size depends on many factors such as the number of files, the number of blocks, and the load on the system. The settings in the referenced table below should work for typical Hadoop clusters where the number of blocks is very close to the number of files (generally the average ratio of number of blocks per file in a system is 1.1 to 1.2). Some clusters might require further tweaking of the following settings. Also, it is generally better to set the total Java heap to a higher value. http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/ref-80953924-1cbf-4655-9953-1e744290a6c3.1.html
... View more
07-14-2016
04:03 AM
Hi Team, When I'm trying to view HAWQ automated/manual with Ambari in HDP 2.5 it's throwing me page not found the error. Can someone please provide me airticle?
... View more
Labels:
07-14-2016
03:29 AM
@Rey L May I know is Ranger installed in your cluster?
... View more
07-11-2016
03:07 PM
@Sunile Manjee Okay, Even after setting this property, still do I need to register jar files? Because I'm getting the same error even after setting the property.
... View more
07-08-2016
06:19 PM
@Sunile Manjee Thanks for the quick response. Is this the correct way? mkdir /usr/hdp/current/hive-server2/auxlib/ copy required jars to /usr/hdp/current/hive-server2/auxlib/ loaction. Added below property to hive-site.xml <property>
<name>hive.aux.jars.path</name>
<value>/usr/hdp/current/hive-server2/auxlib/</value>
</property>
... View more
07-08-2016
02:14 PM
Hi, In HIVE CLI we can directly add jar files to their session but in Beeline we need to have admin permissions to add the jar file to the session as we are using SQL Standard Based Hive Authorization.
1. What is the best way to grant privileges to Beeline users in order for them to add jars/files? We don't want to grant admin privileges to all beeline users. 2. Is there any best practices in regarding the use of adding jars from beeline? Thanks in advance.
... View more
Labels:
- Labels:
-
Apache Hive