Member since
07-18-2016
262
Posts
12
Kudos Received
21
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 6721 | 09-21-2018 03:16 AM | |
| 3229 | 07-25-2018 05:03 AM | |
| 4183 | 02-13-2018 02:00 AM | |
| 1958 | 01-21-2018 02:47 AM | |
| 38066 | 08-08-2017 10:32 AM |
03-22-2017
09:53 AM
Scenario as below 1) Have two tables A and B , having the column Account_Number. 2) If we did the join it should run fine and give result with Account_Number. 3) However this Account_number to be masked in out put ( its critical data ). 4) we have get out put, Account_number should not display all data, its should be like XXXXXX12345. Solition 1: In Ranger we can disable the Account_number colum name, if we did it join objeration will fail. Join has to run on Account_number only and this data need to masked ? Is it possible to do. Your concerns are appriciated.
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Hive
11-13-2016
08:02 AM
Thanks for your reply. Following this i did it 1) I have up-grade the amabri from 2.0 to 2.2 2) After up-grade ambari working fine, with existing 2.1 HDP. 3) Is it possible to up-grade HDP 2.1 to 2.3 Using Ambari 2.2 using Local Repository ? Kindly advise with your suggestion.
... View more
11-12-2016
08:42 AM
Please help for HDP up-grade from 2.1 to 2.3 1) Is it possible to up-grade using Ambari 2.1. 2) When i am looking for the documents Manual up-grade is there , is possible from Ambari 2,1 ? your suggestion are welcome
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
08-23-2016
02:26 AM
1 Kudo
If you have same components on Target server its fine. 1) Distcp is one off the best option for data transfer between cluster for HDFS https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_Sys_Admin_Guides/content/using_distcp.html 2) Hbase follow below link it has two method. http://hbase.apache.org/book.html#ops.backup :- Using Distcp for full dump https://hbase.apache.org/book.html#ops.snapshots :- SnapShot database for Hbase 3) Hive Metastore , check what type of database uisng like Mysql, take full export of Mysql database as below. For full dump, you can use "root" user mysqldump -u [username]-p [password][dbname]> filename.sql And if you wish to zip it at the sametime: mysqldump -u [username]-p [password][db]| gzip > filename.sql.gz You can then move this file between servers with: scp user@xxx.xxx.xxx.xxx:/path_to_your_dump/filename.sql.gz your_detination_path/ Once copied import the all objects to my sql database and start he hive server
... View more
08-22-2016
02:04 PM
Please check option as below . hdfs dfs -ls /tmp | sort -k6,7 hadoop 2.7.x ls command , the below following options available : Usage: hadoop fs -ls [-d] [-h] [-R] [-t] [-S] [-r] [-u] <args>
Options:
-d: Directories are listed as plain files.
-h: Format file sizes in a human-readable fashion (eg 64.0m instead of 67108864).
-R: Recursively list subdirectories encountered.
-t: Sort output by modification time (most recent first).
-S: Sort output by file size.
-r: Reverse the sort order.
-u: Use access time rather than modification time for display and sorting.
... View more
08-22-2016
05:52 AM
Thanks for your reply, Yep i understand the your update, yes i need install the all RPM packages, even if install without using Ambari. I am correcting my question for this and update it
... View more
08-20-2016
01:41 PM
Thanks you all for your reply. Now added the HDFS service to cluster below Error While Deploying the HDFS service 1) Connected the Internet and downloaded, installed the dependent package 2) Proceeded with HDFS Deployment, it went fine. Steps followed as below. 1) #yum install libgudev1-147-2.73.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install 1:phonon-backend-gstreamer-4.6.2-28.el6_5.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install cvs-1.11.23-16.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install redhat-lsb-compat-4.0-7.el6.centos.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install perl-ExtUtils-MakeMaker-6.55-141.el6_7.1.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install gettext-0.17-18.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install mailx-12.4-8.el6_6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install qt3-3.3.8b-30.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install ghostscript-fonts-5.50-23.2.el6.noarch --downloaddir=/etc/yum.repos.d/rpm_packag #yum install gstreamer-0.10.29-1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install 1:qt-sqlite-4.6.2-28.el6_5.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install lcms-libs-1.19-1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install iso-codes-3.16-2.el6.noarch --downloaddir=/etc/yum.repos.d/rpm_packag #yum install redhat-lsb-graphics-4.0-7.el6.centos.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install mesa-dri-drivers-11.0.7-4.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install time-1.7-38.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install xz-4.999.9-0.5.beta.20091007git.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install libXv-1.0.9-2.1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install xz-lzma-compat-4.999.9-0.5.beta.20091007git.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install poppler-utils-0.12.4-10.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install ghostscript-8.70-21.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install redhat-lsb-core-4.0-7.el6.centos.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install nc --downloaddir=/etc/yum.repos.d/rpm_packag #yum install libmng-1.0.10-4.1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install xml-common-0.6.3-33.el6.noarch --downloaddir=/etc/yum.repos.d/rpm_packag #yum install mesa-dri1-drivers-7.11-8.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install perl-Test-Simple-0.92-141.el6_7.1.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install poppler-data-0.4.0-1.el6.noarch --downloaddir=/etc/yum.repos.d/rpm_packag #yum install man-1.6f-32.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install libvisual-0.4.0-10.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install 1:qt-4.6.2-28.el6_5.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install tmpwatch-2.9.16-6.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install gdbm-devel-1.8.0-39.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install 1:perl-ExtUtils-ParseXS-2.2003.0-141.el6_7.1.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install urw-fonts-2.4-11.el6.noarch --downloaddir=/etc/yum.repos.d/rpm_packag #yum install 1:qt-x11-4.6.2-28.el6_5.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install mesa-libGLU-11.0.7-4.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install redhat-lsb-4.0-7.el6.centos.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install perl-Test-Harness-3.17-141.el6_7.1.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install openjpeg-libs-1.3-11.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install libXxf86vm-1.1.3-2.1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install foomatic-4.0.4-5.el6_7.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install foomatic-db-ppds-4.0-7.20091126.el6.noarch --downloaddir=/etc/yum.repos.d/rpm_packag #yum install 1:libtheora-1.1.0-2.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install gstreamer-plugins-base-0.10.29-2.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install 1:cups-1.4.2-74.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install gstreamer-tools-0.10.29-1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install redhat-lsb-printing-4.0-7.el6.centos.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install bc-1.06.95-1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install perl-CGI-3.51-141.el6_7.1.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install liboil-0.3.16-4.1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install mesa-dri-filesystem-11.0.7-4.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install 4:perl-devel-5.10.1-141.el6_7.1.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install libXt-1.1.4-6.1.el6.x86_64: failure --downloaddir=/etc/yum.repos.d/rpm_packag #yum install cdparanoia-libs-10.2-5.1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install db4-cxx-4.7.25-20.el6_7.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install at-3.1.10-48.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install db4-devel-4.7.25-20.el6_7.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install foomatic-db-filesystem-4.0-7.20091126.el6.noarch --downloaddir=/etc/yum.repos.d/rpm_packag #yum install patch-2.6-6.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install pax-3.4-10.1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install poppler-0.12.4-10.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install foomatic-db-4.0-7.20091126.el6.noarch --downloaddir=/etc/yum.repos.d/rpm_packag #yum install ed-1.1-3.3.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install portreserve-0.0.4-11.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install snappy-1.1.0-1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag #yum install snappy-devel --downloaddir=/etc/yum.repos.d/rpm_packag # #yum install poppler-data-0.4.0-1.el6.noarch --downloaddir=/etc/yum.repos.d/rpm_packag # # #After above still some are missing . installed one by one # # #yum install urw-fonts-2.4-11.el6.noarch # #yum install xz-lzma-compat-4.999.9-0.5.beta.20091007git.el6.x86_64 # #yum install redhat-lsb-graphics-4.0-7.el6.centos.x86_64 #yum install xz-4.999.9-0.5.beta.20091007git.el6.x86_64 #yum install ghostscript-8.70-21.el6.x86_64 #yum install redhat-lsb-core-4.0-7.el6.centos.x86_64 #yum install nc-1.84-24.el6.x86_64 #yum install man-1.6f-32.el6.x86_64 #yum install tmpwatch-2.9.16-6.el6.x86_64 #yum install poppler-utils-0.12.4-10.el6.x86_64 #yum install pax-3.4-10.1.el6.x86_64 #yum install openjpeg-libs-1.3-11.el6.x86_64 #yum install foomatic-4.0.4-5.el6_7.x86_64 #yum install foomatic-db-ppds-4.0-7.20091126.el6.noarch #yum install redhat-lsb-printing-4.0-7.el6.centos.x86_64 #yum install foomatic-db-4.0-7.20091126.el6.noarch #yum install bc-1.06.95-1.el6.x86_64 #yum install perl-CGI-3.51-141.el6_7.1.x86_64 #yum install ed-1.1-3.3.el6.x86_64 #yum install libXt-1.1.4-6.1.el6.x86_64 #yum install at-3.1.10-48.el6.x86_64 #yum install time-1.7-38.el6.x86_64 #yum install foomatic-db-filesystem-4.0-7.20091126.el6.noarch #yum install patch-2.6-6.el6.x86_64 #yum install redhat-lsb-4.0-7.el6.centos.x86_64 #yum install poppler-0.12.4-10.el6.x86_64 #yum install 1:cups-1.4.2-74.el6.x86_64 #yum install redhat-lsb-compat-4.0-7.el6.centos.x86_64 #yum install portreserve-0.0.4-11.el6.x86_64 #cd /etc/yum.repos.d/rpm_packag #rpm -Uvh *.rpm #cd /etc/yum.repos.d/rpm_packag #rpm -Uvh *.rpm One of installed packages example [root@centos ~]# yum install foomatic-4.0.4-5.el6_7.x86_64
Loaded plugins: fastestmirror
Setting up Install Process
Loading mirror speeds from cached hostfile
* base: buaya.klas.or.id
* extras: centos.exabytes.com.my
* updates: buaya.klas.or.id
Resolving Dependencies
--> Running transaction check
---> Package foomatic.x86_64 0:4.0.4-5.el6_7 will be installed
--> Processing Dependency: foomatic-db for package: foomatic-4.0.4-5.el6_7.x86_64
--> Processing Dependency: cups for package: foomatic-4.0.4-5.el6_7.x86_64
--> Running transaction check
---> Package cups.x86_64 1:1.4.2-74.el6 will be installed
--> Processing Dependency: portreserve for package: 1:cups-1.4.2-74.el6.x86_64
---> Package foomatic-db.noarch 0:4.0-7.20091126.el6 will be installed
--> Processing Dependency: foomatic-db-ppds = 4.0-7.20091126.el6 for package: foomatic-db-4.0-7.20091126.el6.noarch
--> Processing Dependency: foomatic-db-filesystem = 4.0-7.20091126.el6 for package: foomatic-db-4.0-7.20091126.el6.noarch
--> Running transaction check
---> Package foomatic-db-filesystem.noarch 0:4.0-7.20091126.el6 will be installed
---> Package foomatic-db-ppds.noarch 0:4.0-7.20091126.el6 will be installed
---> Package portreserve.x86_64 0:0.0.4-11.el6 will be installed
--> Finished Dependency Resolution Dependencies Resolved ================================================================================================================================================================================================
Package Arch Version Repository Size
================================================================================================================================================================================================
Installing:
foomatic x86_64 4.0.4-5.el6_7 base 251 k
Installing for dependencies:
cups x86_64 1:1.4.2-74.el6 base 2.3 M
foomatic-db noarch 4.0-7.20091126.el6 base 980 k
foomatic-db-filesystem noarch 4.0-7.20091126.el6 base 4.4 k
foomatic-db-ppds noarch 4.0-7.20091126.el6 base 19 M
portreserve x86_64 0.0.4-11.el6 base 23 k Transaction Summary
================================================================================================================================================================================================
Install 6 Package(s) Total download size: 23 M
Installed size: 44 M
Is this ok [y/N]: y Downloading Packages:
http://buaya.klas.or.id/centos/6.8/os/x86_64/Packages/cups-1.4.2-74.el6.x86_64.rpm: [Errno 12] Timeout on http://buaya.klas.or.id/centos/6.8/os/x86_64/Packages/cups-1.4.2-74.el6.x86_64.rpm: (28, 'connect() timed out!')
Trying other mirror.
(1/6): cups-1.4.2-74.el6.x86_64.rpm | 2.3 MB 00:01
(2/6): foomatic-4.0.4-5.el6_7.x86_64.rpm | 251 kB 00:00
(3/6): foomatic-db-4.0-7.20091126.el6.noarch.rpm | 980 kB 00:00
(4/6): foomatic-db-filesystem-4.0-7.20091126.el6.noarch.rpm | 4.4 kB 00:00
(5/6): foomatic-db-ppds-4.0-7.20091126.el6.noarch.rpm | 19 MB 00:09
(6/6): portreserve-0.0.4-11.el6.x86_64.rpm | 23 kB 00:00
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 558 kB/s | 23 MB 00:41 Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : foomatic-db-filesystem-4.0-7.20091126.el6.noarch 1/6
Installing : portreserve-0.0.4-11.el6.x86_64 2/6
Installing : 1:cups-1.4.2-74.el6.x86_64 3/6
Installing : foomatic-db-ppds-4.0-7.20091126.el6.noarch 4/6
Installing : foomatic-db-4.0-7.20091126.el6.noarch 5/6
Installing : foomatic-4.0.4-5.el6_7.x86_64 6/6
Verifying : foomatic-db-filesystem-4.0-7.20091126.el6.noarch 1/6
Verifying : foomatic-4.0.4-5.el6_7.x86_64 2/6
Verifying : portreserve-0.0.4-11.el6.x86_64 3/6
Verifying : foomatic-db-4.0-7.20091126.el6.noarch 4/6
Verifying : foomatic-db-ppds-4.0-7.20091126.el6.noarch 5/6
Verifying : 1:cups-1.4.2-74.el6.x86_64 6/6 Installed:
foomatic.x86_64 0:4.0.4-5.el6_7 Dependency Installed:
cups.x86_64 1:1.4.2-74.el6 foomatic-db.noarch 0:4.0-7.20091126.el6 foomatic-db-filesystem.noarch 0:4.0-7.20091126.el6 foomatic-db-ppds.noarch 0:4.0-7.20091126.el6
portreserve.x86_64 0:0.0.4-11.el6 Complete!
[root@centos ~]#
... View more
08-19-2016
06:41 AM
2 Kudos
You can do by as below. 1) Create the user at OS level and assign the group and owner to user. 2) Grant all permission to that user only in Hive as below. Need to login with root use
set role admin; #hive hive>set role admin; (admin role should have insert update and delete , which ever you want to give) Hive>use database; Hive>grant all on table table_name to role os_user_name with grant option; By this "os_user_name" only can able to access, other wont able to see the tables. It has worked me at Hive permission level at USER.
... View more
08-18-2016
12:13 PM
ok i have faced same issue while Installing Ambari, please follow below link it may be help. 1) It has some dependent packages like unzip those and other as well 2)https://community.hortonworks.com/questions/49909/ambari-insall-start-test-failed-21.html# Connect to Internet download below rpm packages and proceed with installation. bash# install mpfr-2.4.1 tzdata-2016f unzip bash# install tzdata-2016f bash# install libgomp ppl bash# install libgc cpp gcc # yum install python-devel-2.6.6-64.el6.x86_64 if the above steps are help, please close the request.
... View more
08-18-2016
12:11 PM
If small files are there in HDFS storage is it going to occupy the Namenode Heap Memory ? 1) What is work around for total Name Node Memory Heap size usage. 2) Is any other things need to consider, YARN jobs ...etc your concerns are welcome.
... View more