Disk usage in HDFS

lenu · ‎11-01-2018

Hello everybody, For ex. I have 1 TB of HDFS disk usage . I want to restrict the access to all the other disk drives /data/ /apps/ /user/ /tmp/ /path1 /path2 In the above directories , I want my development team to use only 30 GB in /data where they can create/delete/modify the folders . If they want to use any other folder than /data . they should ask the operation team for permission to access the folders if not they should get permission denied error message.. I can exclude /tmp/,/user with 10Gb and 5GB respectiverly . What tools are good for this and how can we achieve this scenario. Sorry for the basic question . @Geoffrey Shelton Okot Sir you could also help me in this regard. @Jay Kumar SenSharma

lenu · ‎10-30-2018

I want to setup Ranger with Policies on HDFS,Hive,HBase on HDP 2.6.3. i have never done this before. Can anyone share the step by step commands/procedure on how to go about it? i am totally new to Ranger. i am using Centos.

lenu · ‎10-29-2018

Hello everybody, My disk space is consuming more and more in HDFS and when I dig , I can see HBASE took 400GB of Old Wals .However If I look at Wals . it is almost empty . So Can I remove the complete OLDWals from the machine . Does it affect any data. I have read WALs are used only when there is crash in memory before writing it to disk . Still do not want to take the risk . $ hdfs dfs -du -h /apps/hbase/data 0/apps/hbase/data/.tmp ©37 0/apps/hbase/data/MasterProcWALs 12.6 K /apps/hbase/data/WALs ---- ONLY KBs 7.5 G /apps/hbase/data/archive 0/apps/hbase/data/corrupt 52.9 T /apps/hbase/data/data 42/apps/hbase/data/hbase.id 7/apps/hbase/data/hbase.version 379.5 G /apps/hbase/data/oldWALs -- Large data My Question is if I set any parameter in Ambari to auto purge on regular interval and how can I check the replication is enabled or not . hbase.master.logcleaner.ttl where to set if in case i do not want to delete it .

lenu · ‎10-28-2018

Simple and Cool. However the table is updated every other hour . It is taking very long time for 900GB to CTAS. The thing is to store TB of data for the first time and then 100GB daily incremental like insert/update/delete in HBase and to make it available for Business analysts . It is taking more than 40+ minutes to retrieve a single query. Loading the data in HBASE takes only 10 to 20 minutes.Any other approach Shu.Kindly give me some spark

lenu · ‎10-28-2018

@Shu Do you have steps to export the HBASE to Hive ORC table. I have tried the performance tuning already it didnt come up properly. Thank you very much helping

lenu · ‎10-28-2018

Hello Everybody, We have HBASE Table with around 10 Million records and when we integrate with Hive . It is taking more than 30 minutes to produce the results . If we try to do it in HBase it is fast. Is there anyway to manage the situation . Or 1.can I export all the data from HBase to Hive 2.How can we avoid full scan in HBase tables from Hive . Sorry for the basic questions

lenu · ‎10-15-2018

Hi All, Nifi is writing the data into PUTHDFS as JSON files. The file permissions and owner of the files are as follows File_name1 rwx-- nifi:hadoop There are two things which we don't understand . We do not have Nifi LDAP integrated so far . it runs on default user Nifi. I cannot ask nifi to change the permission while writing the data or change the owner which is accessible to hive . We are getting permission denied errors while reading the data from hive . We cannot change the permissions in nifi as it is not added in superusers group . We did tried to add nifi as superusers group . but failed to change the owner information . How can I write the files which is accessible for both hive and Nifi or how can I change the owner/permissions in nifi after writing the data into HDFS .

lenu · ‎10-12-2018

This post is currently awaiting moderation. If you believe this to be in error, contact a system administrator. https://community.hortonworks.com/questions/225074/nifi-to-write-data-into-hdfs-as-different-user.html

lenu · ‎10-04-2018

Hi All, Is there any way to find who has restarted the services through Ambari .In our team six people working on different shifts and teams . It is very difficult to track who has initiated the service restart . For example . Some one restarted Namenode and we were unable to find who has initiated the restart via ambari . Attached the ideal image of Ambari services restart . 10377-timedout.png We have searched the logs with all 6 userid . The log returned nothing with the userids . Is there any alternative ways or how we can easily identify who has done the restart from Ambari .

lenu · ‎10-03-2018

Hi All, We have 10 node cluster. We have only few teams are using the cluster at the moment.What do you suggest fresh install or Upgrade . If so why ? Could you please explain the pain points of both . What is the best practice here . Clean installation of OS, HDP , HDF or upgrade of HDP and HDF . If fresh install . we will take a back of all the data to another machine and reinstall everything .

Online	Offline
Last Visited	‎12-26-2018 12:29 PM

Date Registered	‎09-18-2018 05:33 AM
Last Visited	‎12-26-2018 12:29 PM
Total Messages Posted	13

Disk usage in HDFS

Ranger step by step document to install and setup ...

HBASE OldWals not deleting

Re: Underlying HBASE Table is taking 30+ minutes f...

Re: Underlying HBASE Table is taking 30+ minutes f...

Underlying HBASE Table is taking 30+ minutes for s...

Nifi to write data into HDFS as different user

Cannot post questions

Find the user who restarted any services

Upgrade or Fresh install