Member since
09-18-2018
13
Posts
0
Kudos Received
0
Solutions
11-01-2018
03:51 PM
Hello everybody, For ex. I have 1 TB of HDFS disk usage . I want to restrict the access to all the other disk drives /data/ /apps/ /user/ /tmp/ /path1 /path2 In the above directories , I want my development team to use only 30 GB in /data where they can create/delete/modify the folders . If they want to use any other folder than /data . they should ask the operation team for permission to access the folders if not they should get permission denied error message.. I can exclude /tmp/,/user with 10Gb and 5GB respectiverly . What tools are good for this and how can we achieve this scenario. Sorry for the basic question . @Geoffrey Shelton Okot Sir you could also help me in this regard. @Jay Kumar SenSharma
... View more
Labels:
- Labels:
-
Apache Hadoop
10-30-2018
03:22 AM
I want to setup Ranger with Policies on HDFS,Hive,HBase on HDP 2.6.3. i have never done this before. Can anyone share the step by step commands/procedure on how to go about it? i am totally new to Ranger. i am using Centos.
... View more
Labels:
- Labels:
-
Apache Ranger
10-29-2018
06:24 PM
Hello everybody, My disk space is consuming more and more in HDFS and when I dig , I can see HBASE took 400GB of Old Wals .However If I look at Wals . it is almost empty . So Can I remove the complete OLDWals from the machine . Does it affect any data. I have read WALs are used only when there is crash in memory before writing it to disk . Still do not want to take the risk . $ hdfs dfs -du -h /apps/hbase/data 0/apps/hbase/data/.tmp ©37 0/apps/hbase/data/MasterProcWALs 12.6 K /apps/hbase/data/WALs ---- ONLY KBs 7.5 G /apps/hbase/data/archive 0/apps/hbase/data/corrupt 52.9 T /apps/hbase/data/data 42/apps/hbase/data/hbase.id 7/apps/hbase/data/hbase.version 379.5 G /apps/hbase/data/oldWALs -- Large data My Question is if I set any parameter in Ambari to auto purge on regular interval and how can I check the replication is enabled or not . hbase.master.logcleaner.ttl where to set if in case i do not want to delete it .
... View more
Labels:
- Labels:
-
Apache HBase
10-28-2018
04:45 PM
Simple and Cool. However the table is updated every other hour . It is taking very long time for 900GB to CTAS. The thing is to store TB of data for the first time and then 100GB daily incremental like insert/update/delete in HBase and to make it available for Business analysts . It is taking more than 40+ minutes to retrieve a single query. Loading the data in HBASE takes only 10 to 20 minutes.Any other approach Shu.Kindly give me some spark
... View more
10-28-2018
03:24 PM
@Shu Do you have steps to export the HBASE to Hive ORC table. I have tried the performance tuning already it didnt come up properly. Thank you very much helping
... View more
10-28-2018
07:28 AM
Hello Everybody, We have HBASE Table with around 10 Million records and when we integrate with Hive . It is taking more than 30 minutes to produce the results . If we try to do it in HBase it is fast. Is there anyway to manage the situation . Or 1.can I export all the data from HBase to Hive 2.How can we avoid full scan in HBase tables from Hive . Sorry for the basic questions
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache HBase
-
Apache Hive
10-15-2018
04:58 PM
Hi All, Nifi is writing the data into PUTHDFS as JSON files. The file permissions and owner of the files are as follows File_name1 rwx-- nifi:hadoop There are two things which we don't understand . We do not have Nifi LDAP integrated so far . it runs on default user Nifi. I cannot ask nifi to change the permission while writing the data or change the owner which is accessible to hive . We are getting permission denied errors while reading the data from hive . We cannot change the permissions in nifi as it is not added in superusers group . We did tried to add nifi as superusers group . but failed to change the owner information . How can I write the files which is accessible for both hive and Nifi or how can I change the owner/permissions in nifi after writing the data into HDFS .
... View more
Labels:
- Labels:
-
Apache NiFi
10-12-2018
10:02 PM
This post is currently awaiting moderation. If you believe this to be in error, contact a system administrator. https://community.hortonworks.com/questions/225074/nifi-to-write-data-into-hdfs-as-different-user.html
... View more
- Tags:
- Hadoop Core
- hdpcd
10-04-2018
07:39 PM
Hi All, Is there any way to find who has restarted the services through Ambari .In our team six people working on different shifts and teams . It is very difficult to track who has initiated the service restart . For example . Some one restarted Namenode and we were unable to find who has initiated the restart via ambari . Attached the ideal image of Ambari services restart . 10377-timedout.png We have searched the logs with all 6 userid . The log returned nothing with the userids . Is there any alternative ways or how we can easily identify who has done the restart from Ambari .
... View more
Labels:
- Labels:
-
Apache Ambari
10-03-2018
07:55 PM
Hi All, We have 10 node cluster. We have only few teams are using the cluster at the moment.What do you suggest fresh install or Upgrade . If so why ? Could you please explain the pain points of both . What is the best practice here . Clean installation of OS, HDP , HDF or upgrade of HDP and HDF . If fresh install . we will take a back of all the data to another machine and reinstall everything .
... View more