I am copying a large number of small files (hl7 message files) from Linux local storage to hdfs. I wonder whether this is a performance difference between copying files one by one (though a script) or just using one statement like "hadoop fs -put ./* /hadoop_path".   Additional background info: some files have space in their file name, if I use the command "hadoop fs -put ./* /hadoop_path", I got the error "put: unexpected URISyntaxException" for those files. If there is no performance difference, I would just copy file one at a time and my script replaces the space with "%20". Otherwise, I have to rename all files, replacing spaces with underscores, and then use batch copy. ... View more

My HBase cluster uses basic/simple authentication (no Kerberos). All my test cases below are done on one node. On that node, a user "testuser" has it primary group as "developer".   By hbase superuser hbase(main):005:0> grant '@developer','RWXCA','@test' Took 0.1722 seconds   By testuser hbase(main):007:0> list test:table1 1 row(s) Took 0.0500 seconds hbase(main):014:0> scan 'test:table1' ROW COLUMN+CELL ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'testuser' (table=test:table1, action=READ)   Why does testuser, who has the full permission on the "test" namespace, get the permission error?   In addition, I feel that, when I run the grant command grant '@developer','RWXCA','@test' The permission is somehow granted at the namespace level, but not on every table inside the namespace "test". To prove that, as the "testuser", I can create a new table 'test:table2', add data to that new table, and run a scan on that new table successfully. But still I cannot run the scan on another table in the "test" namespace.   I understand this is a strange case. I wonder anyone else experienced the same problem. Right now my workround is to grant the permission to "testuser" directly. This way testuser can scan the table without permission error. ... View more

I am running an HDP cluster without Knox, Ranger, or Keberos enabled. All Ambari users need to log in with a password and each node has local user accounts with a password. Question #1: what level of security does this cluster have, - no security or basic security? Question #2: Because there is no centralized security management, how does HDP manage the HDFS permissions?  The user/group accounts used in HDFS permissions are actually local users and authenticated by the local Linux OS, right? Question #3: I used the command below to change the owner of an HDFS folder to the ambari user admin. I understand that hadoop is a group name, but it is not an ambari group. So how does this work - a combination of ambari user account and an linux group?   hdfs dfs -chown admin:hadoop /user/admin     ... View more

I create two tables with the same name, one in Hive and the other in HBase. In Atlas, I found the hive table using the search: Search by Type: hive_table Search by Text: the table name However, I could not find the hbase table using the same search criteria Search by Type: hbase_table Search by Text: the table name If I just search by type hbase_table, the hbase table is in the result. What do I miss? ... View more

I am using HDP 3.1. I deleted a directory of an Hbase table by mistake. The deleted hdfs directory is /apps/hbase/data/data/testing/testtable1. The table has 12 regions. After the deletion, I rebooted hbase and the Ambari Hbase UI shows 12 regions in transition.  In Hbase shell, testing:testtable1 does not exist anymore. I tried to recreate it but Hbase shell just hanged, most likely due to those regions in transition. I ran the hbck -details and got the following report         sudo -u hbase hbase hbck -details Number of regions in transition: 12 testing:testtable1,,1566972509039.ab8696ff1a7dc14cd90d138844a71bf6. state=OPENING, ts=Wed Aug 28 08:49:37 PDT 2019 (PT36M51.518S ago), server=null testing:testtable1,1242054,1566972509039.8b11666a10241ec267692e9f09a85c3e. state=OPENING, ts=Wed Aug 28 08:49:37 PDT 2019 (PT36M51.524S ago), server=null           I tried the hbck repair option but got the warning that the repair option has been deprecated.   Options available for medadata repairs are: Metadata Repair options supported as of version 2.0: (expert features, use with caution!) -fixVersionFile Try to fix missing hbase.version file in hdfs. -fixReferenceFiles Try to offline lingering reference store files -fixHFileLinks Try to offline lingering HFileLinks -noHdfsChecking Don't load/check region info from HDFS. Assumes hbase:meta region info is good. Won't check/fix any HDFS issue, e.g. hole, orphan, or overlap -ignorePreCheckPermission ignore filesystem permission pre-check   So which option should I use?   ... View more