I am reading "Enable the Atlas Book & Bridge for Apache HBase" https://atlas.apache.org/Hook-HBase.html, and get stuck on the step below Link Atlas hook jars in HBase classpath - 'ln -s <atlas package>/hook/hbase/* <hbase-home>/lib/' Where is the HBase classpath in HDP? I checked the file hbase_site.xml but still got lost. From the file,   # Extra Java CLASSPATH elements. Optional. export HBASE_CLASSPATH=${HBASE_CLASSPATH}   Is the classpath env variable specified somewhere else? ... View more

I create two tables with the same name, one in Hive and the other in HBase. In Atlas, I found the hive table using the search: Search by Type: hive_table Search by Text: the table name However, I could not find the hbase table using the same search criteria Search by Type: hbase_table Search by Text: the table name If I just search by type hbase_table, the hbase table is in the result. What do I miss? ... View more

I am using HDP 3.1. I deleted a directory of an Hbase table by mistake. The deleted hdfs directory is /apps/hbase/data/data/testing/testtable1. The table has 12 regions. After the deletion, I rebooted hbase and the Ambari Hbase UI shows 12 regions in transition.  In Hbase shell, testing:testtable1 does not exist anymore. I tried to recreate it but Hbase shell just hanged, most likely due to those regions in transition. I ran the hbck -details and got the following report         sudo -u hbase hbase hbck -details Number of regions in transition: 12 testing:testtable1,,1566972509039.ab8696ff1a7dc14cd90d138844a71bf6. state=OPENING, ts=Wed Aug 28 08:49:37 PDT 2019 (PT36M51.518S ago), server=null testing:testtable1,1242054,1566972509039.8b11666a10241ec267692e9f09a85c3e. state=OPENING, ts=Wed Aug 28 08:49:37 PDT 2019 (PT36M51.524S ago), server=null           I tried the hbck repair option but got the warning that the repair option has been deprecated.   Options available for medadata repairs are: Metadata Repair options supported as of version 2.0: (expert features, use with caution!) -fixVersionFile Try to fix missing hbase.version file in hdfs. -fixReferenceFiles Try to offline lingering reference store files -fixHFileLinks Try to offline lingering HFileLinks -noHdfsChecking Don't load/check region info from HDFS. Assumes hbase:meta region info is good. Won't check/fix any HDFS issue, e.g. hole, orphan, or overlap -ignorePreCheckPermission ignore filesystem permission pre-check   So which option should I use?   ... View more

I am building a data ingestion process of loading csv files from local storage to hdfs and then use hive to clean data and then load clean data files into HBase. In my dev environment Ranger has not been added to Ambari yet. I did not create any new users either and only used users like root, hive, and hdfs. I ran into "Permission Denied" issue from time and time. I feel I have to use Ranger, because Ranger is a framework of comprehensive data security across the Hadoop platform. So if I want to make one user, like "root" to be the super user in my dev environment without ever running into "Permission Denied" error again, Ranger is the place to make all the configuration change. Am I on the right track? ... View more

When loading a csv file into an HBase table, some bad lines are dropped. How I can identify which lines are dropped? A similar case is here. https://community.hortonworks.com/questions/73985/no-data-shown-in-hbase-after-importtsv.html Running ImportTsv generates big log files, as mentioned here. https://community.hortonworks.com/articles/4942/import-csv-data-into-hbase-using-importtsv.html. Maybe the log file can help me but I do not know where those log files are. I expect they are in hadoop storage, rather than linux local storage. So I look into the my hadoop path /app-logs/hdfs/logs/application_1557438882545_0077. The folder name "application_1557438882545_0077" is the mapreduce job id. There is one file inside this folder and the file name is name1.abc.local_45454_1562965335601. This file is not in a human-readable format. ... View more

I follow the instruction on the link below to load csv data into a Phoenix data using MapReduce. https://phoenix.apache.org/bulk_dataload.html The loading was successful with a csv files of 4k rows. When I tried a file with millions of records, I kept getting errors of "illegal data" or "The data exceeds the max capacity for the data type" or " CSV record does not have enough values (has 13, but needs 14." At the end of the loading, I got a summary like Job Counters                 Failed map tasks=4                 Killed reduce tasks=1                 Launched map tasks=8                 Launched reduce tasks=1 Not a single record is loaded. I know most of rows in the file are with the correct layout. Is there a way to ignore the layout error, or export rows of invalid layout? ... View more

I created an external table whose definition contains a varchar column. Create table test1_dat ( SUBNO bigint , AFFILCODE varchar(5) , EFFDT string )) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'            STORED AS TEXTFILE            LOCATION '/data/testdb/test1_dat'; The following Hql statement returns no records. select * from jaffilm0_dat where rtrim(AFFILCODE)='LTC' limit 10 ; The following Hql statements return correct records. select * from jaffilm0_dat where rtrim(AFFILCODE)='LTC' limit 10 ; select * from jaffilm0_dat where AFFILCODE='LTC  ' limit 10 ;  -- adding two trailing spaces Basically, AFFILCODE is treated as char(5) by Hive. Is this supposed to happen? Is there any issue with my table definition script? ... View more