question Re: CDH 5.5.0 - Getting Started Tables not visible though install job completed in Support Questions

CDH 5.5.0 - Getting Started Tables not visible though install job completed

sivaj — Fri, 16 Sep 2022 09:51:11 GMT

I had an issue with CDH 5.4 hanging. I prefered to remove it and re-install the latest version.

I downloaded and installed CDH 5.5 and tried to use the Getting Started.

After several tries, I was able to (using Community forums) complete my sudo install done correctly.

After the /user/hive/warehouse/categories finally were created correctly,. I went to Hive and Impala editor and tried to refresh table after entering Invalidate metadata command.

For some reason, I am unable to see the tables, through I see the parquet files with bytes on each of the tables (customers, departments etc)

Please help let me know if I am missing anything.

Thanks

Re: CDH 5.5.0 - Getting Started Tables not visible though install job completed

sivaj — Fri, 04 Dec 2015 18:27:21 GMT

to experts

sqoop import-all-tables \
> -m 3 \
> --connect jdbc:mysql://208.113.123.213:3306/retail_db \
> --username=retail_dba \
> --password=cloudera \
> --compression-codec=snappy \
> --as-parquetfile \
> --warehouse-dir=/user/hive/warehouse \
> --hive-import
was the original command I used (after cleaning files)

should I redo (after deleting connected folders) using

sqoop import-all-tables \
> -m 3 \
> --connect jdbc:mysql://208.113.123.213:3306/retail_db \
> --username=retail_dba \
> --password=cloudera \
> --compression-codec=snappy \
> --as-avrofile \
> --warehouse-dir=/user/hive/warehouse \
> --hive-import

Re: CDH 5.5.0 - Getting Started Tables not visible though install job completed

Sean — Fri, 04 Dec 2015 18:43:09 GMT

Can you post the output of your Sqoop job? I'm wondering if there were errors when it was doing the --hive-import part. There's 2 stages: writing the files in the new data format to HDFS, and then defining the schema for the tables in Hive's Metastore. It sounds like that 2nd stage failed...

Re: CDH 5.5.0 - Getting Started Tables not visible though install job completed

sivaj — Fri, 04 Dec 2015 18:54:31 GMT

Sean

Should I look in the log files under var/log/hive/ ?

Let me know.

Here is the file location though for the successful process (for
/ user/ hive/ warehouse/ products)

         Name    Size    User    Group    Permissions    Date
               cloudera    supergroup    drwxr-xr-x    December 03, 2015 09:47 PM
       .        cloudera    supergroup    drwxr-xr-x    December 03, 2015 09:47 PM
       .metadata        cloudera    supergroup    drwxr-xr-x    December 03, 2015 09:47 PM
       .signals        cloudera    supergroup    drwxr-xr-x    December 03, 2015 09:47 PM
       6f7ab0da-3cbf-40ee-a74a-d73683c68c91.parquet    43.8 KB    cloudera    supergroup    -rw-r--r--    December 03, 2015 09:47 PM

hive-metastore.log file

2015-12-04 10:12:55,165 WARN [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@33267d64]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 12002ms
No GCs detected
2015-12-04 10:13:41,510 WARN [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@33267d64]: common.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Detected pause in JVM or host machine (eg GC): pause of approximately 13720ms
No GCs detected

Re: CDH 5.5.0 - Getting Started Tables not visible though install job completed

Sean — Fri, 04 Dec 2015 20:05:52 GMT

I was referring to the output of your Sqoop command - they are printed to
the terminal, not written to a log file. However the log snippets you did
post do indicate a potential problem: if Hive was pausing too much for
garbage collection, then Sqoop might have given up / timed out when doing
the import. You may not have enough memory for the services to run well.

Re: CDH 5.5.0 - Getting Started Tables not visible though install job completed

sivaj — Fri, 04 Dec 2015 22:54:54 GMT

Sean

What is the recommended memory.

I have used 3, 4, 6 on CDH 5.4 and even up to 8MB

Can i try the sudo code using avrotable instead of parquet option.

Let me know.

Re: CDH 5.5.0 - Getting Started Tables not visible though install job completed

Sean — Fri, 04 Dec 2015 23:04:52 GMT

Well there are a lot of variables so a simple "minimum requirement" is a
tough number to give. The tutorial was originally written for a 4-node
cluster with 16 GB of RAM per node, and that's a little bit small for the
master node. The QuickStart VM has a version of the tutorial with a smaller
dataset. You can get away with 4 GB (but this includes the graphical
desktop, so let's say 3 GB for a server) if you don't use Cloudera Manager
and manage everything yourself (note that this is pretty complex). If you
use the "Cloudera Express" option for Cloudera Manager, 8 GB is the
absolute minimum, and if you're going to try out "Cloudera Enterprise" you
need at least 10 GB. But the number of nodes, exactly which services you're
running, exactly what else is going on on the machines, etc. all affects
this.

Re: CDH 5.5.0 - Getting Started Tables not visible though install job completed

Sean — Fri, 04 Dec 2015 23:41:24 GMT

To answer your other question though, I wouldn't expect a different data format to make a difference here. There's enough competition for memory on the system that Hive is constantly doing garbage collection, and that shouldn't have anything to do with what format Sqoop is using for the data.

Re: CDH 5.5.0 - Getting Started Tables not visible though install job completed

sivaj — Fri, 04 Dec 2015 23:52:11 GMT

Sean

I understand.

I am trying to emphasize here that the job did finally run without delay or failure. It did put 6 or so tables in the directory.

But the job didnt show those tables on Hive or Impala editor.

I tried to add the table using the file option but the parquet format didnt seem to convert well for the table to show up.

Bottom line, the sqoop job put the file where it should be put but I just dont see it showing up on the editor (database does not reflects those tables have been created,

That is why I am asking are there other ways to get the table to show.

When I need CM or CM Enterprise I would choose another laptop. Right now I have a regular laptop for my learning and demo purpose.

Re: CDH 5.5.0 - Getting Started Tables not visible though install job completed

Sean — Sat, 05 Dec 2015 00:07:53 GMT

Remember that schema and data are two separate things in Hadoop. The files
in the directory, they are simply data files. For tables to show up in Hive
or Impala, you have to import or define the schema for those tables in Hive
Metastore. I believe the reason you're not seeing the tables is because the
logs you posted show that Hive is constantly struggling with garbage
collection. My guess is that Sqoop tried to import the schema into Hive but
timed out - but I don't know for sure unless you can post the text
outputted of the Sqoop command.

To be clear - are you running a QuickStart VM? I'm a little unclear on
exactly what your environment is.

Re: CDH 5.5.0 - Getting Started Tables not visible though install job completed

sivaj — Sat, 05 Dec 2015 00:13:18 GMT

Sean - yes, it is a Cloudera Quickstart VM.

I deleted CDH 5.4 I was using for 4 - 6 weeks since I had some issue and Windows upgrade.

Loading CDH 5.5 is what caused this problem.

Re: CDH 5.5.0 - Getting Started Tables not visible though install job completed

sivaj — Sat, 05 Dec 2015 01:11:01 GMT

Sean

It finally worked.

After your inputs and reading several of your replies on other posts, I did the following:

1. On my Cloudera VM i reset the memory to 8GB. I had others I removed (2 other images).

2. I restarted the VM and deleted the files under hive with this command - sudo -u hdfs hadoop fs -rm -r /user/hive/warehouse/\*

3. I redid the sudo comand per tutorial 1 using

sqoop import-all-tables \
-m 3 \
--connect jdbc:mysql://208.113.123.213:3306/retail_db \
--username=retail_dba \
--password=cloudera \
--compression-codec=snappy \
--as-avrofile \
--warehouse-dir=/user/hive/warehouse \
--hive-import

Note - I used the avro format

4. After job finished, I went to Hive Editor and refreshed Database. After refresh, it showed the 6 tables I was expecting

5. I then went to Impala editor and did the same refresh (after Invalidate metadata; show tables:)

6. Next, I went to the tables and looked at table statistics. That ran a job and I found the table entries started showing up (statistics updated with row, byte size etc).

Bottom line, the avro format worked.

I have to work on the table right now so I did not use the parquet file format. I shall retry this once my immediate work is done.

Without your assistance on ths particular posts and without reading your other posts this would not have been solved

I thank you for that and appreciate your help.