About Daggers

Daggers · ‎06-02-2020

If I run the 'ALTER TABLE blah ARCHIVE partition blah' how can I then later determine what partitions have been archived ? I am asking this question because I want to code a script that archives old partitions that have not yet been archived. It would be great if the answer was something like 'look in the hive metastore using query xxx' rather than 'inspect each partition folder and determine if it has any .har files in it' 🙂 Thanks

Daggers · ‎04-01-2020

I am getting the same problem and I am on 5.16 My research shows that the redirect should happen automatically but that does not seem to be happening

Daggers · ‎12-18-2019

I wonder if there is a way to ensure that all the files have finished being written to /tmp/log (The location at my site of yarn.nodemanager.remote-app-log-dir ) before I copy them ?

Daggers · ‎12-18-2019

Hey great suggestion ! I think this might work.

Daggers · ‎12-15-2019

Hi, If I try and get the logs for an application like this: yarn logs -applicationId application_1575531060741_10424 The command fails because I am not running it as the application owner. I need to run it like this: yarn logs -applicationId application_1575531060741_10424 -appOwner hive The problem is I want to write out all the yarn logs to the os so I can ingest them into splunk. If I try and figure out the appOwner for each application then this is awkward and time consuming even in a script. Is there a better way to dump all the yarn logs to the os ? Thanks

Daggers · ‎10-22-2019

We have a user who has left the organisation so we want to migrate all their workflows to another user. Problem is we can logon to HUE with their account in order to export the WF's because we don't have their id. Any solution to this ?

Daggers · ‎08-28-2019

How do I change the description/comment of an existing hive database ?

Daggers · ‎07-17-2019

Hi, We are using CDH 5.13.3 When I am in the scheduler tab of the Job Browser in HUE I cant find the coordinator that runs my workflow. Why is this ? In order to find the coordinator I have to click on the WF then from there click on the coordinator. Thanks Steve.

Daggers · ‎07-14-2019

Do table statistics get replicated as part of hive BDR replication ?

Daggers · ‎07-14-2019

Thanks for that

Daggers · ‎07-10-2019

Did you find the logs in the end ? I cant find them myself

Daggers · ‎05-29-2019

Hi, I want to find who created a bunch of tables so I can contact them. Is navigator the best way to do this ? If you search the audits in Navigator It seems that in HIVE the operation is CREATETABLE and in Impala it is CREATE_TABLE which makes it awkward. Also if we are only retaining 3 months of data in Navigator then is there an option to find the creator of a table older than 3 months ? Thanks

Daggers · ‎05-29-2019

Hi, When I query the metastore table called TBLS and look at the OWNER column I see sometimes tables are owned by HIVE and sometimes by a normal user. What are the situations that determine who is the owner ? Thanks

Daggers · ‎02-20-2019

Hi, We have HA enabled for the metastore and have 2 instances running. Is it possible to configre a load balancer in front of these ? and if so is this normal practice ? Thanks

Daggers · ‎02-17-2019

Hi, I have 6 datanodes and if I run this chart query: select impala_admission_controller_local_num_admitted_running WHERE poolName="root.default" I get six lines on my graph. But each one has a human unfriendly name like: impala-IMPALAD-00a06701e5a2bac3b893e59fde762af5:root.default How to I get that display that name as the real hostname ? Thanks

Daggers · ‎02-13-2019

Excellent ! these specific examples are what is needed.

Daggers · ‎01-17-2019

Yes and that is not there. Also this message in the log of impala indicates it is set(I think) impalad.pxlbig04.nz.thenational.com.impala.log.INFO.20190117-104024.29754:I0117 10:43:37.781446 34548 admission-controller.cc:510] Schedule for id=d34a40c71691c6d7:a345018b00000000 in pool_name=root.default cluster_mem_needed=90.00 GB PoolConfig: max_requests=25 max_queued=15 max_mem=1024.00 GB So the question is how come I get this error in HUE(impala) sometimes: Rejected query from pool root.default: queue full, limit=15, num_queued=16 [‎14/‎01/‎2019 1:23 p.m.] Oh !!!! I think I have answered my own question. There must have been 25 already running so the limit I have hit must be max_queued... yes ?

Daggers · ‎01-17-2019

Hi, I can find this kind of message in the Impala daemon logs: impalad.pxlbig04.nz.thenational.com.impala.log.INFO.20190117-104024.29754:I0117 10:42:17.900411 9206 admission-controller.cc:515] Stats: agg_num_running=7, agg_num_queued=0, agg_mem_reserved=525.00 GB, local_host(local_mem_admitted=0, num_admitted_running=0, num_queued=0, backend_mem_reserved=75.00 GB) Which shows agg_num_running=7 Why is this same kind of metric not available in the CM graphs ? I want to know how close to the max running queries of 25 that I set in the default queue on Impala Admission Control.

Daggers · ‎01-17-2019

I don't think so. There is nothing to tell me otherwise. When I go into admission control it is set to 25. Is there a way to double check ?

Daggers · ‎01-17-2019

I know this an old post but I have had a similar problem where I changed max running queries to 25 under 'Impala admission control' but then a couple of days later got an error in HUE(impala) saying max queries of 15 was exceeded.... what is up with that ? it seems the setting of 25 has not taken effect.

Daggers · ‎09-06-2018

Thanks. Well yes I think it is what I need however.... it seems some of my tables have that date changing multiple times during a day. I looked at navigator to try and track down who may have been issuing 'COMPUTE STATS' commands but found nothing. To make the whole picture even stranger is the fact that the table is replicated at the database level using BDR. But the master metadata is different from the target metadata. The target table is the one I am looking at, this is the one getting update multiple times a day. Any suggestions on how to find what is making this LAST_ANALYZED timestamp to increment all the time ?

Daggers · ‎08-29-2018

I want to generate 'compute statistics <table_name>' commands for all non-partitoned tables that have not been analyzed for at least 1 month. If that information was available I would imagine it would be in the hive metastore database which the rest of the table metadata... however I cant find anything. Thanks Steve.

Daggers · ‎08-26-2018

Hi, I have exactly the issue described in this post and I am running CDH 5.13.3 I have tried putting this in the hql script as mentioned above: set hive.variable.substitute=true; and that does not help either. Can someone advise what to try ? Thanks

Daggers · ‎05-30-2017

Hi, I would like to be able to find the same information. I really need to be able to query it from a database. I thought it might be available in the Sentry or the Hive back end databases but I just can find anything. Is anyone able to offer any solutions

Daggers · ‎05-04-2017

Sadly I dont think the Oracle jdbc driver is in place for me to try that which is why I used HUE in the first place. I'll see if someone can install it for me.

Daggers · ‎05-03-2017

Hi, My name is Steve I am an Oracle DBA who is getting to grips(well trying) with Hadoop. I work for a Bank and we have just installed 2x11 node clusters and I am looking to build a team to support it. I am also trying to understand data loading and ETL processes using the hadoop tools like sqoop

Daggers · ‎05-03-2017

I have the same issue but I am running my sqoop command as part of an oozie workflow in HUE. My job finishes successfully but the table does not get created under the database that I specify or under the default database. I can see the files that get imported but they are under my users default directory. Any suggestions ?

Online	Offline
Last Visited	‎07-26-2020 06:50 PM

Date Registered	‎05-03-2017 04:26 PM
Last Visited	‎07-26-2020 06:50 PM
Total Messages Posted	26

How to tell what HIVE table partitions have been a...

Re: Resource manager API redirection doesn't show ...

Re: Using yarn logs command

Re: Using yarn logs command

Using yarn logs command

Migrate all oozie workflows from one user to anoth...

edit the description of a database

Cant find my coordinator in HUE eaily in the Job b...

Do table statistics get replicated as part of hive...

Re: BDR - Hive Replication Log Directory

Re: BDR - Hive Replication Log Directory

Best way to find who created a table

What determines who the "owner" of a table is ?

Load balancer for metastore

Giving an alias to the to the data points in the C...

Re: Number of impala queries running - graph

Re: Impala queries defaults to a max 20 queries an...

Number of impala queries running - graph

Re: Impala queries defaults to a max 20 queries an...

Re: Impala queries defaults to a max 20 queries an...

Re: What was the date that a HIVE/IMPALA table get...

What was the date that a HIVE/IMPALA table gets an...

Re: Compiling statement: FAILED: ParseException : ...

Re: Sentry query to know the group or role for a p...

Re: after sqoop hive-import, tables not listed

Re: INTRODUCE YOURSELF: Please Reply here to tell ...

Re: after sqoop hive-import, tables not listed