About thewayofthinkin

thewayofthinkin · ‎10-22-2019

Hello, I have some question about setting up Impala and Datanode config. Context - using CDH 5.16.2 - Datanode - CentOS 6.9 Question1: Does `hdfs-site.xml` have to be copied to /etc/impala/conf/ and modified? OR, can `hdfs-site.xml` be created with these config is enough like this? <property> <name>dfs.client.read.shortcircuit</name> <value>true</value> </property> <property> <name>dfs.domain.socket.path</name> <value>/var/lib/hadoop-hdfs/socket._PORT</value> </property> <property> <name>dfs.client.file-block-storage-locations.timeout.millis</name> <value>10000</value> </property> <property> <name>dfs.datanode.hdfs-blocks-metadata.enabled</name> <value>true</value> </property> This is /etc/hadoop/conf/hdfs-site.xml <configuration> <property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property> <property> <name>dfs.domain.socket.path</name> <value>/var/lib/hadoop-hdfs/socket._PORT</value> </property> <property> <name>dfs.client.read.shortcircuit</name> <value>true</value> </property> <property> <name>dfs.client.read.shortcircuit.skip.checksum</name> <value>false</value> </property> <property> <name>dfs.client.file-block-storage-locations.timeout</name> <value>10000</value> </property> <property> <name>dfs.datanode.hdfs-blocks-metadata.enabled</name> <value>true</value> </property> <property> <name>dfs.block.size</name> <value>134217728</value> </property> <property> <name>dfs.datanode.du.reserved</name> <value>1073741824</value> </property> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.data.dir</name> <value>/data0/hadoop/hdfs/data,/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data,/data3/hadoop/hdfs/data,/data4/hadoop/hdfs/data,/data5/hadoop/hdfs/data,/data6/hadoop/hdfs/data,/data7/hadoop/hdfs/data,/data8/hadoop/hdfs/data,/data9/hadoop/hdfs/data</value> </property> <property> <name>dfs.name.dir</name> <value>/data0/hadoop/hdfs/name,/data1/hadoop/hdfs/name,/data2/hadoop/hdfs/name,/data3/hadoop/hdfs/name,/data4/hadoop/hdfs/name,/data5/hadoop/hdfs/name,/data6/hadoop/hdfs/name,/data7/hadoop/hdfs/name,/data8/hadoop/hdfs/name,/data9/hadoop/hdfs/name</value> </property> <property> <name>fs.checkpoint.dir</name> <value>/data0/hadoop/hdfs/secondary,/data1/hadoop/hdfs/secondary,/data2/hadoop/hdfs/secondary,/data3/hadoop/hdfs/secondary,/data4/hadoop/hdfs/secondary,/data5/hadoop/hdfs/secondary,/data6/hadoop/hdfs/secondary,/data7/hadoop/hdfs/secondary,/data8/hadoop/hdfs/secondary,/data9/hadoop/hdfs/secondary</value> </property> <property> <name>dfs.permissions.superusergroup</name> <value>hadoop</value> </property> <property> <name>dfs.client.use.datanode.hostname</name> <value>true</value> </property> </configuration> Question2: Which value of properties in /etc/impala/conf/hdfs-site.xml will be read by Impala? The reason of asking this question is that what if there is different value for `dfs.data.dir` property like /etc/impala/conf/hdfs-site.xml has ... <property> <name>dfs.data.dir</name> <value>/data0/hadoop/hdfs/data,/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data,/data3/hadoop/hdfs/data,/data4/hadoop/hdfs/data</value> </property> ... but, /etc/hadoop/conf/hdfs-site.xml has ... <property> <name>dfs.data.dir</name> <value>/data0/hadoop/hdfs/data,/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data,/data3/hadoop/hdfs/data,/data4/hadoop/hdfs/data,/data5/hadoop/hdfs/data,/data6/hadoop/hdfs/data,/data7/hadoop/hdfs/data,/data8/hadoop/hdfs/data,/data9/hadoop/hdfs/data</value> </property> ... Can this make any performance issue? Thank you for your comment/reply in advance 🙂

thewayofthinkin · ‎09-07-2017

Hello, Have you encountered this problem? $ beeline -u jdbc:hive2:// Could not find valid SPARK_HOME while searching ['/home', '/usr/bin'] /usr/bin/beeline: line 32: /bin/spark-class: No such file or directory /usr/bin/beeline: line 32: exec: /bin/spark-class: cannot execute: No such file or directory

thewayofthinkin · ‎04-28-2017

Alon, I'm sorry that this is not answer to your question. actually, I have question for you. In order to use the python package you're using, do I have to run cloudera manager on my cluster? Thank you Gatsby

thewayofthinkin · ‎04-28-2017

Joe, ah. i see so, at any given time, a file handle is used by only one thread. it means a file handle is not used by multiple threads at the same time. I'm very wonrdering how you know about this very well 🙂 Thank you very much Gatsby

thewayofthinkin · ‎04-28-2017

Joe, By the way, do you think one cached file handle is used by multiple threads? Gatsby

thewayofthinkin · ‎04-28-2017

Joe, Thank you for your comment. Your comment really help me confirm and understand how file-handle-cache works. Like you said, over time, the hit-count is going up and miss-cout is becoming stable 🙂

thewayofthinkin · ‎04-24-2017

Yeap, you're right

thewayofthinkin · ‎04-24-2017

Hello, I have a qusetion about these values in impala-server metric. I set `max_cached_file_handles` to 10,000 max_cached_file_handles (uint64) Maximum number of HDFS file handles that will be cached. Disabled if set to 0. 0 10000 However, there are still missed ` impala-server.io.mgr.cached-file-handles-miss-count`. impala-server.io.mgr.cached-file-handles-hit-count 668 Number of cache hits for cached HDFS file handles impala-server.io.mgr.num-cached-file-handles 253 Number of currently cached HDFS file handles in the IO manager. impala-server.io.mgr.cached-file-handles-miss-count 1467 Number of cache misses for cached HDFS file handles Can you tell me how to reduce the miss-count? Thank you Gatsby

thewayofthinkin · ‎03-21-2017

Alex, Thank you again. Subquery approach has been recommended to our team as a long term solution. However, for short-tem solution to avoid regression impact, using view with limited partitions has been selected. If I remember correctly, in MySQL `table A` data can be limited by `ON Clause` before joining so that cadidates for join can be reduced. Thank you for your valuable comment. Gatsby

thewayofthinkin · ‎03-20-2017

Alex, First of all, thank you very much for your explanation. You're right. the second query selects partition in table A. And, I'm fully aware of the difference between the first one( using on clause ) and second on ( where clause ) like the way you explained. The reason different variances were tried to find out ways to limit table A data before joinging two tables. ( yes, second query doesn't work this way ) In MySQL, table A could be limited by `ON clause`, but with Impala, I don't know how to do it. Do you think using subquery is the best way? Thank you Gatsby

thewayofthinkin · ‎03-20-2017

Hi Henry, I have a question for you and it is about `partition pruning` ( about pruning ) Let's say there are two tables A and B. And, each table is partitioned by yearweek. And, here is the query I'd like to run. ( Yes. I need to use left join to get result what I want ) SELECT * FROM A LEFT OUTER JOIN B ON A.account_id = B.account_id AND A.yearweek = 201710 and B.yearweek = 201710 Even this doesn't select specific partition in `table A` SELECT * FROM A LEFT OUTER JOIN B ON A.account_id = B.account_id AND B.yearweek = 201710 WHERE A.yearweek = 201710 Like you said, `A.yearweek` = 201710 on `On clause` couldn't select partition yearweek=201710. This might be filter is applied from `Left to Right`. In order to select specific partition for `table A`. I used `dynamic partition` and updated query like this. SELECT * FROM (SELECT * FROM A WHERE yearweek = 201710) a LEFT OUTER JOIN B b ON a.account_id = b.account_id AND b.yearweek = 201710 Do you think this is best I can do? Or is there way to limit data for `table A` by using `On clause`? And, is there any refrerence you would recommend for me to upderstand how JOIN works in Impala and Hive? Thank you very much in advance. Gatsby

thewayofthinkin · ‎02-02-2017

Thank you. I will check with team to set the timeout. Gatsby

thewayofthinkin · ‎02-02-2017

From log history, I found out someone keep running 'invalidate metadata' without table name. 😞 Thank you for your comment Gatsby

thewayofthinkin · ‎02-02-2017

yeap. you're right. I will take a look log. Thank you Gatsby

thewayofthinkin · ‎02-01-2017

I found out this open ticket - https://issues.cloudera.org/browse/IMPALA-1575 Probably the query you saw is from Hue. Gatsby

thewayofthinkin · ‎02-01-2017

@gaurang Today, I had some issue with slow quries. And, the issue was related to metadata Catalog Daemon caches. How often do you make quries to that TABLE/VIEW ( I don't think your issue is related to VIEW )? In my case, metadata for TABLE was reloaded very often because Catalog Daemon flushes out metadata. Take a look your catalog daemon and check if TABLE metadata is cached. Gatsby

thewayofthinkin · ‎02-01-2017

Hello, I have a question about metadata loaded in Impala catalog daemon. If I understand correctly, catalog daemon read TABLE metadata from Hive metadata store and caches in memory. My question is what trigger flushing out this cached TABLE metadata. The reason I ask this question is that I noticed that TABLE metadata is flushed out from catalog daemon after some time. And, since TABLE metadata doesn't exist, catalog daemon tries to load metadata again from metastore. Is there some kind of configuration I can set to control TABLE metadata lifecycle ( or memory )? Thank you Gatsby

thewayofthinkin · ‎02-01-2017

@Lars Volker I have a question for you. How long the metadata loaded from Hive metastore by Impala Catalog Daemon stay in memory? I'm using Impala 2.7 ( KUDU ). It seems the metadata is flushed more often than before. Is there any configuration for life cycle for metadata in catalog daemon has? @gaurang I'm asking this question here because I guess @Lars Volker answer can help resolve your issue. Thank you Gatsby

thewayofthinkin · ‎02-01-2017

@dsss yeap. I agree with you. we can't just kill those quries manually. I think most of times quries in exception status and in filght quries from Hue. @Lars Volker Is there way to log slow quries like MySQL slow quries? Is there way to set timeout for long running quries so that those quries can be killed? Thank you Gatsby

thewayofthinkin · ‎01-31-2017

@Tim Armstrong Thank you very much for your explanation. 🙂 Gatsby

thewayofthinkin · ‎01-31-2017

FYI, `COMPUTE STATS` can run with first level partition. https://issues.cloudera.org/browse/IMPALA-1570

thewayofthinkin · ‎01-31-2017

@Lars Volker I thought that way too. Thank you for confirming it 🙂 Gatsby

thewayofthinkin · ‎01-31-2017

@venkatsambath already answered to your question. One thing I'd like to add is that please run simple `SELECT` query for the table you want to reload metadata after running `INAVLIDATE META`. By doing that, you can trigger loading metadata in Impala. Gatsby

thewayofthinkin · ‎01-31-2017

You might need to click the `cancel` link next to query. Although a query is under exception status, it seems still running and using resources Gatsby

thewayofthinkin · ‎01-31-2017

@Lars Volker I have a question for you. Does VIEW still need to load metadata separately even if metadata of TABLE for VIEW is already loaded? @gaurang which CDH/Impala are you using? Thank you Gatsby

thewayofthinkin · ‎01-31-2017

Thank you. Then, why does this say that Impala2.8/CDH5.10? http://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_new_features.html#new_features_280

thewayofthinkin · ‎01-31-2017

Hi, I noticed that CDH5.10 is realesed and has Impala 2.8. However, it seems there are not Impala 2.8 RPM packages for RedHat6. Can you tell me where I can them? Thank you Gatsby

thewayofthinkin · ‎01-26-2017

I found out what caused this issue in this case. Nodes have different Impala version, such as 2.8, 2.4 etc. The interesting thing is that `compute stats` operation didn't work if run in Impala 2.8, but the operation works in Impala 2.4. Anyway, I hope this can help you solve your problem. Thank you Gatsby

thewayofthinkin · ‎01-25-2017

Is this related to kerberos authentication??

thewayofthinkin · ‎01-25-2017

Thank you

Online	Offline
Last Visited	‎10-22-2019 04:34 PM

Date Registered	‎12-30-2015 01:25 PM
Last Visited	‎10-22-2019 04:34 PM
Total Messages Posted	68
Total Kudos Received	2

Re: ExecPlanRequest rpc exception error while COMP...

which properties in hdfs-site.xml are ready by Imp...

beeline doesn't start due to missing /bin/spark-cl...

Re: After CDH upgrade from 5.9 to 5.11 Impala quer...

Re: do you know impala-server.io.mgr.cached-file-h...

Re: do you know impala-server.io.mgr.cached-file-h...

Re: do you know impala-server.io.mgr.cached-file-h...

Re: Need help with Impala 2.8 on CDH 5.10 upgrade

do you know impala-server.io.mgr.cached-file-handl...

Re: Impala runtime filter not working as expected

Re: Impala runtime filter not working as expected

Re: Impala runtime filter not working as expected

Re: queries in flight issue

Re: What / When flush out table metadata in catalo...

Re: When querying a VIEW, query planning takes a l...

Re: queries in flight issue

Re: When querying a VIEW, query planning takes a l...

What / When flush out table metadata in catalog da...

Re: When querying a VIEW, query planning takes a l...

Re: queries in flight issue

Re: CDH5.10 redhat repo doesn't have Impala 2.8 RP...

Re: WARNINGS: Too many partitions selected, doing ...

Re: When querying a VIEW, query planning takes a l...

Re: Impala/Hive metadata problem

Re: queries in flight issue

Re: When querying a VIEW, query planning takes a l...

Re: CDH5.10 redhat repo doesn't have Impala 2.8 RP...

CDH5.10 redhat repo doesn't have Impala 2.8 RPM pa...

Re: ExecPlanRequest rpc exception error while COMP...

Re: ExecPlanRequest rpc exception error while COMP...

Re: how to remove yarn application queue?