Member since
02-22-2016
27
Posts
0
Kudos Received
0
Solutions
03-24-2020
10:27 PM
We do have below entries and we have confirmed there are no firewall rules. One more thing i missed to mention we started seeing this issues when we started upgrading OS in the cluster nodes. From OEL 6.x to OEL 7.x. But this seems to be happening on both type of host also looking at the logs. 127.0.0.1 localhost.localdomain localhost # special IPv6 addresses ::1 localhost6.localdomain6 localhost6 fe00::0 ipv6-localnet ff00::0 ipv6-mcastprefix ff02::1 ipv6-allnodes ff02::2 ipv6-allrouters ff02::3 ipv6-allhosts
... View more
03-24-2020
03:57 AM
@Shelton Thanks for responding. Why would the same error comes for communication within the host also for different ports? any clues. We are using static IP (private IP for cluster communication) and it is specified /etc/hosts across all hosts.
... View more
03-24-2020
03:22 AM
We are seeing lot of no route to host in datanode logs and impala queries are also failing due to this. We are seeing this within the nodes and between nodes also. Issue is happening with multiple nodes. Host inspector are running with no issues.
We did lot of checks with OS and network team we couldn't find any. Any help on this.
1004:DataXceiver error processing WRITE_BLOCK operation src: /192.168.225.165:55010 dst: /192.168.225.165:1004 java.net.NoRouteToHostException: No route to host
1004:DataXceiver error processing WRITE_BLOCK operation src: /192.168.225.68:35322 dst: /192.168.225.68:1004 java.net.NoRouteToHostException: No route to host
1004:DataXceiver error processing WRITE_BLOCK operation src: /192.168.225.171:40718 dst: /192.168.225.165:1004 java.net.NoRouteToHostException: No route to host
... View more
Labels:
- Labels:
-
Apache Impala
03-16-2017
02:30 AM
Thx @csguna for the detailed explanation. Much appreaciated . So i think there is not much difference in terms size , for snappy compressed and non compressed parquet table.
... View more
03-09-2017
09:12 PM
@csguna@saranvisa Thx for the detailed response. I have 2 follow up questions (sorry i am just learning) 1) Since snappy is not too good at compression (disk), what would be the difference on disk space for a 1 TB table when stored as parquet only and parquet with snappy compression. 2) Is it possible to compress a non-compressed parquet table later with snappy?
... View more
03-09-2017
09:00 PM
@Wilfred Thx for the reply. First yes we have sentry enabled. I already explored Hive Sentry Subject Name attribute in CM (we are on Cloudera 5.8.3) but for most of the jobs there is no value for that attirbute. Not sure from where it gets the user name and which condition it does not. Difficulty is there with mapreduce also, but with Hive on spark it increased. As hive on MR atleast show the query details (select * ....) where as in HoS, owner is hive and application is hive on spark for all the jobs (difficult to troubleshoot when you 100/1000 of these kind of jobs).
... View more
03-08-2017
07:59 AM
Hi, 1) If we create a table (both hive and impala)and just specify stored as parquet . Will that be snappy compressed by default in CDH? 2) If not how do i identify a parquet table with snappy compression and parquet table without snappy compression?. Also how to specify snappy compression for table level whiel creating and also at global level, even if nobody specified at table level (all table stored as parquet should be snappy compressed). Please help
... View more
Labels:
- Labels:
-
Hive
-
HiveOnSpark
-
Impala
03-05-2017
10:02 PM
Thanks. I had look on the given links. But whatever metrics i noticed (still beginner in ts query) is yarn application metrics shows the rate 1/second like this. Is it possible to show the actual number to get a trend how many jobs/type are running on the cluster.
... View more
03-03-2017
01:32 AM
1) I want to create a dashboard in Cloudera Manager contanis yarn and hdfs charts. Yarn: weekly chart with daily numbers (y axis number and x axis date) 1) Total number of completed applications (succeeded+finished+failed+killed+error) and number based on application type (MR,SPARK,TEZ etc.) HDFS: 1) hdfs usage growth per week for a month as a line graph How to achieve it
... View more
Labels:
- Labels:
-
Cloudera Manager
02-15-2017
08:37 AM
Hi, Our set up has sentry enabled and recently added hive on spark as well.So after this it is very difficult to identify the application owner (always hive) and application query (always hive on spark)? This is making life difficult in identifying and troubleshooting apps. Any help on this or any alternate
... View more
Labels:
- Labels:
-
Hive
-
HiveOnSpark
-
Security
-
Sentry
02-15-2017
08:31 AM
Hi I wanted to list down all insert statements run in impala for a specific duration (impala queries), How do i query that. I am looking for something like statement like insert* (it should list all the statment which starts with insert). But this option not available. Pls suggest any alternatives.
... View more
Labels:
- Labels:
-
Cloudera Manager
-
Impala
06-27-2016
04:01 AM
Hi, How do i set up an alert for hosts when it crosses physical_memory_used a particular value? like 100 GB or something for 5 minutes.
... View more
06-07-2016
02:56 AM
Hi , I would like to set up alerts ( mail alert) for the below events . How to achieve this using cloudera manager? Whenever a job fails. When a job is in Accepted state for more than 10 minutes.
... View more
05-27-2016
12:45 AM
Is it possible to list hive tables based on the storage format? We want to implement a standard format for tables; before that we want to list down existing tables based on storage and identify which are not in standard format
... View more
- Tags:
- Hive