Hello All,    I am new to Spark environment. I have converted Hive query to Spark Scala. Now I am loading data and doing performance testing. Below are details on loading 3 weeks data. Cluster level small file avg size is set to 128 MB.  1. New temp table where I am loading data is ORC formatted as current Hive table is ORC stored.  2. Hive table each partition folder size is 200 MB.  3. I am using repartition(1) in spark code so that it will create one 200MB part file in each partition folder(to avoid small file issue). With this job is completing in 23 to 26 mins.  4. If I don't use repartition(), job is completing in 12 to 13 mins. But problem with this approach is, it is creating 800 part files (size <128MB) in each partition folder.    I am quite not sure on how to reduce processing time and not create small files at the same time. Could anyone please help me in this situation.  ... View more

Hello Community,    I am seeing different value at Teradata and Hive when I trigger same query.   Teradata:  SELECT CASE WHEN 6240.00 <> 0 AND (30/(6240.00)) > 0 THEN Y ELSE N END AS "Col1" Output: N Hive: SELECT CASE WHEN 6240 <> 0 AND (30/6240) > 0 THEN Y ELSE N END AS "Col1" Output: Y   Can anyone please help me solving this? I need Hive result to be written as Teradata one.    Thank you.  ... View more

We are migrating to new cluster. We established hostname switch using DNS forwarding.We wanted to have hostname testing for every service that we have so that users will use the same old hostnames to connect to new cluster.   Hostname testing for impala worked as expected when checked with impala-shell command. Currently, we are testing for hive.   HiveServer2 is running on <New_FQDN> in New Cluster. Mapped old server for this is different <Old_FQDN>   Beeline String is only working if we give new server principal name.   !connect jdbc:hive2://<Old_FQDN>:10000/default;principal=hive/<New_FQDN>@REALM   We are using Active Directory. How can we make this work with old server principal.  ... View more

We are migrating to new cluster. We performed HostName Redirection by alaising every old server in old cluster alias to every new server in new cluster. This way users will use the same old hostnames to connect to new cluster.    I am currently testing HostName testing for every service that we have. Hostname testing for impala worked as expected when checked with impala-shell command.    I am able to check all the jobs ran using Job History Server URL which has new server hostname. But when I give equivalent hostname that we mapped during hostname reidrection, it is saying this.    HTTP ERROR 403 Problem accessing /jobhistory. Reason: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)   Powered by Jetty://   Same is the case when I try to access Resource Manager and Namenode UI.  ... View more

Hi,    HDFS and Hive Replication schedule is failing with beloew error.    18/08/22 19:40:02 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:PROXY) via hdfs/<FQDN>@realm (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.YarnRuntimeException): org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Could not load history file hdfs://nameservice9:8020/user/history/done_intermediate/hdfs/job_1534964122122_0006-1534978377784-hdfs-HdfsReplication-1534981192465-20-1-SUCCEEDED-root.users.hdfs-1534978383044.jhist at org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:199) at org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:218) at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:208) at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler$1.run(HistoryClientService.java:204) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:204) at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getJobReport(HistoryClientService.java:236) at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getJobReport(MRClientProtocolPBServiceImpl.java:122) at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:275) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2220) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2214) Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Could not load history file hdfs://nameservice9:8020/user/history/done_intermediate/hdfs/job_1534964122122_0006-1534978377784-hdfs-HdfsReplication-1534981192465-20-1-SUCCEEDED-root.users.hdfs-1534978383044.jhist at org.apache.hadoop.mapreduce.v2.hs.CompletedJob.loadFullHistoryData(CompletedJob.java:341) at org.apache.hadoop.mapreduce.v2.hs.CompletedJob.<init>(CompletedJob.java:101) at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager$HistoryFileInfo.loadJob(HistoryFileManager.java:479) at org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.loadJob(CachedHistoryStorage.java:180) at org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.access$000(CachedHistoryStorage.java:52) at org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage$1.load(CachedHistoryStorage.java:103) at org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage$1.load(CachedHistoryStorage.java:100) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228) at com.google.common.cache.LocalCache.get(LocalCache.java:3965) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829) at com.google.common.cache.LocalCache$LocalManualCache.getUnchecked(LocalCache.java:4834) at org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:193) ... 18 more Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=READ, inode="/user/history/done_intermediate/hdfs/job_1534964122122_0006-1534978377784-hdfs-HdfsReplication-1534981192465-20-1-SUCCEEDED-root.users.hdfs-1534978383044.jhist":hdfs:supergroup:-rwxrwx--- ... View more