Member since
01-31-2019
26
Posts
7
Kudos Received
4
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 9502 | 01-30-2020 08:10 AM | |
| 5312 | 08-02-2019 02:35 AM | |
| 1928 | 04-24-2019 10:07 AM | |
| 7630 | 04-24-2019 02:27 AM |
03-24-2020
07:17 AM
1 Kudo
HI Rosa, Sorry I gave up that time because It was an urgent matter , so I just took the short way and used hive to hash my data and put it in a table , where I can run my queries later with Impala , I'll come back for it later for sure since hive is a bit slow while having java based functions. I'd recommend you to try with C language ,it's suitable for impala tho it will work faster . So please If you came up with anything share it with us , otherwise I'll post for sure my solution once it's done . best luck Bilal
... View more
01-30-2020
08:10 AM
1 Kudo
After some researches I did, it seems that Impala does not support GenericUDFs yet. https://issues.apache.org/jira/browse/IMPALA-7877 https://issues.apache.org/jira/browse/IMPALA-8369 so I'll just try to create my own function for Impala.
... View more
01-29-2020
08:35 AM
Hi all, I'm trying to create a function to use in imapla. my function is simply re-using hive's sha2() function. the creation of the function goes smoothly : create function to_sha2(string,int) returns string location 'user/hive/hive.jar' symbol='org.apache.hadoop.hive.ql.udf.generic.GenericUDFSha2' ;
but when I try to use it doesn't work launching this warning : select to_sha2('test',256);
Query State: EXCEPTION
Query Status: ClassCastException: class org.apache.hadoop.hive.ql.udf.generic.GenericUDFSha2
I have tried to search for UDFSha2 that doesn't contain the Generic word in the hive's jar but I couldn't find it. the original built-in function in hive : sha2(string/binary, len) - Calculates the SHA-2 family of hash functions (SHA-224, SHA-256, SHA-384, and SHA-512). | the other functions are working normally in impala (for example I have tried to create UDF MD5 function from hive's jar and it was working ). so my question is do I have to create my own UDF-Sha2 function? or there is a saving situation for my case, any help will be appreciated Impala version 2.9 Hive : 1.1.0 CDH : 5.12
... View more
Labels:
- Labels:
-
Apache Impala
08-02-2019
02:35 AM
hello @GopiG , have you tried setting the executor's and the driver's params in spark-defaults.conf ? spark.driver.extraJavaOptions -Duser.timezone=UTC spark.executor.extraJavaOptions -Duser.timezone=UTC you can set the default time zone UTC or any example you want like GMT+8 etc... cheers.
... View more
04-24-2019
10:07 AM
1 Kudo
hi @DataMike , The Balancer role is normally added (by default) when the HDFS service is installed, so the "Balancer" Resides usally in your nameNode , but to make sure where it's assigned you can check HDFS->Instances then check the role Type , you'll find the role 'balancer' assigned to a host ( usally it's the nameNode ) . for your second question I guess it's better to use the nameNode just to maintain the architecture since we're talking about checking all the other DataNodes,moving blocks...
... View more
04-24-2019
02:27 AM
@bgooley , thank you for your feedback and your clear explanation , in fact the problem was resolved by removing the contents of /var/lib/hadoop-yarn/yarn-nm-recovery/ directory and then the Nodemanager role started successfully. the solution that I've found was from : https://community.cloudera.com/t5/Batch-Processing-and-Workflow/Yarn-NodeManager-fails-to-start-and-crashing-with-SIGBUS/m-p/66590#M3611
... View more
04-23-2019
10:57 AM
hello folks , the nodeManager has suddently stopped in a instance (while stille running for other nodes/intances ). so when I try to start/restart it -via cloudera manager - , an error is shown in the first step : Failed to start role. and I'm using CentOS release 6.10 (Final) please what do you suggest me to look or check in order to resolve this problem ? here's my stdout log : Tue Apr 23 10:18:56 PDT 2019
JAVA_HOME=/usr/java/jdk.1.8.0_144
using /usr/java/jdk.1.8.0_144 as JAVA_HOME
using 5 as CDH_VERSION
using /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop-yarn as CDH_YARN_HOME
using /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop-mapreduce as CDH_MR2_HOME
using /var/run/cloudera-scm-agent/process/23960-yarn-NODEMANAGER as CONF_DIR
CONF_DIR=/var/run/cloudera-scm-agent/process/23960-yarn-NODEMANAGER
CMF_CONF_DIR=/etc/cloudera-scm-agent
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0x7) at pc=0x00007f8c1fde51a1, pid=3004, tid=0x00007f8c4f44c700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_144-b01) (build 1.8.0_144-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.144-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libleveldbjni-64-1-8170950501904951615.8+0x491a1] leveldb::ReadBlock(leveldb::RandomAccessFile*, leveldb::ReadOptions const&, leveldb::BlockHandle const&, leveldb::BlockContents*)+0x191
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
# and this is my log.out error : NodeManager Node Manager health check script is not available or doesn't have execute permission, so not starting the node health script runner. AsyncDispatcher Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher AsyncDispatcher Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher AsyncDispatcher Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService AsyncDispatcher Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices AsyncDispatcher Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl AsyncDispatcher Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
... View more
Labels:
- Labels:
-
Apache YARN