Member since
11-24-2017
76
Posts
8
Kudos Received
5
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2223 | 05-14-2018 10:28 AM | |
3844 | 03-28-2018 12:19 AM | |
1866 | 02-07-2018 02:54 AM | |
2513 | 01-26-2018 03:41 AM | |
3975 | 01-05-2018 02:06 AM |
12-16-2018
08:52 AM
Hi @csguna, CDH version is 5.13.2
... View more
12-16-2018
01:24 AM
Hi @Jerry, thank you for the reply. If I understand correctly you are saying that if not explicitly specified values for mapreduce.map.memory.mb and mapreduce.reduce.memory.mb YARN will assign to the job the minimum container memory value yarn.scheduler.minimum-allocation-mb, (1 GB in this case) ? Because from what I can read in the description fields on the Cloudera Manager, I though that if the values for mapreduce.map.memory.mb and mapreduce.reduce.memory.mb are left to zero, the memory assigned to a job should be inferred by the map maximum heap and heap to container ratio: Could you explain please how this work?
... View more
12-14-2018
02:41 AM
Hi everyone, I have a cluster where each worker has 110 GB of RAM.
On the Cloudera Manager I've configured the following Yarn memory parameters:
yarn.nodemanager.resource.memory-mb
80 GB
yarn.scheduler.minimum-allocation-mb
1 GB
yarn.scheduler.maximum-allocation-mb
20 GB
mapreduce.map.memory.mb
0
mapreduce.reduce.memory.mb
0
yarn.app.mapreduce.am.resource.mb
1 GB
mapreduce.job.heap.memory-mb.ratio
0,8
mapreduce.map.java.opts
-Djava.net.preferIPv4Stack=true
mapreduce.reduce.java.opts
-Djava.net.preferIPv4Stack=true
Map Task Maximum Heap Size
0
Reduce Task Maximum Heap Size
0
One of my goal was to let YARN to autochoose the correct Java Heap size for the jobs using the 0,8 ratio as the upperbound (20 GB * 0,8 = 16 GB), thus I've leave all the heap and mapper/reducer settings to zero.
I have this hive job which perfoms some joins between large tables. Just running the job as it is I get a failure:
Container [pid=26783,containerID=container_1389136889967_0009_01_000002] is running beyond physical memory limits. Current usage: 2.7 GB of 2 GB physical memory used; 3.7 GB of 3 GB virtual memory used. Killing container.
If I explicitly set the memory requirements for the job in the hive code, it completes succesfully:
SET mapreduce.map.memory.mb=8192;
SET mapreduce.reduce.memory.mb=16384;
SET mapreduce.map.java.opts=-Xmx6553m;
SET mapreduce.reduce.java.opts=-Xmx13106m;
My question: why does not YARN automatically gives this job enough memory to complete succesfully?
Since I have specified 20 GB as the maximum container size and 0,8 as the maximum heap ratio, I was expecting that YARN could give a max of 16 GB to each mapper/reducer without have to me esplicitly specify these parameters.
Could someone please explain what's going on?
Thanks for any information.
... View more
Labels:
11-26-2018
01:42 AM
Thank you very much @Harsh J! If I got it correctly these parameters oozie.launcher.mapreduce.map.java.opts oozie.launcher.mapreduce.reduce.java.opts oozie.launcher.yarn.app.mapreduce.am.command-opts control the maximum amount of memory allocated for the Oozie launcher. What are the equivalent parameters to control the memory allocated for the action instead (e.g. a Sqoop action), as shown in the image?
... View more
11-20-2018
02:19 AM
Hi @Harsh J, thank you very much for these informations (I am using Oozie server build version: 4.1.0-cdh5.13.2)! So if I understand correctly I need to add two properties in the oozie actions configuration, one specifying the launcher queue and one specifying the job queue. Below it is shown a sqoop action where I have added these two properties (in bold): <action name="DLT01V_VPAXINF_IMPORT_ACTION">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>oozie.launcher.mapred.job.queue.name</name>
<value>oozie_launcher_queue</value>
</property>
<property>
<name>mapred.job.queue.name</name>
<value>job_queue</value>
</property>
<property>
<name>oozie.launcher.mapreduce.map.java.opts</name>
<value>-Xmx4915m</value>
</property>
<property>
<name>oozie.launcher.mapreduce.reduce.java.opts</name>
<value>-Xmx9830m</value>
</property>
<property>
<name>oozie.launcher.yarn.app.mapreduce.am.command-opts</name>
<value>-Xmx4915m</value>
</property>
</configuration>
[...]
</sqoop>
[...]
</action> I have some questions: Do I need to define the queues "oozie_launcher_queue" and "job_queue" somewhere on the CDH or can I just use them providing the names? If yes, how should I define these queues? There are recommended settings? In case of a Spark action, do I still to specify the queue? If yes, with which property (since Spark does not use MapReduce)? Does it make sense to specify values for oozie.launcher.mapreduce.map.java.opts, oozie.launcher.mapreduce.reduce.java.opts, oozie.launcher.yarn.app.mapreduce.am.command-opts as I did in the example? I am asking because I've noticed in the Yarn ResourceManager that the Oozie launchers take a big amount of memory (about 30 GB each), is this normal? Thank you for the support!
... View more
11-19-2018
02:24 AM
Hello everyone! I have a typical scenario where there are multiple pipelines running on Oozie, each one with different dependencies and time schedules. These pipelines comprise different jobs like Hive, Spark, Java etc. Many of these jobs are heavy on memory, the cluster has a total of 840 GB of RAM, so let's say that the memory is enough to complete any of these jobs but could not be enough to allow several of these jobs to run and complete at the same time. Sometimes happen that few of these jobs need to run concurrently, in this case I've noticed a sort of starvation in YARN. None of the jobs continues the execution, there are a lot of heartbeats in the logs, and none eventually completes. YARN is set to use the Fair Scheduler, I would imagine that in a situation like this it should give resources at least to one of the job but it seems that all the jobs are fighting for resources and YARN is not capable to handle the impasse. I would like to know which are the best practices to handle these type of scenarios. Do I need to define different YARN queues with different resources/priority (actually all the jobs run on the default queue)?
... View more
Labels:
- Labels:
-
Apache Oozie
-
Apache YARN
08-01-2018
01:05 PM
Hello everyone, I have a Spark application which runs fine with test tables but fails in production where there are tables with 200 million records and about 100 columns. From the logs the error seems related to Snappy codec, although these tables have been saved in Parquet without compression, and also at write time I have explicitly turned off compression with: sqlContext.sql("SET hive.exec.compress.output=false")
sqlContext.sql("SET parquet.compression=NONE")
sqlContext.sql("SET spark.sql.parquet.compression.codec=uncompressed") The error is the following: 2018-08-01 16:19:45,467 [dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler - ShuffleMapStage 183 (saveAsTable at Model1Prep.scala:776) failed in 543.126 s due to Job aborted due to stage failure: Task 169 in stage 97.0 failed 4 times, most recent failure: Lost task 169.3 in stage 97.0 (TID 15079, prwor-e414c813.azcloud.local, executor 2): java.io.IOException: FAILED_TO_UNCOMPRESS(5)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391)
at org.xerial.snappy.Snappy.uncompress(Snappy.java:427)
at org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:127)
at org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
at org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:159)
at org.apache.spark.storage.BlockManager.wrapForCompression(BlockManager.scala:1280)
at org.apache.spark.util.collection.unsafe.sort.UnsafeSorterSpillReader.<init>(UnsafeSorterSpillReader.java:54)
at org.apache.spark.util.collection.unsafe.sort.UnsafeSorterSpillWriter.getReader(UnsafeSorterSpillWriter.java:148)
at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.getSortedIterator(UnsafeExternalSorter.java:416)
at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:117)
at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:170)
at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:90)
at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:64)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:88)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Why is this happenig if compression is turned off? Could it be compression is used anyway during shuffle phases? The cluster has the following characteristics: 2 master nodes 7 worker nodes Each node has: cpu: 16 cores ram: 110GB hdfs disks: 4x1TB These are the YARN settings for memory (GB): yarn.nodemanager.resource.memory-mb 84 yarn.scheduler.minimum-allocation-mb 12 yarn.scheduler.maximum-allocation-mb 84 mapreduce.map.memory.mb 6 mapreduce.reduce.memory.mb 12 mapreduce.map.java.opts 4,8 mapreduce.reduce.java.opts 9,6 yarn.app.mapreduce.am.resource.mb 6 yarn.app.mapreduce.am.command-opts 4,8 yarn.scheduler.maximum-allocation-vcores 5 SPARK on YARN settings: spark.shuffle.service.enabled: ENABLED spark.dynamicAllocation.enabled: ENABLED SPARK job submission settings: --driver-memory 30G --executor-cores 5 --executor-memory 30G Has anyone any hint on why is this happening?
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN
07-24-2018
06:59 AM
1 Kudo
You can use JSON Serde. You have to create the table with a structure that maps the structure of the json. For example: data.json {"X": 134, "Y": 55, "labels": ["L1", "L2"]}
{"X": 11, "Y": 166, "labels": ["L1", "L3", "L4"]} create table CREATE TABLE Point
(
X INT,
Y INT,
labels ARRAY<STRING>
)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
STORED AS TEXTFILE
LOCATION 'path/to/table'; Then you should upload your json file in the location path of the table, giving the right permissions and you are good to go.
... View more
05-27-2018
03:11 PM
up! Has anyone succesfully imported a Teradata table in avro format using the Cloudera connector?
... View more
05-23-2018
01:37 AM
Thank you very much @Harsh J for the detailed answer. I will forward it to the cluster administrators, hoping they will follow the loadbalancer way you suggested ^_^.
... View more
05-23-2018
01:26 AM
Thanks, I indeed end up using Maven and plugins.d folder on Flume. Forgot to update the topic, thank you guys for the help!
... View more
05-18-2018
06:40 AM
Hello everyone, I have High Availabity of HiveServer2 enabled on a kerberized cluster.
I can succesfully connect to beeline with the following command:
beeline -u "jdbc:hive2://trmas-fc2d552a.azcloud.local:2181,trmas-c9471d78.azcloud.local:2181,trmas-6b8bc78c.azcloud.local:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;ssl=true;"
My problem is when I try to use Hive2 credentials in Oozie (which afaik uses beeline and jdbc connection as above):
<credentials>
<credential name="hive2_credentials" type="hive2">
<property>
<name>hive2.jdbc.url</name>
<value>jdbc:hive2://trmas-fc2d552a.azcloud.local:2181,trmas-c9471d78.azcloud.local:2181,trmas-6b8bc78c.azcloud.local:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;ssl=true;</value>
</property>
<property>
<name>hive2.server.principal</name>
<value>hive/_HOST@AZCLOUD.LOCAL</value>
</property>
</credential>
</credentials>
I took the value of hive2.server.principal fron the hive.server2.authentication.kerberos.principal property in the hive-site.xml, is this correct?
This is the hive2 oozie action:
<action cred="hive2_credentials" name="HIVE2_ACTION_NODE">
<hive2 xmlns="uri:oozie:hive2-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<job-xml>${package}/hive-site.xml</job-xml>
<jdbc-url>jdbc:hive2://trmas-fc2d552a.azcloud.local:2181,trmas-c9471d78.azcloud.local:2181,trmas-6b8bc78c.azcloud.local:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;ssl=true;</jdbc-url>
<script>${package}/my_query.hql</script>
<param>nameNode=${nameNode}</param>
</hive2>
<ok to="END_NODE"/>
<error to="KILL_NODE"/>
</action>
I got this error when running the workflow:
Connecting to jdbc:hive2://trmas-fc2d552a.azcloud.local:2181,trmas-c9471d78.azcloud.local:2181,trmas-6b8bc78c.azcloud.local:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;ssl=true;
Error: Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 configs from ZooKeeper (state=08S01,code=0)
No current connection
Intercepting System.exit(2)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive2Main], exit code [2]
Does anyone know how to solve this issue?
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Oozie
-
Kerberos
05-15-2018
03:07 AM
I don't understand how includePattern works in Flume. It should accept a regular expression, but the only one that is working to me is the following: # accepts only .zip files
agent.sources.src1.includePattern=^.*\.zip$ But if for example I want to match only files with format WWWW.DDDDD.zip (e.g. abc.123.zip) the followings does not work: agent.sources.src1.includePattern=^[^\.]+\.[^\.]+\.zip$ agent.sources.src1.includePattern=^\w+\.\d+\.zip$ Does anyone know how regex expressions work in Flume?
... View more
Labels:
- Labels:
-
Apache Flume
05-14-2018
10:28 AM
Thanks @Harsh J, indeed I've finally solved using hdfs://hanameservice for name node and yarnrm for the job tracker.
... View more
05-10-2018
03:06 AM
Hello everyone, I have a cluster with HDFS High Availability (HA) enabled. The cluster has two NameNode, one active and on in standby state, plus 3 journal nodes, a balancer and failover controllers. My question: how should I configure Oozie workflows for nameNode and jobTracker parameters in job.properties file in order to point always to the active NameNode and JobTracker (in case of a failure or a manual switch of the NameNode)? Thanks for any information
... View more
Labels:
- Labels:
-
Apache Oozie
-
HDFS
04-17-2018
06:37 AM
@Harsh J Thank you, unfortunately I have access only to edge node (I can't ssh to masters and workers). I have access to web interfaces though (CM, HUE, Yarn, etc) thus if there is anything I can check from there let me know.
... View more
04-17-2018
04:23 AM
@Harsh J In Cloudera Manager I went to Oozie server instance and check logs from there but there is nothing useful in Stdout and Stderr, are these the logs you were talking about? Also I am not sure where can I find logs about HMS, can you provide some details?
... View more
04-16-2018
07:58 AM
@Harsh J Thank you very much for the answer, I was able to generate the keytab file and put it in the lib folder of the Oozie application, but when I run I got the following error: Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Login failure for user: icon0104@AZCLOUD.LOCAL from keytab icon0104.keytab javax.security.auth.login.LoginException: Pre-authentication information was invalid (24)
org.apache.hadoop.security.KerberosAuthException: Login failure for user: icon0104@AZCLOUD.LOCAL from keytab icon0104.keytab javax.security.auth.login.LoginException: Pre-authentication information was invalid (24)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1130)
at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:562)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:154)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:178)
at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:90)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:81)
at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:57)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:235)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) I am also a bit confused about all this procedure, I mean shouldn't Oozie be able to get a Kerberos delegation token on behalf of my user without the need for me to provide the keytab file? I have also tried (again) to use hcat credentials specifiying the configuration you suggested in the other post with the following workflow: <workflow-app xmlns="uri:oozie:workflow:0.5" name="oozie_spark_wf">
<credentials>
<credential name="hive2_credentials" type="hive2">
<property>
<name>hive2.jdbc.url</name>
<value>jdbc:hive2://trmas-fc2d552a.azcloud.local:10000/default;ssl=true</value>
</property>
<property>
<name>hive2.server.principal</name>
<value>hive/trmas-fc2d552a.azcloud.local@AZCLOUD.LOCAL</value>
</property>
</credential>
<credential name="hcat_cred" type="hcat">
<property>
<name>hcat.metastore.uri</name>
<value>thrift://trmas-fc2d552a.azcloud.local:9083</value>
</property>
<property>
<name>hcat.metastore.principal</name>
<value>hive/trmas-fc2d552a.azcloud.local@AZCLOUD.LOCAL</value>
</property>
</credential>
</credentials>
<start to="spark_action"/>
<action cred="hcat_cred" name="spark_action">
<spark xmlns="uri:oozie:spark-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="hdfs://trmas-6b8bc78c.azcloud.local:8020/user/icon0104/spark_hive_output"/>
</prepare>
<configuration>
<property>
<name>spark.yarn.security.tokens.hive.enabled</name>
<value>false</value>
</property>
</configuration>
<master>yarn-cluster</master>
<name>OozieSparkAction</name>
<class>my.Main</class>
<jar>/home/icon0104/oozie/ooziespark/lib/ooziespark-1.0.jar</jar>
<spark-opts>--files ${nameNode}/user/icon0104/oozie/ooziespark/hive-site.xmlL</spark-opts>
</spark>
<ok to="END_NODE"/>
<error to="KILL_NODE"/>
</action>
<kill name="KILL_NODE">
<message>${wf:errorMessage(wf:lastErrorNode())}</message>
</kill>
<end name="END_NODE"/>
</workflow-app> But the Spark action goes in START_RETRY state with the same error : JA009: org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred while processing HCat request : TException while getting delegation token.. Cause : org.apache.thrift.transport.TTransportException Thanks for the support!
... View more
04-16-2018
05:12 AM
@saranvisa Thank you, if I uderstand correctly I need to provide a keytab file on HDFS and pass it as a file in the Oozie Spark action. What I am missing here is how can generate this keytab file as non proviliged user. I can kinit but I have no privileges for kadmin command. Do I need to contact an administrator or are other ways to get this keytab file?
... View more
04-16-2018
04:07 AM
@saranvisa @suresh.sethu I've tried to kinit before launching Oozie Spark action in yarn-cluster mode but it fails anyway. In the logs I found a lot of the following warnings: 2018-04-16 10:59:05,874 [main] INFO org.apache.spark.deploy.yarn.YarnSparkHadoopUtil - getting token for namenode: hdfs://hanameservice/user/icon0104/.sparkStaging/application_1523441517429_3067
2018-04-16 10:59:06,004 [main] WARN org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException as:icon0104 (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error and the following exception: diagnostics: User class threw exception: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient I've also tried to run the spark program directly from the shell with spark-submit using --master yarn-cluster but got the following error: Delegation Token can be issued only with kerberos or web authentication Any idea how to solve?
... View more
04-16-2018
01:55 AM
Thank you, exactly what I was thinking. With all queries aggregated in one script I gain speed (no overhead on Yarn containers) but in case of error I loose granularity for debug.
... View more
04-15-2018
12:51 AM
Hello everyone, when performing Hive commands inside Oozie is it ok to aggregate them in one script, or it is better to split up in different Hive action/script? For example I need to create several views, shoould I put each view creation in a distinct Hive action/script or can I put all the views creation in a single one? Which is the best practice and why?
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Oozie
04-11-2018
12:30 AM
Any feedback from Cloudera guys?
... View more
04-10-2018
11:45 AM
Thank you very much for the detailed answer @mzkdm. This is indeed a very interesting point. Do you think could make sense to have daily-based partitions, since my main ingestion workflow run once a day? And how can I force Hive or Impala users to use the last point-in-time data? Thanks for the help!
... View more
04-09-2018
12:11 PM
Hi @saranvisa, thanks for the answer. Do I need to do this every time before running the Oozie Spark action? Because this is a coordinator-scheduled workflow that I need to run several times per day.
... View more
04-06-2018
01:59 AM
Hello everyone
I need to run a Spark-SQL action on an Hive table. I am having problems on authentication (the cluster is Kerberos-secured).
I've tried first with hive2 credentials because they work with my other hive2 actions, my I got a failure (I suppose this type of credentials can only be used with hive2 actions?):
2018-04-06 08:37:21,831 [Driver] INFO org.apache.hadoop.hive.ql.session.SessionState - No Tez session required at this point. hive.execution.engine=mr.
2018-04-06 08:37:22,117 [Driver] INFO hive.metastore - Trying to connect to metastore with URI thrift://trmas-fc2d552a.azcloud.local:9083
2018-04-06 08:37:22,153 [Driver] ERROR org.apache.thrift.transport.TSaslTransport - SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
[...]
I've also tried with hcat credentials, but with this one I got a START_RETRY state of the actin with the following error:
JA009: org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred while processing HCat request : TException while getting delegation token.. Cause : org.apache.thrift.transport.TTransportException
This is the workflow.xml:
<workflow-app
xmlns="uri:oozie:workflow:0.5" name="oozie_spark_wf">
<credentials>
<credential name="hive2_credentials" type="hive2">
<property>
<name>hive2.jdbc.url</name>
<value>jdbc:hive2://trmas-fc2d552a.azcloud.local:10000/default;ssl=true</value>
</property>
<property>
<name>hive2.server.principal</name>
<value>hive/trmas-fc2d552a.azcloud.local@AZCLOUD.LOCAL</value>
</property>
</credential>
<credential name="hcat_cred" type="hcat">
<property>
<name>hcat.metastore.uri</name>
<value>thrift://trmas-fc2d552a.azcloud.local:9083</value>
</property>
<property>
<name>hcat.metastore.principal</name>
<value>hive/trmas-fc2d552a.azcloud.local@AZCLOUD.LOCAL</value>
</property>
</credential>
</credentials>
<start to="spark_action"/>
<action cred="hcat_cred" name="spark_action">
<spark
xmlns="uri:oozie:spark-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${nameNode}/user/icon0104/output"/>
</prepare>
<master>yarn-cluster</master>
<mode>cluster</mode>
<name>OozieSpark</name>
<class>my.Main</class>
<jar>/home/icon0104/oozie/ooziespark/lib/ooziespark-1.0.jar</jar>
<spark-opts>--files ${nameNode}/user/icon0104/oozie/ooziespark/hive-site.xml</spark-opts>
</spark>
<ok to="END_NODE"/>
<error to="KILL_NODE"/>
</action>
<kill name="KILL_NODE">
<message>${wf:errorMessage(wf:lastErrorNode())}</message>
</kill>
<end name="END_NODE"/>
</workflow-app>
This is the hive-site.xml:
<?xml version="1.0" encoding="UTF-8"?>
<!--Autogenerated by Cloudera Manager-->
<configuration>
<property>
<name>hive.metastore.uris</name>
<value>thrift://trmas-fc2d552a.azcloud.local:9083</value>
</property>
<property>
<name>hive.metastore.client.socket.timeout</name>
<value>300</value>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
</property>
<property>
<name>hive.warehouse.subdir.inherit.perms</name>
<value>true</value>
</property>
<property>
<name>hive.auto.convert.join</name>
<value>false</value>
</property>
<property>
<name>hive.auto.convert.join.noconditionaltask.size</name>
<value>20971520</value>
</property>
<property>
<name>hive.optimize.bucketmapjoin.sortedmerge</name>
<value>false</value>
</property>
<property>
<name>hive.smbjoin.cache.rows</name>
<value>10000</value>
</property>
<property>
<name>hive.server2.logging.operation.enabled</name>
<value>true</value>
</property>
<property>
<name>hive.server2.logging.operation.log.location</name>
<value>/var/log/hive/operation_logs</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>-1</value>
</property>
<property>
<name>hive.exec.reducers.bytes.per.reducer</name>
<value>67108864</value>
</property>
<property>
<name>hive.exec.copyfile.maxsize</name>
<value>33554432</value>
</property>
<property>
<name>hive.exec.reducers.max</name>
<value>1099</value>
</property>
<property>
<name>hive.vectorized.groupby.checkinterval</name>
<value>4096</value>
</property>
<property>
<name>hive.vectorized.groupby.flush.percent</name>
<value>0.1</value>
</property>
<property>
<name>hive.compute.query.using.stats</name>
<value>false</value>
</property>
<property>
<name>hive.vectorized.execution.enabled</name>
<value>true</value>
</property>
<property>
<name>hive.vectorized.execution.reduce.enabled</name>
<value>false</value>
</property>
<property>
<name>hive.merge.mapfiles</name>
<value>true</value>
</property>
<property>
<name>hive.merge.mapredfiles</name>
<value>false</value>
</property>
<property>
<name>hive.cbo.enable</name>
<value>false</value>
</property>
<property>
<name>hive.fetch.task.conversion</name>
<value>minimal</value>
</property>
<property>
<name>hive.fetch.task.conversion.threshold</name>
<value>268435456</value>
</property>
<property>
<name>hive.limit.pushdown.memory.usage</name>
<value>0.1</value>
</property>
<property>
<name>hive.merge.sparkfiles</name>
<value>true</value>
</property>
<property>
<name>hive.merge.smallfiles.avgsize</name>
<value>16777216</value>
</property>
<property>
<name>hive.merge.size.per.task</name>
<value>268435456</value>
</property>
<property>
<name>hive.optimize.reducededuplication</name>
<value>true</value>
</property>
<property>
<name>hive.optimize.reducededuplication.min.reducer</name>
<value>4</value>
</property>
<property>
<name>hive.map.aggr</name>
<value>true</value>
</property>
<property>
<name>hive.map.aggr.hash.percentmemory</name>
<value>0.5</value>
</property>
<property>
<name>hive.optimize.sort.dynamic.partition</name>
<value>false</value>
</property>
<property>
<name>hive.execution.engine</name>
<value>mr</value>
</property>
<property>
<name>spark.executor.memory</name>
<value>268435456</value>
</property>
<property>
<name>spark.driver.memory</name>
<value>268435456</value>
</property>
<property>
<name>spark.executor.cores</name>
<value>4</value>
</property>
<property>
<name>spark.yarn.driver.memoryOverhead</name>
<value>26</value>
</property>
<property>
<name>spark.yarn.executor.memoryOverhead</name>
<value>26</value>
</property>
<property>
<name>spark.dynamicAllocation.enabled</name>
<value>true</value>
</property>
<property>
<name>spark.dynamicAllocation.initialExecutors</name>
<value>1</value>
</property>
<property>
<name>spark.dynamicAllocation.minExecutors</name>
<value>1</value>
</property>
<property>
<name>spark.dynamicAllocation.maxExecutors</name>
<value>2147483647</value>
</property>
<property>
<name>hive.metastore.execute.setugi</name>
<value>true</value>
</property>
<property>
<name>hive.support.concurrency</name>
<value>true</value>
</property>
<property>
<name>hive.zookeeper.quorum</name>
<value>trmas-6b8bc78c.azcloud.local,trmas-c9471d78.azcloud.local,trmas-fc2d552a.azcloud.local</value>
</property>
<property>
<name>hive.zookeeper.client.port</name>
<value>2181</value>
</property>
<property>
<name>hive.zookeeper.namespace</name>
<value>hive_zookeeper_namespace_CD-HIVE-LTqXUcrR</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>trmas-6b8bc78c.azcloud.local,trmas-c9471d78.azcloud.local,trmas-fc2d552a.azcloud.local</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hive.cluster.delegation.token.store.class</name>
<value>org.apache.hadoop.hive.thrift.MemoryTokenStore</value>
</property>
<property>
<name>hive.server2.enable.doAs</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.sasl.enabled</name>
<value>true</value>
</property>
<property>
<name>hive.server2.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>hive.metastore.kerberos.principal</name>
<value>hive/_HOST@AZCLOUD.LOCAL</value>
</property>
<property>
<name>hive.server2.authentication.kerberos.principal</name>
<value>hive/_HOST@AZCLOUD.LOCAL</value>
</property>
<property>
<name>hive.server2.use.SSL</name>
<value>true</value>
</property>
<property>
<name>spark.shuffle.service.enabled</name>
<value>true</value>
</property>
</configuration>
In Oozie configuration I have the following credentials classes enabled:
hcat=org.apache.oozie.action.hadoop.HCatCredentials,hbase=org.apache.oozie.action.hadoop.HbaseCredentials,hive2=org.apache.oozie.action.hadoop.Hive2Credentials
Can anyone help? What am I missing?
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Oozie
-
Apache Spark
-
Kerberos
04-06-2018
01:43 AM
I've solved the jdbc issue by enabling SSL in the connection string: jdbc:impala://trwor-dafb587f.azcloud.local:21050;SSL=1;AuthMech=1;KrbAuthType=0;KrbHostFQDN=trwor-dafb587f.azcloud.local;KrbServiceName=impala Still not luck with impala-shell connection. If I run "klist" I got: Ticket cache: FILE:/tmp/krb5cc_699006375_ASnf44
Default principal: icon0104@AZCLOUD.LOCAL
Valid starting Expires Service principal
04/06/2018 08:38:44 04/06/2018 18:38:44 krbtgt/AZCLOUD.LOCAL@AZCLOUD.LOCAL
renew until 04/13/2018 08:38:44 Thanks for the support
... View more
04-02-2018
12:41 PM
Update: I've tried to switch to ClouderaImpalaJDBC_2.5.43.1063 driver (using JDBC41). With the following connection string (to infer authentication): jdbc:impala://trwor-dafb587f.azcloud.local:21050;AuthMech=1;KrbAuthType=0;KrbHostFQDN=trwor-dafb587f.azcloud.local;KrbServiceName=impala Now the error shown is the following: java.sql.SQLException: [Simba][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Simba][ImpalaJDBCDriver](500169) Unable to connect to server: [Simba][ImpalaJDBCDriver](500591) Kerberos Authentication failed..
at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source)
at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createClient(Unknown Source)
at com.cloudera.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source)
at com.cloudera.impala.core.ImpalaJDBCConnection.establishConnection(Unknown Source)
at com.cloudera.jdbc.core.LoginTimeoutConnection.connect(Unknown Source)
at com.cloudera.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
at com.cloudera.jdbc.common.AbstractDriver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:233)
at ico.az.deploy.TestSuite.testTeradata(TestSuite.java:101)
at ico.az.deploy.TestSuite.run(TestSuite.java:314)
Caused by: com.cloudera.support.exceptions.GeneralException: [Simba][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Simba][ImpalaJDBCDriver](500169) Unable to connect to server: [Simba][ImpalaJDBCDriver](500591) Kerberos Authentication failed..
... 11 more
Caused by: java.lang.RuntimeException: [Simba][ImpalaJDBCDriver](500169) Unable to connect to server: [Simba][ImpalaJDBCDriver](500591) Kerberos Authentication failed.
at com.cloudera.hivecommon.api.HiveServerPrivilegedAction.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source)
at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createClient(Unknown Source)
at com.cloudera.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source)
at com.cloudera.impala.core.ImpalaJDBCConnection.establishConnection(Unknown Source)
at com.cloudera.jdbc.core.LoginTimeoutConnection.connect(Unknown Source)
at com.cloudera.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
at com.cloudera.jdbc.common.AbstractDriver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:233)
at ico.az.deploy.TestSuite.testTeradata(TestSuite.java:101)
at ico.az.deploy.TestSuite.run(TestSuite.java:314)
at ico.az.deploy.TestSuite.main(TestSuite.java:350)
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:258)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
... 15 more Please let me know if is there anything else I can try.
... View more
04-02-2018
11:15 AM
Hello everybody I am working on a CDH 5.13.2 cluster configured with Kerberos and LDAP authentication. I need to connect to Impala thorugh jdbc and impala-shell, but I am having problems on both (Impala queries on HUE work fine). For impala-shell I've tried: impala-shell -k -i trwor-b9a4f2a7.azcloud.local ---> Starting Impala Shell using Kerberos authentication Using service name 'impala' Error connecting: TTransportException, TSocket read 0 bytes *********************************************************************************** Welcome to the Impala shell. (Impala Shell v2.10.0-cdh5.13.2 (dc867db) built on Fri Feb 2 10:46:38 PST 2018) I've also tried without Kerberos: impala-shell -i trwor-b9a4f2a7.azcloud.local
--->
Starting Impala Shell without Kerberos authentication
Error connecting: TTransportException, TSocket read 0 bytes
Kerberos ticket found in the credentials cache, retrying the connection with a secure transport.
Error connecting: TTransportException, TSocket read 0 bytes
*********************************************************************************** Welcome to the Impala shell. (Impala Shell v2.10.0-cdh5.13.2 (dc867db) built on Fri Feb 2 10:46:38 PST 2018)
In both cases I got a TTransportException. I am having trouble also for connecting to Impala through jdbc (using Cloudera_ImpalaJDBC4_2.5.5.1007 driver): String impalaConnectionUrl = "jdbc:impala://trwor-dafb587f.azcloud.local:21050;AuthMech=1;KrbRealm=AZCLOUD.LOCAL;KrbHostFQDN=trwor-dafb587f.azcloud.local;KrbServiceName=impala";
try {
Connection impalaConn = DriverManager.getConnection(impalaConnectionUrl);
[...]
}
catch (SQLEception ex) {
[...]
} ----> java.sql.SQLException: [Simba][ImpalaJDBCDriver](500310) Invalid operation: Unable to connect to server:; at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createTransport(HiveServer2ClientFactory.java:224) at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createClient(HiveServer2ClientFactory.java:52) at com.cloudera.impala.hivecommon.core.HiveJDBCConnection.connect(HiveJDBCConnection.java:597) at com.cloudera.impala.jdbc.common.BaseConnectionFactory.doConnect(BaseConnectionFactory.java:219) at com.cloudera.impala.jdbc.common.AbstractDriver.connect(AbstractDriver.java:216) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:233) at ico.az.deploy.TestSuite.testTeradata(TestSuite.java:98) at ico.az.deploy.TestSuite.run(TestSuite.java:311) Caused by: com.cloudera.impala.support.exceptions.GeneralException: [Simba][ImpalaJDBCDriver](500310) Invalid operation: Unable to connect to server:; ... 9 more Caused by: java.lang.RuntimeException: Unable to connect to server: at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory$1.run(HiveServer2ClientFactory.java:150) at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory$1.run(HiveServer2ClientFactory.java:141) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:356) at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createTransport(HiveServer2ClientFactory.java:140) at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createClient(HiveServer2ClientFactory.java:52) at com.cloudera.impala.hivecommon.core.HiveJDBCConnection.connect(HiveJDBCConnection.java:597) at com.cloudera.impala.jdbc.common.BaseConnectionFactory.doConnect(BaseConnectionFactory.java:219) at com.cloudera.impala.jdbc.common.AbstractDriver.connect(AbstractDriver.java:216) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:233) at ico.az.deploy.TestSuite.testTeradata(TestSuite.java:98) at ico.az.deploy.TestSuite.run(TestSuite.java:311) at ico.az.deploy.TestSuite.main(TestSuite.java:347) Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:258) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory$1.run(HiveServer2ClientFactory.java:146) ... 13 more Regarding connection string parameters: hostname : the host name where is running an Impala daemon, I took this one from Cloudera Manager->Impala->Instances->Impala daemon (there is one deamon running in each worker node, thus I've just choosen the first one). port : taken from Impala Daemon HiveServer2 Port property property in the Impala Configuration. AuthMech : according to the jdbc driver documentation 1 is for Kerberos authentication. KrbRealm : I took this one from the param default_realm in the /etc/krb5.conf file on the edge node, is this correct? KrbHostFQDN : same as Impala daemon hostname, correct? KrbServiceName : should be "impala" the default, and it is also the nameof Impala Kerberos Principal on the CM, correct? These are the relevant properties I found on the Cloudera Manager (read only access) for Impala and Kerberos: I am trying Kerberos authentication because it seems LDAP authentication is disabled for Impala: What am I doing wrong?
... View more
Labels:
- Labels:
-
Apache Impala
-
Cloudera Manager
-
Kerberos