Member since
01-05-2016
55
Posts
37
Kudos Received
6
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 573 | 10-21-2019 05:16 AM | |
| 3422 | 01-29-2018 07:05 AM | |
| 2168 | 06-27-2017 06:42 AM | |
| 35645 | 05-26-2016 04:05 AM | |
| 24277 | 05-17-2016 02:15 PM |
05-26-2020
05:01 AM
Hi, as we all know CDH 6.3.3 and subsequent versions are not available anymore under "Express licensing" model. Yet, I was in the process of setting up a 6.3.1 installation and apparently this is not possible, because a valid authentication (enterprise) is required. What am I doing wrong? Below, a screenshot from the Installation Guide: But when I try to import the 6.3.1 Repository (WITHOUT Username and Password as stated on the documentation itself for version < 6.3.3 I get the following error: # rpm --import https://archive.cloudera.com/p/cm6/6.3.1/redhat7/yum/RPM-GPG-KEY-cloudera
curl: (22) The requested URL returned error: 401 Authentication required
errore: https://archive.cloudera.com/p/cm6/6.3.1/redhat7/yum/RPM-GPG-KEY-cloudera: lettura importazione fallita(2). What am I doing wrong? @Cloudera1 Is this a mistake I'm committing or is it a problem with Cloudera implementing their policies in a wrong way? Thanks for any insights
... View more
Labels:
10-21-2019
05:16 AM
1 Kudo
You can query the API exposed by Cloudera Manager and simplify your life. For example, you can run the following: curl -u <CM_USER>:<CM_PASSWD> http://<CM_IP_ADDRESS>:7180/api/v19/clusters/<CLUSTER_NAME>/services/hive2 You'll get a Json answer in reply to your Query, with all the details related to the desired service's status. You can finally parse your Json answer (e.g. using "jq" or directly inside your bash script) and take the desired actions HTH
... View more
- Tags:
- The
10-20-2019
12:32 PM
Hello Community, after upgrading from CDH 5.11 to CDH 6.3 I've noticed that when launching an Oozie Workflow from the HUE GUI, I can't see the "progress bars" on the single Actions making up a Workflow Graph. I have just one progress bar in the last Action of the Graph, but it doesn't help in understanding the real progress in complex Workflows made up of many sub-actions, maybe taking place in parallel etc. It shouldn't be a browser issue, as I've tried on Chrome, Safari and Firefox Am I doing something wrong here? Configurations parameters that I should set up in HUE 4.4.0 Config, Oozie Config etc? Thanks in advance for any hints and helpful insights!
... View more
Labels:
- Labels:
-
Cloudera Hue
11-12-2018
12:09 PM
Hi, late reply but I hope iy can still be useful. To achieve what you want you should do something like: dataframe2 = dataframe1.persist(StorageLevel.MEMORY_AND_DISK) HTH
... View more
01-29-2018
07:26 AM
1) Apparently, yes 2) The name of the user you're trying to use to log in to the remote system, I suppose. Pls note that the user you specify here would be the user "oozie" will run as, so you'd eventually get other problems of unpredictable nature when using Oozie 3) I don't really know, sorry about that... The fact is that even if I'm pretty sure to have understood the cause of your issue, I never had to deal with it directly myself. Maybe the easier way could be to follow the additional suggestions I wrote in my previous answer (give permissions to OS User "yarn" to "ssh" and/or "su"). Or, maybe, another possibility would be for you to create a "yarn" user on the remote system and grant this user with the correct permissions to get to the final working directory I hope you'll manage to get through the problems and make it 🙂
... View more
01-29-2018
07:05 AM
This is probably related to the fact that the shell action, when run from Oozie, runs as user "yarn" and not as the desired user you're specifying in the ssh command You can refer to this thread for more information about the issue: https://community.cloudera.com/t5/Batch-Processing-and-Workflow/How-to-run-Oozie-workfllow-or-action-as-another-user/td-p/26794 It should all boil down (in case your cluster is not secured with kerberos) to try and set up your environment, specifically the " linux-container-executor" configuration parameter (you go in Cloudera Admin UI --> Yarn --> Configuration). It's all explained in the linked document. Another alternative could be to grant OS user "yarn" with permissions to execute "ssh" and/or "su" so you can eventually switch user in your script before executing the remote ssh command HTH
... View more
09-26-2017
09:14 AM
Have you tried a configuration similar to: - 3 executors (1 per datanode) - Setting a very low initial memory setting for the executors (e.g. 1 GBytes) - Limiting the VCPU to 1 f or the executors (at least initially) - Maybe you can try with 2 VCPUs and 2 GBytes for the driver initially - It's important to allow a conspicuous overhead to the amount of RAM usable by the executors if they run out of initial resources (that 1 GBytes we specified before), but just when/if they need it. So set the following parameters accordingly: spark.yarn.executor.memoryOverhead = 8 GBytes spark.yarn.driver.memoryOverhead = 8 GBytes - You can read the following docs to get a better grasp on the concepts behind resources allocation: https://www.cloudera.com/documentation/enterprise/5-11-x/topics/cdh_ig_yarn_tuning.html https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_running_spark_on_yarn.html The first Doc also includes a link to a "Yarn Tuning Spreadsheet". The Docs contain details on how to configure all the features I've been talking about just above 🙂 HTH
... View more
07-22-2017
08:45 AM
Thanks mbigelow, following your suggestions I solved the massive error logging issue. I've processed in a Json validator the specific log file referenced in the Java stack trace: /user/spark/applicationHistory/application_1494352758818_0117_1 But the format was correct, according to the validator. So I just moved it away in a temporary directory. As soon as I did it, the error messages stopped clogging the system logs. So it was probably corrupted in a very subtle way... But it was definitely corrupted That Json file has been indeed generated by the Spark Action that is giving me problems, but it was an OLD file. New instances of that Spark Action are generating new Json logs, but they are not giving any troubles to the History Server (stopped having tons of exceptions logged as I just said) Unfortunately, the Spark job itself is still failing and it's needing further investigation on my side, so apparently this is not related to that specific error message. But I've solved an annoying problem, and at the same time I have cleared out the possibility of the Spark Action issue being related to that java exception Thanks!
... View more
07-19-2017
08:58 AM
Hi mbigelow, I've tried what ypu suggested (stop hive + running the action from the dropdown menu). Process was successful, but the warning in spark CLI is still there...
... View more
07-19-2017
07:05 AM
Additional info. If I run spark CLI (where my spark procedures are working, btw, differently from when they are launched in Oozie), as soon as I try to define a Dataframe I receive the following warning that I've never seen before the upgrade: In [17]: utenti_DF = sqlContext.table("xxxx.yyyy")
17/07/19 15:48:58 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.1.0
17/07/19 15:48:58 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException Anyway, as I repeat, from CLI things work. I just thought this could be relevant
... View more
07-19-2017
06:47 AM
Hi all, after recently upgrading to CDH 5.11 I get tons of the following "Unexpected end-of-input" log entries related to "SPARK" (running on YARN) and classified as "ERRORS". I'm experiencing malfunctionings (failed Oozie Jobs) and I believe they are related to these errors, so I'd really like to solve the causing issue and see if the situation gets any better. In the logs, "source" is: FsHistoryProvider And "message" is: Exception encountered when attempting to load application log hdfs://xxxxx.xxxxx.zz:8020/user/spark/applicationHistory/application_1494352758818_0117_1
com.fasterxml.jackson.core.JsonParseException: Unexpected end-of-input: was expecting closing quote for a string value
at [Source: java.io.StringReader@1fec7fc4; line: 1, column: 3655]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1369)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:599)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:532)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString2(ReaderBasedJsonParser.java:1517)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString(ReaderBasedJsonParser.java:1505)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.getText(ReaderBasedJsonParser.java:205)
at org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:20)
at org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:42)
at org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:35)
at org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:28)
at org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:42)
at org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:35)
at org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:42)
at org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:35)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:2888)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2034)
at org.json4s.jackson.JsonMethods$class.parse(JsonMethods.scala:19)
at org.json4s.jackson.JsonMethods$.parse(JsonMethods.scala:44)
at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)
at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$replay(FsHistoryProvider.scala:583)
at org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$16.apply(FsHistoryProvider.scala:410)
at org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$16.apply(FsHistoryProvider.scala:407)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$mergeApplicationListing(FsHistoryProvider.scala:407)
at org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$checkForLogs$3$$anon$4.run(FsHistoryProvider.scala:309)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748) Any suggestions/ideas? Thanks!
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN
06-27-2017
06:42 AM
In the end I've been able to solve the issue. I've been tricked by the fact that applying again from scratch the "YARN Resources Allocation Tuning Guide" proposed a (in my opinion) misleading way of calculating a few important parameters. Guide can be found here: https://www.cloudera.com/documentation/enterprise/5-10-x/topics/cdh_ig_yarn_tuning.html In a matter of fact, the Guide contains a downloadable XLS file which is a tool for calculating optimal parameters. This XLS automatically calculates and proposes a few values to be assigned to YARN configuration: As you can see above, at step 4 I got proposed "2" for "yarn.nodemanager.resource.cpu-vcores" and "5632" for " yarn.nodemanager.resource.memory-mb" I later found out that the correct values to be assigned to those configurations are the 2 values proposed at "step 5" Definitely, partly my fault (I do not have deep knowledge of YARN configuration). But partly misleading doc indeed. I am now fine tuning, trying different settings for the various java heap sizes etc Still I have no idea why everything was working fine until recently and stopped working after upgrading to 5.11, as I did not change any configuration while upgrading and physical resources are identical
... View more
06-26-2017
12:37 PM
As I believe that the problem is definitely due to differences betweek CDH 5.7 and CDH 5.11 in how resources are allocated to containers by YARN, I've tried to follow again from scratch the YARN Tuning Guide. The latest version of the YARN Tuning Guide available is apparently for CDH 5.10: https://www.cloudera.com/documentation/enterprise/5-10-x/topics/cdh_ig_yarn_tuning.html In that page, an XLS Sheet is available to help out planning the various parameters in a correct and working fashion. No luck. I always find myself with jobs stuck in "ACCEPTED" mode and never starting to run. I also found this interesting thread suggesting how to configure Dynamic Resource Pools for YARN: https://www.cloudera.com/documentation/enterprise/5-10-x/topics/cm_mc_resource_pools.html#concept_xkk_l1d_wr__section_c3f_vwf_4n I tried to limit the "number of concurrent jobs" to just 2 in the relevant Configuration Page of the Dynamic Resource Pools, but again, no success. Can anybody please point me out whatever new feature that could have been implemented in CDH 5.11 and related to YARN Resources Allocation (and that I have not mentioned here), because my Workflows were running smoothly before the upgrade, and now I'm facing heavy troubles! Workarounds are welcome too, as well as methods for monitoring/tracing resources usage in a way allowing me to understand what parameters I've been set up in a way that is not functional anymore in CDH 5.11 Thanks a lot for any hints or insights!
... View more
06-21-2017
11:31 AM
Hello, after successfully upgrading a small (5 nodes) CDH 5.7 cluster to CDH 5.11, I am experiencing various problems on existing Oozie Workflows that used to work correctly. The most significant example: I have this Workflow scheduling 8 jobs in parallel (mix of Hive, Shell and Sqoop actions). The 8 jobs are acquired and start running. But the 8 sub-jobs performing the action are stuck in "ACCEPTED" status and never switch to "RUNNING" state. After hours of work, I've not been able to find anything significant in the logs, apart from a few complaining about log4j. So I decided to upgrade JDK from 1.7 to 1.8 too, but without any improvement in the situation. Any help or suggestion pointing me in the right direction in solving this would be very very much appreciated! Thanks
... View more
Labels:
02-06-2017
10:58 AM
Hi, thanks for your reply. I have tried to create the Table first, and indeed the overall behaviour changed. Now I get a different exception, which I'm going to paste just below. Strange thing is that the class referenced in the exception, "ClientBackoffPolicyFactory", IS loaded and present (it's in "--jars", as detailed in earlier posts above this one). Here is the main excerpt from the error stack: 2017-02-06 19:31:51,426 WARN [Thread-8] ipc.RpcControllerFactory (RpcControllerFactory.java:instantiate(78)) - Cannot load configured "hbase.rpc.controllerfactory.class" (org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory) from hbase-site.xml, falling back to use default RpcControllerFactory
2017-02-06 19:31:51,431 ERROR [Thread-8] datasources.InsertIntoHadoopFsRelation (Logging.scala:logError(95)) - Aborting job.
java.io.IOException: java.lang.reflect.InvocationTargetException
...
Caused by: java.lang.reflect.InvocationTargetException
...
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.client.backoff.ClientBackoffPolicyFactory$NoBackoffPolicy
...
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
...
Caused by: java.lang.reflect.InvocationTargetException
...
Caused by: java.lang.UnsupportedOperationException: Unable to find org.apache.hadoop.hbase.client.backoff.ClientBackoffPolicyFactory$NoBackoffPolicy
...
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.client.backoff.ClientBackoffPolicyFactory$NoBackoffPolicy
...
2017-02-06 19:31:51,538 ERROR [Driver] yarn.ApplicationMaster (Logging.scala:logError(74)) - User application exited with status 1 Given that "hbase-site.xml" is mentioned in the error stack, I'm also pasting that file just below: <?xml version="1.0" encoding="UTF-8"?>
<!--Autogenerated by Cloudera Manager-->
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://xxx01.yyy.it:8020/hbase</value>
</property>
<property>
<name>hbase.replication</name>
<value>true</value>
</property>
<property>
<name>hbase.client.write.buffer</name>
<value>2097152</value>
</property>
<property>
<name>hbase.client.pause</name>
<value>100</value>
</property>
<property>
<name>hbase.client.retries.number</name>
<value>35</value>
</property>
<property>
<name>hbase.client.scanner.caching</name>
<value>100</value>
</property>
<property>
<name>hbase.client.keyvalue.maxsize</name>
<value>10485760</value>
</property>
<property>
<name>hbase.ipc.client.allowsInterrupt</name>
<value>true</value>
</property>
<property>
<name>hbase.client.primaryCallTimeout.get</name>
<value>10</value>
</property>
<property>
<name>hbase.client.primaryCallTimeout.multiget</name>
<value>10</value>
</property>
<property>
<name>hbase.regionserver.thrift.http</name>
<value>false</value>
</property>
<property>
<name>hbase.thrift.support.proxyuser</name>
<value>false</value>
</property>
<property>
<name>hbase.rpc.timeout</name>
<value>60000</value>
</property>
<property>
<name>hbase.snapshot.enabled</name>
<value>true</value>
</property>
<property>
<name>hbase.snapshot.master.timeoutMillis</name>
<value>60000</value>
</property>
<property>
<name>hbase.snapshot.region.timeout</name>
<value>60000</value>
</property>
<property>
<name>hbase.snapshot.master.timeout.millis</name>
<value>60000</value>
</property>
<property>
<name>hbase.security.authentication</name>
<value>simple</value>
</property>
<property>
<name>hbase.rpc.protection</name>
<value>authentication</value>
</property>
<property>
<name>zookeeper.session.timeout</name>
<value>60000</value>
</property>
<property>
<name>zookeeper.znode.parent</name>
<value>/hbase</value>
</property>
<property>
<name>zookeeper.znode.rootserver</name>
<value>root-region-server</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>xxx02.yyy.it,xxx01.yyy.it,xxx03.yyy.it</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.rest.ssl.enabled</name>
<value>false</value>
</property>
</configuration> Thanks for any help... Meanwhile, I'll go on testing things, but maybe I'll try to find a workaround and do something completely different. This is starting to be a bit too much for my skills / patience 🙂
... View more
02-05-2017
04:02 PM
Update: Adding the following comma separated list of jars to the Spark Action's "Options List" I've been able to get past the Java exceptions: --jars hdfs:///user/xxx/ETL/SCRIPTS/SPARK/hive-hbase-handler-1.1.0-cdh5.7.0.jar,hdfs:///user/xxx/ETL/SCRIPTS/SPARK/hbase-server-1.2.0-cdh5.7.0.jar,hdfs:///user/xxx/ETL/SCRIPTS/SPARK/hbase-client-1.2.0-cdh5.7.0.jar Now I'm facing another problem. The workflow still fails, and in the Spark Action's log I see the following error: Traceback (most recent call last):
File "AnalisiVoipOut1.py", line 42, in <module>
... pyspark.sql.utils.AnalysisException: u'path hdfs://myhostname.mydomain.it:8020/user/hive/warehouse/xxx.db/analisi_voipout_utenti_credito_2016_rif already exists.;'
2017-02-05 22:46:51,405 ERROR [Driver] yarn.ApplicationMaster (Logging.scala:logError(74)) - User application exited with status 1 Now, I can confirm that even changing the Table's name every time (therefore: THAT PATH DOES'NT EXIST), I still get the error everytime I run the workflow. I've alsto thought that, maybe, given that I was using 4 executors, maybe fore some strange reason they got in confilct generating the output directory on hdfs. Therefore I also made several tries with just one executor, but with no success. This " HBaseStorageHandler" thing is really puzzling me, can anybody help me in getting past this last error? Thanks
... View more
02-05-2017
02:58 AM
Hello, I am on CDH 5.7 I have an HBase table. To be able to access it from Impala/Hive, I have created a Hive external table pointing to it, using HBaseStorageHandler, using the following approach: CREATE EXTERNAL TABLE hbase_utenti(key string, val1 string, ... , valX string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf1:a,cf1:b, ... ,cfX:X")
TBLPROPERTIES("hbase.table.name" = "utenti"); This WORKS smoothly, I can query the table from Impala/Hive Now I have the following PySpark script that I want to run via Spark Action (just an excerpt): ## IMPORT FUNCTIONS
import sys
import getopt
from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
from pyspark.sql import HiveContext
from pyspark.sql.functions import *
from pyspark.sql.types import *
## CREATE SQLCONTEXT
sconf = SparkConf().setAppName("ChurnVoip").setMaster("yarn-cluster").set("spark.driver.memory", "3g").set("spark.executor.memory", "1g").set("spark.executor.cores", 2)
sc = SparkContext(conf=sconf)
sqlContext = HiveContext(sc)
## CREATE MAIN DATAFRAME
hbase_utenti_DF = sqlContext.table("msgnet.hbase_utenti")
## SAVE HBASE TABLE AS "REAL" HIVE TABLE
hbase_utenti_filtered_DF = hbase_utenti_DF \
.select(hbase_utenti_DF["USERID"].alias("HHUTE_USERID"), \
hbase_utenti_DF["RIVENDITORE"].alias("HHUTE_RIVENDITORE"), \
hbase_utenti_DF["DATA_COLL"].alias("HHUTE_DATA_COLL"), \
hbase_utenti_DF["CREDITO"].alias("HHUTE_CREDITO"), \
hbase_utenti_DF["VIRTUAL"].alias("HHUTE_VIRTUAL"), \
hbase_utenti_DF["POST"].alias("HHUTE_POST")) \
.filter("((HHUTE_RIVENDITORE = 1) and \
(HHUTE_DATA_COLL='2016-12-30T00:00:00Z'))") \
.saveAsTable('msgnet.analisi_voipout_utenti_credito_2016_rif') Please note that if I run the above Python script via "spark-submit" OR if I run Pyspark shell and copy-paste the code into it, IT WORKS smoothly Problem arise when I try to schedule an Oozie Spark Action doing the job. In this case, I get the following exception: ...
2017-02-05 10:47:52,106 ERROR [Thread-8] hive.log (MetaStoreUtils.java:getDeserializer(387)) - error in initSerDe: java.lang.ClassNotFoundException Class org.apache.hadoop.hive.hbase.HBaseSerDe not found
java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.hbase.HBaseSerDe not found
...
hbase_utenti_DF = sqlContext.table("msgnet.hbase_utenti")
File "/hdp01/yarn/nm/usercache/msgnet/appcache/application_1486053545979_0053/container_1486053545979_0053_02_000001/pyspark.zip/pyspark/sql/context.py", line 593, in table
---
py4j.protocol.Py4JJavaError: An error occurred while calling o56.table.
: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.hadoop.hive.hbase.HBaseStorageHandler Now, I'm pretty sure the whole problem boils down to Jars not being included. But I've tried and tried to add Paths and Variables to my Oozie Spark Action, with no success. What I've tried to do includes the following: - Adding the following "Hadoop Properties" to my Workflow Settings (gear icon on top right of the Workflow in Hue): spark.executor.extraClassPath /opt/cloudera/parcels/CDH/lib/hive/lib/*
spark.executor.extraClassPath /opt/cloudera/parcels/CDH/lib/hbase/lib/* - Adding the following to my Spark Action's --files
hdfs:///user/<XXX>/hive-site.xml,
hdfs:///user/<XXX>/hive-hbase-handler-1.1.0-cdh5.7.0.jar,
hdfs:///user/<XXX>/hbase-client-1.2.0-cdh5.7.0.jar,
hdfs:///user/<XXX>/hbase-server-1.2.0-cdh5.7.0.jar,
hdfs:///user/<XXX>/hbase-hadoop-compat-1.2.0-cdh5.7.0.jar,
hdfs:///user/<XXX>/hbase-hadoop2-compat-1.2.0-cdh5.7.0.jar,
hdfs:///user/<XXX>/metrics-core-2.2.0.jar
--driver-class-path /etc/hbase/conf:/opt/cloudera/parcels/CDH/lib/hbase/lib/* Can anybody please shed some light on what I'm missing here? I'm really out of ideas, after trying and trying. Thanks for any insight!
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache Oozie
-
Apache Spark
01-03-2017
12:38 PM
Hi AdrianMonter, sorry to say I haven't found a specific solution for the Avro file format in the meanwhile. I'm sticking to Parquet file format since I had this problem, and for now it covers all my needs... Maybe in the latest CDH/Spark releases this has been fixed? Maybe somebody from @Former Member can tell us someting more?
... View more
10-17-2016
01:40 PM
2 Kudos
Hi aj, yes I did manage to solve it. Please, take a look at the following thread and see if it can be of help. It may seem a bit unrelated from the "test.py not found" issue, but it contains detailed info about how to specify all the needed parameters to let the whole thing run smoothly: http://community.cloudera.com/t5/Batch-Processing-and-Workflow/Oozie-workflow-Spark-action-using-simple-Dataframe-quot-Table/m-p/40834 HTH
... View more
08-15-2016
05:15 AM
Thank you. Useful insight and crystal clear argumentation, as usual from you. I have to say in the meanwhile I had the chance to study a bit more, and in the end I came to a conclusion which matches your considerations, therefore I'm glad that apparently I moved in the right direction. In a matter of fact I've seen this Open Source project here http://opentsdb.net , and I've seen that generally speaking the approach they use is the last that you explained. To provide a practical example, in my case: - A new record every week for the same Customer Entity - Therefore, column Versioning is NOT used at all! (like you suggested) - "Speaking" record key e.g. "<CUST_ID> + <YYYY-MM-DD>" - This sort of Key is not monotonically increasing, because the "CUST_ID" part is "stable", so this approach should be good also on a "Table Splitting" perspective (when the Table grows, it will split up "evenly" and all the Splits will take care of a part of the future inserts, balancing the Machines Load evenly) - Same set of columns for each record containing the new sampled value for that field for that week e.g. "<Total progressive time used Service X>" This is the approach I used in the end, which has nothing to do with my original idea of using Versions but perfectly matching the last approach you described in your answer. Regarding the Fixed Values (e.g. "Name", "Surname") I've decided to replicate them every week too, as if they were Time Series themselves... I know, waste of storage. Planning on modifying this structure soon and move the Fixed Values in another Table (Hive or HBase, don't know yet) and pick up the information I'd eventually need at the moment (for instance, during Data Processing, I'll join in the relevant Anagraphic Data in the relevant Dataframes via Join). I just wanted to write a few more lines about the issue for the posterity. I hope this post will be useful to people 🙂 Thanks again!
... View more
07-31-2016
08:06 AM
Hi all, I have the following design question for my new table in HBase. Scenario: ------------- - Table will contain Customers Information - Table would be refreshed every week by a procedure, inserting new info (see below) - Row Key would be "Customer ID" (fixed) - There would be fixed contents columns, e.g. "Name", "Surname" - There would be variable contents columns, e.g. "Credit", "No. of Services subscribed", "Total Time used Service X" The question: ------------------ - Should I take advantage of Column Versioning, e.g. every week putting in a new version for Column (e.g) "Total Time used Service X" ? So that the Table would have a fixed number of Columns, some of them with versions and others fixed? - Or is it a better approach to NOT use Column Versioning, and for every new week of Data coming in just add a new Column named (e.g.) "Total Time used Service X - WEEK YY" ? In this case I'd put in the Week Number in the Column Name to be able to look up for it in later analysis Please keep in mind that: ---------------------------------- - The main use will be to process the "Variable Information" columns later using a Spark procedure, therefore it is of CRITICAL IMPORTANCE the ability to process each and every "Time Series" easily, on the fly, without convoluted workarounds to manage e.g. Column Name and then loop through Columns in weird ways (this is why at the moment I'm thinking the "Column Versioning" solution would be the best one, but my knowledge of HBase is just basic and I'd like to hear other voices too before making a mistake) - I'm proposing that the Row Key would be FIXED, but I'm open to other suggestions (e.g. multiple rows with variable Key for the same Customer Entity) if this would be the best approach in the described scenario. I just didn't want to mess up things too much describing my problem Any insight and/or link to examples for the particular case will be very much appreciated! Thanks
... View more
Labels:
- Labels:
-
Apache HBase
07-28-2016
03:11 AM
Thanks. Seems a good alternative, and in a matter of fact I was not aware of its availability in CDH 5.7 Marking the thread as solved, even if by now I don't know yet if all the features I'd need will be there in the native hbase-spark connector
... View more
05-29-2016
11:39 AM
Hi all, I wanted to experiment with the "it.nerdammer.bigdata:spark-hbase-connector_2.10:1.0.3" Package (you can find it at spark-packages.org ). It's an interesting addon giving RDD visibility/operativity on hBase tables via Spark. If I run this extension library in a standard spark-shell (with scala support), everything works smoothly : spark-shell --packages it.nerdammer.bigdata:spark-hbase-connector_2.10:1.0.3 \
--conf spark.hbase.host=<HBASE_HOST>
scala> import it.nerdammer.spark.hbase._
import it.nerdammer.spark.hbase._ If I try to run it in a Pyspark shell, therefore my goal is to use the extension with Python, I'm not able to import the Functions and I'm not able to use anything: PYSPARK_DRIVER_PYTHON=ipython pyspark --packages it.nerdammer.bigdata:spark-hbase-connector_2.10:1.0.3 \
--conf spark.hbase.host=<HBASE_HOST>
In [1]: from it.nerdammer.spark.hbase import *
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-1-37dd5a5ffba0> in <module>()
----> 1 from it.nerdammer.spark.hbase import *
ImportError: No module named it.nerdammer.spark.hbase I have tried different combinations of environment variables, parameters, etc when launching Pyspark, but to no avail. Maybe I'm just trying to do something deeply wrong here, maybe it's simply the fact that there is no Python API access to this Library. In a matter of fact, the examples on the Package's home page are all in Scala (but they say you can install the Package in Pyspark too, with the classic "--package" parameter). Can anybody help out with the "ImportError: No module named it.nerdammer.spark.hbase" error message? Thanks for any insight
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache Spark
05-26-2016
04:05 AM
4 Kudos
Update: I got to a working solution, this is a brief Howto to get to the result: JOB MAIN BOX CONFIGURATION (CLICK THE "PENCIL" EDIT ICON ON TOP OF THE WORKFLOW MAIN SCREEN): Spark Master: yarn-cluster
Mode: cluster
App Name: MySpark
Jars/py files: hdfs:///user/hue/oozie/workspaces/hue-oozie-1463575878.15/lib/test.py
Main Class: <WHATEVER_STRING_HERE> (E.g. "clear", or "org.apache.spark.examples.mllib.JavaALS"). We do not have a Main Class in our ".py" script!
Arguments: NO ARGUMENTS DEFINED WORKFLOW SETTINGS (CLICK GEAR ICON ON TOP RIGHT OF THE WORKFLOW MAIN SCREEN): Variables: oozie.use.system.libpath --> true
Workspace: hue-oozie-1463575878.15
Hadoop Properties: oozie.launcher.yarn.app.mapreduce.am.env --> SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
Show Graph Arrows: CHECKED
Version: uri.oozie.workflow.0.5
Job XML: EMPTY
SLA Configuration: UNCHECKED JOB DETAILED CONFIGURATION (CLICK THE "PENCIL" EDIT ICON ON TOP OF THE WORKFLOW MAIN SCREEN AND THE THE TRIANGULAR ICON ON TOP RIGHT OF THE MAIN JOB BOX TO EDIT IT IN DETAIL): - PROPERTIES TAB:
-----------------
Options List: --files hdfs:///user/hue/oozie/workspaces/hue-oozie-1463575878.15/hive-site.xml
Prepare: NO PREPARE STEPS DEFINED
Job XML: EMPTY
Properties: NO PROPERTIES DEFINED
Retry: NO RETRY OPTIONS DEFINED
- SLA TAB:
----------
Enabled: UNCHECKED
- CREDENTIALS TAB:
------------------
Credentials: NO CREDENTIALS DEFINED
- TRANSITIONS TAB:
------------------
Ok End
Ko Kill MANUALLY EDIT MINIMAL "hive-site.xml" FILE TO BE PASSED TO THE SPARK-ON-HIVE CONTAINER TO BE ABLE TO ACCESS THE TABLES METASTORE FROM WHATEVER NODE IN THE CLUSTER, AND UPLOAD IT TO HDFS: vi hive-site.xml
---
<configuration>
<property>
<name>hive.metastore.uris</name>
<value>thrift://<THRIFT_HOSTNAME>:9083</value>
</property>
</configuration>
---
hdfs dfs -put hive-site.xml /user/hue/oozie/workspaces/hue-oozie-1463575878.15 EDIT THE PYSPARK SCRIPT AND UPLOAD IT INTO THE "lib" DIRECTORY IN THE WORKFLOW FOLDER: vi test.py
---
from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
from pyspark.sql import HiveContext
from pyspark.sql.functions import *
sconf = SparkConf().setAppName("MySpark").set("spark.driver.memory", "1g").setMaster("yarn-cluster")
sc = SparkContext(conf=sconf)
sqlCtx = HiveContext(sc)
xxx_DF = sqlCtx.table("table")
yyy_DF = xxx_DF.select("fieldname").saveAsTable("new_table")
---
hdfs dfs -put test.py /user/hue/oozie/workspaces/hue-oozie-1463575878.15/lib NOW YOU CAN SUBLIT THE WORKFLOW IN YARN: - Click the "PLAY" Submit Icon on top of the screen ADDITIONAL INFO: AUTO-GENERATED "workflow.xml": <workflow-app name="Spark_on_Oozie" xmlns="uri:oozie:workflow:0.5">
<global>
<configuration>
<property>
<name>oozie.launcher.yarn.app.mapreduce.am.env</name>
<value>SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark</value>
</property>
</configuration>
</global>
<start to="spark-9fa1"/>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="spark-9fa1">
<spark xmlns="uri:oozie:spark-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<master>yarn-cluster</master>
<mode>cluster</mode>
<name>MySpark</name>
<class>clear</class>
<jar>hdfs:///user/hue/oozie/workspaces/hue-oozie-1463575878.15/lib/test.py</jar>
<spark-opts>--files hdfs:///user/hue/oozie/workspaces/hue-oozie-1463575878.15/hive-site.xml</spark-opts>
</spark>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app> ADDITIONAL INFO: AUTO-GENERATED "job.properties": oozie.use.system.libpath=true
security_enabled=False
dryrun=False
jobTracker=<JOBTRACKER_HOSTNAME>:8032
... View more
05-17-2016
02:15 PM
24 Kudos
Hi, please perform the following actions: 1) Fill in "Source Brokers List" --> "nameOfClouderaKafkaBrokerServer.yourdomain.com:9092". This is the Server (or Servers) where you configured the Kafka Broker (NOT the MirrorMaker). 2) Fill in "Destination Brokers List" --> "nameOfRemoteBrokerServer.otherdomain.com:9092". This is supposed to be a remote Cluster that will receive Topics sent over by your MirrorMaker. If you have one, put in that one. Otherwise just put in another Server in your network, whatever Server. Please note that both this Server Names must be FQDN and resolvable by your DNS (or hosts file), otherwise you'll get other errors. Also the format with the trailing Port Number is mandatory! 3) Click "Continue". Service will NOT start (error). Do not navigate away from that screen 4) Open another Cloudera Manager in another browser pane. You should now see "Kafka" in the list of Services (red, but it should be there). Click on the Kafka Service and then "Configure". 5) Search for the "java heap space" Configuration Property. The standard Java Heap Space you'll find already set up should be 50 MBytes. Put in at least 256 MBytes. The original value is simply not enough. 6) Now search for the "whitelist" Configuration Property. In the field, put in "(?!x)x" (without the quotation marks). That's a regular expression that does not match anything. Given that apparently a Whitelist is mandatory for the Mirrormaker Service to start, and I'm assuming you don't want to replicate any topics remotely right now, just put in something that won't replicate anything e.g. that regular expression. 7) Save the changes and go back to the original Configuration Screen on the othe browser pane. Click "Retry", or wathever, or even exit that screen and manually restart the Kafka Service in Cloudera Manager. That should work, at least it did for me! HTH
... View more
05-14-2016
03:39 PM
Update: If I use "spark-submit", the script runs successfully. Syntax used for "spark-submit": spark-submit \
--master yarn-cluster \
--deploy-mode cluster \
--executor-memory 500M \
--total-executor-cores 1 \
hdfs:///user/hue/oozie/workspaces/hue-oozie-1460736691.98/lib/test.py \
10 Excerpt from output log: 16/05/15 00:30:57 INFO parse.ParseDriver: Parsing command: select * from sales_fact
16/05/15 00:30:58 INFO parse.ParseDriver: Parse Completed
16/05/15 00:30:58 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0-cdh5.5.1
16/05/15 00:30:58 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0-cdh5.5.1
16/05/15 00:30:59 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
16/05/15 00:30:59 INFO spark.SparkContext: Invoking stop() from shutdown hook
... View more
05-14-2016
08:53 AM
Hi all, my CDH test rig is as follows: CDH 5.5.1 Spark 1.5.0 Oozie 4.1.0 I have successfully created a simple Oozie Workflow that spawns a Spark Action using HUE Interface. My intention is to use Yarn in Cluster mode to run the Workflow/Action. It's a Python script, which is as follows (just a test): from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
from pyspark.sql import HiveContext
from pyspark.sql.functions import *
sconf = SparkConf().setAppName("MySpark").set("spark.driver.memory", "1g").setMaster("yarn-cluster")
sc = SparkContext(conf=sconf)
### (1) ALTERNATIVELY USE ONE OF THE FOLLOWING CONTEXT DEFINITIONS:
sqlCtx = SQLContext(sc)
#sqlCtx = HiveContext(sc)
### (2) IF HIVECONTEXT, EVENTUALLY SET THE DATABASE IN USE (SHOULDN'T BE NECESSARY):
#sqlCtx .sql("use default")
### (3) CREATE MAIN DATAFRAME. TRY THE SYNTAXES ALTERNATIVELY, COMBINE WITH DIFFERENT (1):
#cronologico_DF = sqlCtx.table("sales_fact")
cronologico_DF = sqlCtx.sql("select * from sales_fact")
### (4) ANOTHER DATAFRAME
extraction_cronologico_DF = cronologico_DF.select("PRODUCT_KEY")
### (5) USELESS PRINT STATEMENT:
print 'a' When I run the Workflow, a Mapreduce Job is started. Shortly after, a Spark Job is spawned (I can see that from the Job Browser). The Spark Job fails with the following error (excerpt from the Log File of the Spark Acrion): py4j.protocol.Py4JJavaError: An error occurred while calling o51.sql.
: java.lang.RuntimeException: Table Not Found: sales_fact This is my "workflow.xml": <workflow-app name="Churn_2015" xmlns="uri:oozie:workflow:0.5">
<global>
<job-xml>hdfs:///user/hue/oozie/workspaces/hue-oozie-1460736691.98/hive-site.xml</job-xml>
<configuration>
<property>
<name>oozie.launcher.yarn.app.mapreduce.am.env</name>
<value>SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark</value>
</property>
</configuration>
</global>
<start to="spark-3ca0"/>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="spark-3ca0">
<spark xmlns="uri:oozie:spark-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<job-xml>hdfs:///user/hue/oozie/workspaces/hue-oozie-1460736691.98/hive-site.xml</job-xml>
<configuration>
<property>
<name>oozie.use.system.libpath</name>
<value>true</value>
</property>
</configuration>
<master>yarn-cluster</master>
<mode>cluster</mode>
<name>MySpark</name>
<class>org.apache.spark.examples.mllib.JavaALS</class>
<jar>hdfs:///user/hue/oozie/workspaces/hue-oozie-1460736691.98/lib/test.py</jar>
</spark>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app> This is my "job.properties": oozie.use.system.libpath=True
security_enabled=False
dryrun=False
jobTracker=<MY_SERVER_FQDN_HERE>:8032
nameNode=hdfs://<MY_SERVER_FQDN_HERE>:8020 Please note that: 1) I've also uploaded "hive-site.xml" in the same directory as the 2 files described above. As you can see from "workflow.xml", it should also be picked up. 2) The "test.py" script is under a "lib" directory in the Workspace created by HUE. It gets picked up. In that directory I also took care of uploading several Jars belonging to some Derby DB Connector, probably required to collect Stats, to avoid other exceptions being throwed. 3) I've tried adding a workflow property "oozie.action.sharelib.for.spark", with value "hcatalog,hive,hive2", with no success 4) As you can see in the Python Script described above, I've been using alternatively an SQLContext or a HiveContext object inside the Script. Results are the same (the error message is slightly different though). 5) ShareLib should be OK too: oozie admin -shareliblist
[Available ShareLib]
oozie
hive
distcp
hcatalog
sqoop
mapreduce-streaming
spark
hive2
pig I'm suspecting the Tables Metastore is not being read, that's probably the issue. But I ran out of ideas and I'm not able to get it working... Thanks in advance for any feedback!
... View more
Labels:
- Labels:
-
Apache Oozie
05-13-2016
03:03 AM
2 Kudos
I got past this! Still no cigar, though. Now I have another error, but I'm going to work on this. It's something different now... Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, File file:/hdp01/yarn/nm/usercache/admin/appcache/application_1463068686660_0013/container_1463068686660_0013_01_000001/lib/test.py does not exist
java.io.FileNotFoundException: File file:/hdp01/yarn/nm/usercache/admin/appcache/application_1463068686660_0013/container_1463068686660_0013_01_000001/lib/test.py does not exist Many thanks for your help. I'd never be able to figure this out by myself!
... View more
