Member since
11-18-2014
196
Posts
17
Kudos Received
8
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
5601 | 03-16-2016 05:54 AM | |
2274 | 02-05-2016 04:49 AM | |
1495 | 01-08-2016 06:55 AM | |
12308 | 09-29-2015 01:31 AM | |
864 | 05-06-2015 01:50 AM |
01-12-2015
01:05 AM
Note: - I tried the command sudo usermod -G developer Peter on all instances of the cluster => no result - I tried to do the same thing on a quickstart VM and I managed to make it work by only using the interface. I still not manage to make it work on the 7 machine cluster. - When we installed the cluster, we had to create a user in cloudera manager and then create them again in cloudera director. is this normal? can this be linked to my problem?
... View more
01-12-2015
12:29 AM
Thank you david for this complete answer! However, I'm facing some problems... On my namenode, when I do sudo usermod -G developer Peter I get user 'Peter' does not exist. (Note, I created this user with the Hue interface) When I do hdfs groups Peter I get no group. To be precise I get Peter: I checked that the property hadoop.security.group.mapping is org.apache.hadoop.security.ShellBasedUnixGroupsMa pping Note: I created the user, the group and associated the group to the user in Hue interface. And these users, and groups do not seem to be visible on the OS (CentOS).
... View more
01-11-2015
11:04 PM
I saw that wile importing a workflow we can share it, but can we share a workflow that was allready created? or while creating a new workflow?
... View more
01-09-2015
08:40 AM
Hello, I created a workflow with my user, how can I share it with the other users? Thank you!
... View more
01-09-2015
08:02 AM
Hello, I want to do a folder that can be accessed in r, w,x by all the users of the group 'developer' that I created. I also want that all the folders/files created by these users in this folder can be accessed in r,w,x by the users from this group. All the other users should access the folder in r and x. What should I do? I tried: - to create my folder with these rights: drwxrwxr-t+ - the folder that I created is owned by admin who is part of the developer group. All I think is missing is to add the default group of my users the developer group : do something like hadoop def -usermod -G developer Peter But this is not working! Thank you!
... View more
01-08-2015
09:41 AM
I forgot to say, with this workarout in hue query editor the join works fine, but in oozie I get the folowing error: 276 [main] INFO org.apache.hadoop.hive.ql.Driver - Total jobs = 1
6278 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - </PERFLOG method=TimeToSubmit start=1420735265793 end=1420735267866 duration=2073 from=org.apache.hadoop.hive.ql.Driver>
6278 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver>
6278 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=task.MAPREDLOCAL.Stage-5 from=org.apache.hadoop.hive.ql.Driver>
6291 [main] INFO org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask - Generating plan file file:/yarn/nm/usercache/agherman/appcache/application_1420703635560_0322/container_1420703635560_0322_01_000002/tmp/yarn/hive_2015-01-08_16-41-05_793_3655704759550834241-1/-local-10005/plan.xml
6291 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=serializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
6291 [main] INFO org.apache.hadoop.hive.ql.exec.Utilities - Serializing MapredLocalWork via kryo
6465 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - </PERFLOG method=serializePlan start=1420735267879 end=1420735268053 duration=174 from=org.apache.hadoop.hive.ql.exec.Utilities>
6570 [main] WARN org.apache.hadoop.hive.conf.HiveConf - DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
6646 [main] INFO org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask - Executing: /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/hadoop/bin/hadoop jar /yarn/nm/filecache/94/hive-exec-0.13.1-cdh5.3.0.jar org.apache.hadoop.hive.ql.exec.mr.ExecDriver -localtask -plan file:/yarn/nm/usercache/agherman/appcache/application_1420703635560_0322/container_1420703635560_0322_01_000002/tmp/yarn/hive_2015-01-08_16-41-05_793_3655704759550834241-1/-local-10005/plan.xml -jobconffile file:/yarn/nm/usercache/agherman/appcache/application_1420703635560_0322/container_1420703635560_0322_01_000002/tmp/yarn/hive_2015-01-08_16-41-05_793_3655704759550834241-1/-local-10006/jobconf.xml
10922 [main] ERROR org.apache.hadoop.hive.ql.exec.Task - Execution failed with exit status: 2
10922 [main] ERROR org.apache.hadoop.hive.ql.exec.Task - Obtaining error information
10922 [main] ERROR org.apache.hadoop.hive.ql.exec.Task -
Task failed!
Task ID:
Stage-5
Logs:
10922 [main] ERROR org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask - Execution failed with exit status: 2
10932 [main] ERROR org.apache.hadoop.hive.ql.Driver - FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
10932 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - </PERFLOG method=Driver.execute start=1420735267864 end=1420735272520 duration=4656 from=org.apache.hadoop.hive.ql.Driver>
10932 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
10955 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - </PERFLOG method=releaseLocks start=1420735272520 end=1420735272543 duration=23 from=org.apache.hadoop.hive.ql.Driver>
10977 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
10977 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - </PERFLOG method=releaseLocks start=1420735272565 end=1420735272565 duration=0 from=org.apache.hadoop.hive.ql.Driver>
<<< Invocation of Hive command completed <<<
Hadoop Job IDs executed by Hive:
Intercepting System.exit(2)
<<< Invocation of Main class completed <<<
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [2]
Oozie Launcher failed, finishing Hadoop job gracefully
Oozie Launcher, uploading action data to HDFS sequence file: xxxxxxxxxxxxxxxxxxxxxxxxx
Oozie Launcher ends
Thank you!
... View more
01-08-2015
09:38 AM
Hello, Yes ofcourse, DDL for the first table: CREATE TABLE IF NOT EXISTS TableA(
a boolean,
b struct <
c: struct <
d: boolean,
e: string,
f: string,
g: struct <
h: string,
i: string,
j: string,
k: binary>,
l: string,
m: struct <
n: struct <
o: smallint,
p: smallint>,
r: struct <
s: tinyint,
t: smallint,
u: tinyint,
v: smallint>,
z`: struct <
aa: smallint,
bb: smallint>>,
cc: string>,
dd: boolean,
ee: string,
ff: string,
gg: array <struct <
name: string,
value: string>>,
hh: string,
ii: string,
jj: struct <
kk: string,
ll: string>,
mm: string>
)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
STORED AS TEXTFILE
TBLPROPERTIES ("immutable"="true") ; And the DDL for the second table: CREATE TABLE IF NOT EXISTS TableB (
a array <struct <
b: boolean,
c: string,
d: string,
e: array <string>,
f: string,
g: string,
h: string,
i: boolean,
j: string,
k: array <string>,
l: string,
m: string,
n: string,
o: string,
p: string,
q: string,
r: string,
s: boolean,
t: boolean,
y: boolean,
z: boolean,
w: string,
aa: string,
bb: int,
cc: string,
dd: boolean,
ee: string,
ff: string,
gg: string,
hh: boolean,
ii: boolean,
jj: boolean,
kk: boolean,
ll: string,
mm: string,
nn: boolean,
oo: boolean,
pp: array <int>,
qq: string,
tt: string,
rr: string,
zz: string,
aaa: string,
bbb: boolean,
ccc: int,
ddd: string,
eee: boolean,
fff: int>>
)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES ( "ignore.malformed.json" = "true")
STORED AS TEXTFILE; I putted in bold and I underlined the fields on wich I'm doing the join. Note: I managed to find a workaround that is working in hue but is not woking in oozie) by doing the folowing: - altered the first table in order to add the kk column at the same level to the a column. - I changed the config: hive.auto.convert.join=false Thank you!
... View more
01-08-2015
09:04 AM
I'm using CDH5.3 on a 7 nodes cluster..
... View more
01-08-2015
04:00 AM
Ok, In fact the error comes from the fact that I'm using json tables: Caused by: java.lang.ClassCastException: org.apache.hadoop.io.ByteWritable cannot be cast to org.apache.hadoop.hive.serde2.io.ByteWritable
at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableByteObjectInspector.get(WritableByteObjectInspector.java:36)
at org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:198)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:490)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:550)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:550)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:550)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:550)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:443)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:427)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:582)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796) The join is working with other sample tables. How can I fix this? Why is this cast happening?
... View more
01-08-2015
02:44 AM
Note: I also tried: SELECT
*
FROM aLEFT OUTER JOIN b
ON (true)
WHERE (b.event.user.userid is not null) AND (b.event.user.userid LIKE a.account.userid); (Still not working)
... View more
01-08-2015
02:14 AM
I will also mention that this also not working: SELECT
*
FROM a LEFT OUTER JOIN b
ON (a.k10 = b.k10) And that this is working SELECT
*
FROM a WHERE a.k10 IN
(SELECT b.k10 FROM b)
... View more
01-08-2015
02:11 AM
I didn't change anything since yesterday to this query and now I'm having another error...but I's not saying a lot more... org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:147)
at org.apache.hive.service.cli.operation.SQLOperation.access$000(SQLOperation.java:69)
at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:502)
at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:213)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
... View more
01-08-2015
01:49 AM
Hello, I'm having some ETL queries that I put in oozie workflow. However, from time to time (30% of cases) I get a Java Heap Memory. In the rest of the cases I have a sucessful result. I never run multiple workflows in the same time so there should be exactly the same process runing. Where should I look in order to solve this? I tried to increase the java heap for clients/ for metastore. Still the same thing. Thank you!!
... View more
01-08-2015
12:08 AM
1 Kudo
Hello, I'm having a 7 instances cluster and I'm using CHD5. My problem is that I cannot manage to start the hive gateways. When I do start to hive, the gateways don't start. If I want to start them afterwards they are saying to stop hiveserver and hivemetastore. ..so I'm in a no-ending loop. Hive Metastore Server 1 Good Health HiveServer2 2 Good Health Gateway 6 None What should I do in order to start the gateways? Thank you!
... View more
01-07-2015
06:54 AM
Hello, I'm having two tables and I want to insert the joined data into another table. However I'm getting a strange error: 5703 [main] ERROR org.apache.hadoop.hive.ql.Driver - FAILED: SemanticException [Error 10044]: Line 1:18 Cannot insert into target table because column number/types are different 'ssid_part': Cannot convert column 1 from struct<...A...> to struct<.... same structure A ...>.
My query : SELECT
a.k,
a.k1,
a.k2,
b.k3
FROM a LEFT OUTER JOIN b
ON (a.k10 = b.k10) What I am doing wrong? Where should I search for more details on the error? Thank you!
... View more
12-31-2014
12:58 AM
I found my error: - I included the hdfs-site.xml file (and not hive-site.xml) - the name of the file has to be changed (if not I have permission denied) Thank you!
... View more
12-30-2014
07:09 AM
Hello, I did a simple script for creating a table, and it works fine in the Hue browser. CREATE DATABASE IF NOT EXISTS test;
create table test.agh_test_table_creation(
id int,
dtDontQuery string,
name string
) When I schedule the script with oozie I get successful, but the table is not visible in the hue browser.However, the table is present in the folder: user/hive/warehouse Note: My oozie workflow contains the hdfs-site.xml location: <workflow-app name="AGH_TEST_default_workspace_create_table" xmlns="uri:oozie:workflow:0.4">
<start to="AGH_test_default_create_table_test"/>
<action name="AGH_test_default_create_table_test">
<hive xmlns="uri:oozie:hive-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<job-xml>hdfs-site.xml</job-xml>
<script>tes_table_creation_oozie</script>
<file>hdfs-site.xml#hdfs-site.xml</file>
</hive>
<ok to="end"/>
<error to="kill"/>
</action>
<kill name="kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app> I got the hdfs-site.xml by downloading it from Cloudera Manager-> Clusters-> Hive-> Actions-> Download Client configuration Also, the source of the issue is not reling in the derby metastore since the metastore database is not the default one. I'm mentioning this, because of the stackoverflow post: http://stackoverflow.com/questions/23615309/oozie-with-hive-issue What should I do in order to make the table visible in the hue interface? Thank you !
... View more
12-29-2014
01:27 AM
I'm implementing a small hadoop cluster for a POC in my company. I'm trying to import files into HDFS with Flume. Each files contains JSON objects like this (1 "long" line per file): { "objectType" : [ { JSON Object } , { JSON Object }, ... ] } "objectType" is the type the objects in the array (ex: events, users, ...). These files will be processed later by several tasks depending on the "objectType". I'm using the spoolDir source and the HDFS sink. My questions are: Is it possible to keep the source filename when flume write into HDFS (filenames are unique as they contains a timestamp and a UUID in their name) Is there a way to set "deserializer.maxLineLength" to an unlimited value (instead of setting a high value)? I really dn't want to loose data. Which channel is the best, JDBC or File? (I do not have a flow with high throughput) My constraint is that I have to use flume out-of-the-box (no custom elements) as much as possible. Thanks for your help!
... View more
12-23-2014
09:17 AM
2 Kudos
Hello, I'm having an HADOOP_HOME is not set error: DEBUG Shell: Failed to detect a valid hadoop home directory
java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:302)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:327)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79)
at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:104)
at org.apache.hadoop.security.Groups.<init>(Groups.java:86)
at org.apache.hadoop.security.Groups.<init>(Groups.java:66)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:280)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:269)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:246)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:775)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2753)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2745)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2611)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169) However, I know that HADOOP_HOME is depreciated. Also, there is no mapReduce installed on my cluster (only map reduce 2 -yarn is installed). Therefore I thought I should have an HADOOP_PREFIX error. (I do not understand) The error is triggered at the second line of the folowing code. Configuration conf = new Configuration();
FileSystem hdfs = FileSystem.get(conf);
Path pathInputEvent= new Path(FOLDER);
Path localFilePath = new Path(LOCAL_FOLDER);
if(hdfs.exists(pathInputEvent)){
hdfs.delete(pathInputEvent,false);
hdfs.copyFromLocalFile(localFilePath, pathInputEvent);
} I tried to set HADOOP_HOME in eclipse by setting the value /opt/cloudera/parcels/CDH/lib/hadoop to the HADOOP_HOME variable in run configurations->environement Note: I got the value from the forum http://community.cloudera.com/t5/Cloudera-Manager-Installation/Where-is-HADOOP-HOME-lib-on-CDH-5-0-1-Parcels/td-p/13748 This it didn't worked :I had an error that said that hadoop home directory is an absolute path or it doesn't exist - an IOException. Why is the error asking for HADOOP_HOME and not HADOOP_PREFIX? And what value should I set for which variable? Thank you!
... View more
12-18-2014
07:13 AM
Hello, We wanted to deploy a cluster with Cloudera Director with 2 master nodes and 4 slaves nodes. However, we are having some problems: - we cannot add more than 1 master in cloudera director - we cannot make the cluster without a gateway node - we cannot choose a custom service package (we wanted a package with Impala+HBase but without spark/solr) - we cannot change the admin password/make another user Please, can you indicate a documentation that can help us with this ? Is there any feature list documentation for cloudera director? What are the benefits of using Cloudera Director insteaed of Cloduera Manager for deployment? Thank you!
... View more
12-18-2014
01:36 AM
@ jkestelyn Can you please also offer a Cloudera best practice answer?
... View more
12-16-2014
11:42 PM
Thank you very much Orrin for this information!
... View more
12-16-2014
05:39 AM
From the first comment I understood that we should'nt try to use the web interface but I still have 2 choices with cloudera: - use eclipse/maven (like in this example https://github.com/cloudera/cdh-twitter-example) - use scripts (hql scripts/shell ....) What are the best practice Cloudera for developpment?
... View more
12-12-2014
09:51 PM
Hello, My use case and the choosed services are: 1) taking some data from different productors/sources * flume => configuration scripts => custom source (S3) => java No interface so we are constraint to make scripts (?) 2) applying different ETL on the data 2.1) standardize the data (different treatment/ source) 2.2) enrich the data with the help of other sources 2.3) transform data (json) into parquet This part is considered as batch today (there is not a source that is transfering all the data in real time) We are doing the transformation to parquet in the end because, until then, we may need to return the data to other applications in json. *Hive =>? 1? => use the graphical interface? if yes, how should we work as a team on it? naming conventions?) visualisation ok/ debugging ok 2? => use eclipse/maven (like in this example https://github.com/cloudera/cdh-twitter-example) => sping on Hadoop? => versionning ok /debugging ok => the architecture of the project can be seen in the project architecture + Oozie 3? => make .hql scripts since most of the cloudera examples are showing only the HiveQL part of Hive. => debugging ? /the global vision of the project is offered by ozzie 3) render the data to Qlickview or other applications. - make KPI calculations - return the data in tables *Impala * Oozie => ordonnancing all this. Thank you!!
... View more
12-12-2014
02:26 AM
1 Kudo
Hello, There is a thing I cannot mange to understand: - should we use the graphicsl interface (hue) for developping? How are we going to version this thing? - should we use maven and the jdbc connector for developping? - should we do shell scripts for developping? Are there any cloudera best practice about all this? Thank you!
... View more
12-04-2014
05:53 AM
And my pom is containg the folowing components: <dependency><!-- hbase -->
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>${hbase.version}</version>
<scope>provided</scope>
</dependency>
<dependency><!-- hbase -->
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-common</artifactId>
<version>${hbase.version}</version>
<scope>provided</scope>
</dependency>
<dependency><!-- hbase -->
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>${hbase.version}</version>
<scope>provided</scope>
</dependency>
<dependency><!-- zookeeper -->
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>${zookeeper.version}</version>
<scope>provided</scope>
</dependency>
... View more
12-04-2014
04:35 AM
I forgot to say that I also tried to add: <property>
<name>hbase.zookeeper.property.dataDir</name>
<value>usr/local/zookeeper</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property> in my hbase-env.sh script (before that, in this script there was nothing concerning zookeeper) Thank you!
... View more
12-04-2014
04:06 AM
I'm having a simple spark job (done in java) that I'm executing on a cloudera quickstart plateform (learning phase) and I wanted to create a hbase table. However hbase/zookeeper is not found: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/ZooKeeperConnectionException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:319)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ZooKeeperConnectionException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358) I tried to set the HBASE_CLASSPATH from the /usr/lib/hbase/conf/hbase-env.sh to /usr/lib/hbase/ but it didn't worked. My code: Configuration conf = HBaseConfiguration.create();
try {
HBaseAdmin admin = new HBaseAdmin(conf);
} catch (MasterNotRunningException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ZooKeeperConnectionException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
HTableDescriptor tableDescriptor = new HTableDescriptor(TableName.valueOf("people")); What should I do? Thank you!
... View more
11-18-2014
01:55 PM
I tyed to change the user into spark but I don't know the password. I tried cloudera and spark but it didn't work. Then I changed into superuser and in superuser I have another error ./spark-submit --class com.seb.standard_self.App --master "spark://quickstart.cloudera:7077" /home/cloudera/workspace/standard-to-self-explicit/target/standard-self-0.0.1-SNAPSHOT.jar
Exception in thread "main" java.lang.NoClassDefFoundError: org.apache.spark.deploy.SparkSubmit
at gnu.java.lang.MainThread.run(libgcj.so.10)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.SparkSubmit not found in gnu.gcj.runtime.SystemClassLoader{urls=[file:./,file:/usr/lib/spark/conf/,file:/etc/hadoop/conf/,file:/etc/hadoop/conf/,file:/usr/lib/hadoop/../hadoop-hdfs/./], parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
at java.net.URLClassLoader.findClass(libgcj.so.10)
at java.lang.ClassLoader.loadClass(libgcj.so.10)
at java.lang.ClassLoader.loadClass(libgcj.so.10)
at gnu.java.lang.MainThread.run(libgcj.so.10) ... Thank you!
... View more
11-18-2014
01:05 PM
In fact the generated jar wasn't ok(I fixed this in my pom.xml) <build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version>
<configuration>
<archive>
<manifest>
<mainClass>com.seb.standard_self.App</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
</plugins>
</build>
</project> When I run my jar with the spark-submit I get another error (wrights) (still not the error that I get on my eclipse) INFO Utils: Successfully started service 'HTTP file server' on port 41178.
14/11/18 13:02:43 INFO Utils: Successfully started service 'SparkUI' on port 4040.
14/11/18 13:02:43 INFO SparkUI: Started SparkUI at http://10.0.2.15:4040
14/11/18 13:02:57 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=cloudera, access=EXECUTE, inode="/user/spark":spark:spark:drwxr-x---
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:255)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:236)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:178)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137) Thank you! Alina
... View more
- « Previous
- Next »