I have an issue while verifying that I can connect to hbase from Spark (from spark-shell so through scala). scala version: 2.11.7 spark version: 2.2.1 hbase: 1.3.1 I followed this example: https://docs.cloudera.com/runtime/7.1.1/managing-hbase/topics/hbase-example-using-hbase-spark-connector.html . I had to load quite some extra hbase-related jars to get all commands in the example above to work (hbase-client/hbase-common/hbase-spark). Only at the end when I read/write I get an error. This is the full example with output:   scala> import org.apache.hadoop.fs.Path import org.apache.hadoop.fs.Path scala> import org.apache.hadoop.hbase.HBaseConfiguration import org.apache.hadoop.hbase.HBaseConfiguration scala> val conf = HBaseConfiguration.create() warning: Class org.apache.hadoop.hbase.classification.InterfaceAudience not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.classification.InterfaceStability not found - continuing with a stub. conf: org.apache.hadoop.conf.Configuration = Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hbase-default.xml, hbase-site.xml scala> conf.addResource(new Path("/home/jb/Documents/cloud-local/hbase-1.3.1/conf/hbase-site.xml")) scala> import org.apache.hadoop.hbase.spark.HBaseContext import org.apache.hadoop.hbase.spark.HBaseContext scala> new HBaseContext(sc, conf) // "sc" is the SparkContext you created earlier. warning: Class org.apache.hadoop.hbase.classification.InterfaceAudience not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.classification.InterfaceStability not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.classification.InterfaceAudience not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.classification.InterfaceAudience not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.exceptions.DeserializationException not found - continuing with a stub. error: Class org.apache.hadoop.hbase.exceptions.DeserializationException not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.protobuf.generated.ClientProtos not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.classification.InterfaceAudience not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.classification.InterfaceStability not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.classification.InterfaceAudience not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.classification.InterfaceAudience not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.classification.InterfaceStability not found - continuing with a stub. scala> val sql = spark.sqlContext sql: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@76e00bdb scala> import java.sql.Date import java.sql.Date scala> case class Person(name: String, | email: String, | birthDate: java.sql.Date, | height: Float) defined class Person scala> var personDS = Seq( | Person("alice", "alice@alice.com", Date.valueOf("2000-01-01"), 4.5f), | Person("bob", "bob@bob.com", Date.valueOf("2001-10-17"), 5.1f) | ).toDS personDS: org.apache.spark.sql.Dataset[Person] = [name: string, email: string ... 2 more fields] scala> personDS.write.format("org.apache.hadoop.hbase.spark").option("hbase.columns.mapping","name STRING :key, email STRING c:email,birthDate DATE p:birthDate, height FLOAT p:height") .option("hbase.table", "person") .option("hbase.spark.use.hbasecontext", false) .save() name STRING :key, email STRING c:email,birthDate DATE p:birthDate, height FLOAT p:height java.lang.NullPointerException at org.apache.hadoop.hbase.spark.HBaseRelation.<init>(DefaultSource.scala:139) at org.apache.hadoop.hbase.spark.DefaultSource.createRelation(DefaultSource.scala:79) at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:469) at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:50) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56) at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117) at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116) at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92) at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92) at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:609) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:233) ... 48 elided   Same as in this post (https://stackoverflow.com/questions/52372245/hbase-spark-load-data-raise-nullpointerexception-error-scala), so already included this to ensure the details of hbase-site.xml are included, but that does not fix it. This is the createRelation method in DefaultSource:   public BaseRelation createRelation(final SQLContext sqlContext, final SaveMode mode, final Map<String, String> parameters, final Dataset<Row> data) { final HBaseRelation relation = new HBaseRelation((Map)parameters, (Option)new Some((Object)data.schema()), sqlContext); relation.createTable(); relation.insert((Dataset)data, false); return (BaseRelation)relation; }   I guess one of the input params must be null but can't figure out which one.   Input welcome!   ... View more

Dear forum, another case to think about!   I created a table in HIVE as a textfile. When I query it, it looks fine for all records. Next, in Impala, I use the INVALIDATE METADATA statement and afterwards , query the table. Now  Impala shows me for some records a question mark as if there are special characters for a couple of records (�). I notice the data in these fields can not be used anywhere else in following steps (instead of reading values, he will complain the values are not valid). When I examine the textfile on hdfs (through some text editor like sublime with UTF-8 encoding ), I see no special characters and all characters encountered, look as expected. As said, invalidate metadata nor refresh fixes the issue but after a restart of the impala services, the data is available as expected in impala.     Currently we create the table as text file and get the behavior described above. Before we created the table as a parquet file, but then got the error : --> File 'hdfs://ourcluster/user/hive/warehouse/tmp.db/thetable/000000_0' has an invalid version number: <some value> This could be due to stale metadata. Try running "refresh tmp.thetable". --> Note that this <some value> would always be something that comes from the data (a part of a string that is in the data). The refresh would not fix it, (and as said we already do an invalidate metadata). Note that when we restart the impala service, this error goes away and the data can be queried. The files then seem to be "uncorrupted". I have read a similar post elsewhere that suggests the data would be corrupted when one encounters this error.   Note: we use a collect_set function to create the field that gives the problem during the creation of the table in HIVE.Our current trail of thought is that in some cases (15 out of several million) this gives problematic results  but what happens exactly is not understood.   Thanks for any input!     ... View more

We are running an oozie flow. All steps that are hive/impala run fine. One step in the oozie flow is a shell script that launches the hadoop streaming jar. We have run the exact same code on an old cluster (cdh 5.12) and this worked fine. Also everything related to this script runs fine locally. However when ran through oozie it fails. We are using cdh6.2.   The job logs show following error:   Error launching job , Invalid job conf : cache file (mapreduce.job.cache.files) scheme: "hdfs" host: "poccluster" port: -1 file: "/thefilepath" conflicts with cache file (mapreduce.job.cache.files) /user/yarn/.staging/job_1558348093683_0051/files/run_it_parallel.sh Streaming Command Failed!   We have tried clearing the user/app cache in the yarn directories of our users. There is no configuration parameter mapreduce.job.cache.files. The /user/yarn/.staging directory is empty.         The container logs show following error:   java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:410) at org.apache.oozie.action.hadoop.LauncherAM.access$300(LauncherAM.java:55) at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:223) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:217) at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141) Caused by: org.apache.oozie.action.hadoop.LauncherMainException at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:76) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:104) at org.apache.oozie.action.hadoop.ShellMain.main(ShellMain.java:63) ... 16 more This looks like it is related to access rights. We do not have kerberos and sentry is disabled for all services. We are quite sure that this process does not have an error on our side (have used it on different cluster and it runs on a worker node locally without issues).  Possibly the fact that when we run it locally, we are using our own user (and not the users used by oozie/yarn) is related to why we get this error. Any input, maybe especially on the first error from the job logs, would be hugely appreciated.   Thanks! ... View more

I am working with the cloudera quickstart image on docker which includes impala but not kudu. I would like to add kudu. What is the best approach? I am currently pursueing #1: APPROACH 1 1)go inside the docker, install kudu master and server 2)see that kudu master and server are running 3)I tried to add the " -kudu_master_hosts=localhost " to the /etc/default/impalad file, and restart the impala service, but he does not recognize this flag. 3)Hence failure during create table -> it fails since he considers the word primary key as a syntax error. APPROACH 2 I could do it using cloudera manager, using the parcels, but this looks harder to put in a Dockerfile APPROACH 3 Using docker-compose : is there a reference somewhere on exactly how to do it?   Or other suggestions? This sounds a standard problem but I did not find any guideline online.   Thanks! ... View more

Hi all, I have a cluster that was working fine for weeks and am mainly using Impala on Kudu tables. Sentry is running on the cluster. Since recently I get an error for the 'DROP TABLE ' command: `ImpalaRuntimeException: Error making 'dropTable' RPC to Hive Metastore: CAUSED BY: MetaException: Failed to connect to Sentry service null`. I believe that the data indeed was deleted, since a SELECT query on the table will complain he can not find the file in hdfs. When I run the 'invalidate metadata' command before dropping the table, the error goes away, but not always.If I would try to drop the same table again(I believe at this point the data is already removed), after the 2nd attempt I would get following error: `ImpalaRuntimeException: Table xxx no longer exists in the Hive MetaStore. Run 'invalidate metadata xxx' to update the Impala catalog.` Note: this does not happen with tables I created in Hive and that I now try to query in Impala-they are created all in Impala.I did not have the error before, and feel like it started since I have run an 'invalidate metadata statement' for some other reason for the first time recently .     Thanks for input ... View more

Hi all, I notice "ingest (near) real time streaming" is a top for the CCA175 exam. Does this mean one can expect any questions on kafka/flume/spark streaming? I believe this is a new topic that was not included yet e.g. begin 2017. Also back then, the avrotools were part of the skillset, while now I do not see that mentioned anymore. Does anyone have more info on these 2 questions: CCA 175: 1)is kafka/flume/spark streaming part of exam skills? 2)no more avro tools   Many thanks!! ... View more

Dear community, if I want to run the teragen program with output compression, is this the correct command: sudo -u hdfs hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.13.0.jar teragen -D mapreduce.map.output.compress=true -D mapreduce.map.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D mapreduce.output.fileoutputformat.compress=true 1000 /user/dev/teragen   Is the following correct? First option sets intermediate compression, 2nd option specifies it needs to be zipped compression, third option would ensure also the output is zipped. I have seen multiple commands some of them deprecated. I notice that my output of the terragen is still not zipped so something is still not correct   Thanks! ... View more

Dear community, As I am preparing for the CCA131 administrator exam, I notice that two of the required skills have to do with Yarn resource management (configure it based on user feedback and configuration of the fair scheduler). One can go incredibly deep into tuning yarn. Can someone give an idea of what kind of question could be expected? If it is just allocating resources (either static or dynamic as can be done through cloudera manager), it is not that hard, but this question could potentially go so much more into depth (there are tens of configuaration parameters). Does anyone have a clue? Or some relevant examples of the things that one might have to do during the exam?    In general, as mentioned before on this forum, it is still a little vague how much into depth one should be able to go for the CCA131 exam, so all feedback is welcome.   Many thanks! ... View more