I am wondering what difference in IO can be expected for hbase with storage in the cloud VS storage on hdfs. I would expect that when data is retrieved from hdfs, it will be a lot faster than from the cloud (like adls-in my specific case adls gen2=abfs). Is there somewhere where I can test this? Or find a previous study for this? If this is the case, then one would expect that current hbase performance for reading data is a bit less than some years ago when everything was on premise and typically hdfs was used?     Maybe I am missing something obvious here, so any insight is appreciated !   ... View more

I have a couple of quite complicated oracle functions that shuold be migrated to Hadoop. They contain a lot of cursors/loop/begin statements (you create a select, and then loop over all rows herein, with many nested if loops). In oracle you can call them with SELECT  calculatesomstuff(input_nature, input_amount, ).   My input data for which I want to use this function, resides in hive. If it were possible, I would prefer to keep the function (I am sure now it works fine and dont want to have to retest everything since this can takes months), and call it through hadoop by connecting to oracle, but doubt this is easy/possible.   If this is not possible, I wonder what the best way is to migrate this in Hadoop. I see 2 options: 1)udf in hive (but translating long complex plsql to java/c++ sounds like a nuisance-most udfs are for some simple aggregations and operations on few columns-also my sql is far better than java/c++ in which my skills are not so good) 2) write reqular HIVE queries that do the same operations as in the function, in an ETL-like step (with possibly many intermediate tables). Inside an oracle function, a SELECT statement, followed by a loop over all rows comes closest to this. But then an outside user will never be able to call it  as simple as: SELECT calculatesuomestuff(input_nature,input_amount). 3)maybe SPARKSQL ? So question: do I come to the conclusion that a truely complex oracle function is a feature that can not be easily used in Hadoop and oracle here has an advantage? ... View more

Hi, we want to access hive data from a machine outside the cluster (cdh 6.1). The dataset is in a partitioned external hive table. The hive instance is ldap secured with a self-signed certificate. We are looking for the easiest way to query this data, for example through python. Looks a bit harder than first expected. Apparently pyhive is a good candidate, but unfortunately it does not support ldap. There is a 2 year old extra package that can be installed for this, but we had no luck. So the question is: what is the most straightforward way to connect to hive from outside the cluster, without issues with ldap. Does anything simple exist in python, or do we need to go to Squirrel or dbeaver? Does anyone have recent instructions for ldap ?   Note: we were able to connect through impala, but apparently impala 3.1.0 on cdh6.1 does not allow to query partitoned external hive tables trhough impala (tried INVALIDATE METADATA/REFRESH/MSCK REPAIR TABLE). Querying gives: Failed to load metadata for table: 'mydb.mytable' CAUSED BY: TableLoadingException: Unrecognized table type for table: mydb.mytable . ... View more

I have an issue while verifying that I can connect to hbase from Spark (from spark-shell so through scala). scala version: 2.11.7 spark version: 2.2.1 hbase: 1.3.1 I followed this example: https://docs.cloudera.com/runtime/7.1.1/managing-hbase/topics/hbase-example-using-hbase-spark-connector.html . I had to load quite some extra hbase-related jars to get all commands in the example above to work (hbase-client/hbase-common/hbase-spark). Only at the end when I read/write I get an error. This is the full example with output:   scala> import org.apache.hadoop.fs.Path import org.apache.hadoop.fs.Path scala> import org.apache.hadoop.hbase.HBaseConfiguration import org.apache.hadoop.hbase.HBaseConfiguration scala> val conf = HBaseConfiguration.create() warning: Class org.apache.hadoop.hbase.classification.InterfaceAudience not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.classification.InterfaceStability not found - continuing with a stub. conf: org.apache.hadoop.conf.Configuration = Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hbase-default.xml, hbase-site.xml scala> conf.addResource(new Path("/home/jb/Documents/cloud-local/hbase-1.3.1/conf/hbase-site.xml")) scala> import org.apache.hadoop.hbase.spark.HBaseContext import org.apache.hadoop.hbase.spark.HBaseContext scala> new HBaseContext(sc, conf) // "sc" is the SparkContext you created earlier. warning: Class org.apache.hadoop.hbase.classification.InterfaceAudience not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.classification.InterfaceStability not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.classification.InterfaceAudience not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.classification.InterfaceAudience not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.exceptions.DeserializationException not found - continuing with a stub. error: Class org.apache.hadoop.hbase.exceptions.DeserializationException not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.protobuf.generated.ClientProtos not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.classification.InterfaceAudience not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.classification.InterfaceStability not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.classification.InterfaceAudience not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.classification.InterfaceAudience not found - continuing with a stub. warning: Class org.apache.hadoop.hbase.classification.InterfaceStability not found - continuing with a stub. scala> val sql = spark.sqlContext sql: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@76e00bdb scala> import java.sql.Date import java.sql.Date scala> case class Person(name: String, | email: String, | birthDate: java.sql.Date, | height: Float) defined class Person scala> var personDS = Seq( | Person("alice", "alice@alice.com", Date.valueOf("2000-01-01"), 4.5f), | Person("bob", "bob@bob.com", Date.valueOf("2001-10-17"), 5.1f) | ).toDS personDS: org.apache.spark.sql.Dataset[Person] = [name: string, email: string ... 2 more fields] scala> personDS.write.format("org.apache.hadoop.hbase.spark").option("hbase.columns.mapping","name STRING :key, email STRING c:email,birthDate DATE p:birthDate, height FLOAT p:height") .option("hbase.table", "person") .option("hbase.spark.use.hbasecontext", false) .save() name STRING :key, email STRING c:email,birthDate DATE p:birthDate, height FLOAT p:height java.lang.NullPointerException at org.apache.hadoop.hbase.spark.HBaseRelation.<init>(DefaultSource.scala:139) at org.apache.hadoop.hbase.spark.DefaultSource.createRelation(DefaultSource.scala:79) at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:469) at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:50) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56) at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117) at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116) at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92) at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92) at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:609) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:233) ... 48 elided   Same as in this post (https://stackoverflow.com/questions/52372245/hbase-spark-load-data-raise-nullpointerexception-error-scala), so already included this to ensure the details of hbase-site.xml are included, but that does not fix it. This is the createRelation method in DefaultSource:   public BaseRelation createRelation(final SQLContext sqlContext, final SaveMode mode, final Map<String, String> parameters, final Dataset<Row> data) { final HBaseRelation relation = new HBaseRelation((Map)parameters, (Option)new Some((Object)data.schema()), sqlContext); relation.createTable(); relation.insert((Dataset)data, false); return (BaseRelation)relation; }   I guess one of the input params must be null but can't figure out which one.   Input welcome!   ... View more

Dear forum, another case to think about!   I created a table in HIVE as a textfile. When I query it, it looks fine for all records. Next, in Impala, I use the INVALIDATE METADATA statement and afterwards , query the table. Now  Impala shows me for some records a question mark as if there are special characters for a couple of records (�). I notice the data in these fields can not be used anywhere else in following steps (instead of reading values, he will complain the values are not valid). When I examine the textfile on hdfs (through some text editor like sublime with UTF-8 encoding ), I see no special characters and all characters encountered, look as expected. As said, invalidate metadata nor refresh fixes the issue but after a restart of the impala services, the data is available as expected in impala.     Currently we create the table as text file and get the behavior described above. Before we created the table as a parquet file, but then got the error : --> File 'hdfs://ourcluster/user/hive/warehouse/tmp.db/thetable/000000_0' has an invalid version number: <some value> This could be due to stale metadata. Try running "refresh tmp.thetable". --> Note that this <some value> would always be something that comes from the data (a part of a string that is in the data). The refresh would not fix it, (and as said we already do an invalidate metadata). Note that when we restart the impala service, this error goes away and the data can be queried. The files then seem to be "uncorrupted". I have read a similar post elsewhere that suggests the data would be corrupted when one encounters this error.   Note: we use a collect_set function to create the field that gives the problem during the creation of the table in HIVE.Our current trail of thought is that in some cases (15 out of several million) this gives problematic results  but what happens exactly is not understood.   Thanks for any input!     ... View more

We are running an oozie flow. All steps that are hive/impala run fine. One step in the oozie flow is a shell script that launches the hadoop streaming jar. We have run the exact same code on an old cluster (cdh 5.12) and this worked fine. Also everything related to this script runs fine locally. However when ran through oozie it fails. We are using cdh6.2.   The job logs show following error:   Error launching job , Invalid job conf : cache file (mapreduce.job.cache.files) scheme: "hdfs" host: "poccluster" port: -1 file: "/thefilepath" conflicts with cache file (mapreduce.job.cache.files) /user/yarn/.staging/job_1558348093683_0051/files/run_it_parallel.sh Streaming Command Failed!   We have tried clearing the user/app cache in the yarn directories of our users. There is no configuration parameter mapreduce.job.cache.files. The /user/yarn/.staging directory is empty.         The container logs show following error:   java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:410) at org.apache.oozie.action.hadoop.LauncherAM.access$300(LauncherAM.java:55) at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:223) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:217) at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141) Caused by: org.apache.oozie.action.hadoop.LauncherMainException at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:76) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:104) at org.apache.oozie.action.hadoop.ShellMain.main(ShellMain.java:63) ... 16 more This looks like it is related to access rights. We do not have kerberos and sentry is disabled for all services. We are quite sure that this process does not have an error on our side (have used it on different cluster and it runs on a worker node locally without issues).  Possibly the fact that when we run it locally, we are using our own user (and not the users used by oozie/yarn) is related to why we get this error. Any input, maybe especially on the first error from the job logs, would be hugely appreciated.   Thanks! ... View more

I am working with the cloudera quickstart image on docker which includes impala but not kudu. I would like to add kudu. What is the best approach? I am currently pursueing #1: APPROACH 1 1)go inside the docker, install kudu master and server 2)see that kudu master and server are running 3)I tried to add the " -kudu_master_hosts=localhost " to the /etc/default/impalad file, and restart the impala service, but he does not recognize this flag. 3)Hence failure during create table -> it fails since he considers the word primary key as a syntax error. APPROACH 2 I could do it using cloudera manager, using the parcels, but this looks harder to put in a Dockerfile APPROACH 3 Using docker-compose : is there a reference somewhere on exactly how to do it?   Or other suggestions? This sounds a standard problem but I did not find any guideline online.   Thanks! ... View more

Hi all, I have a cluster that was working fine for weeks and am mainly using Impala on Kudu tables. Sentry is running on the cluster. Since recently I get an error for the 'DROP TABLE ' command: `ImpalaRuntimeException: Error making 'dropTable' RPC to Hive Metastore: CAUSED BY: MetaException: Failed to connect to Sentry service null`. I believe that the data indeed was deleted, since a SELECT query on the table will complain he can not find the file in hdfs. When I run the 'invalidate metadata' command before dropping the table, the error goes away, but not always.If I would try to drop the same table again(I believe at this point the data is already removed), after the 2nd attempt I would get following error: `ImpalaRuntimeException: Table xxx no longer exists in the Hive MetaStore. Run 'invalidate metadata xxx' to update the Impala catalog.` Note: this does not happen with tables I created in Hive and that I now try to query in Impala-they are created all in Impala.I did not have the error before, and feel like it started since I have run an 'invalidate metadata statement' for some other reason for the first time recently .     Thanks for input ... View more

Hi all, I notice "ingest (near) real time streaming" is a top for the CCA175 exam. Does this mean one can expect any questions on kafka/flume/spark streaming? I believe this is a new topic that was not included yet e.g. begin 2017. Also back then, the avrotools were part of the skillset, while now I do not see that mentioned anymore. Does anyone have more info on these 2 questions: CCA 175: 1)is kafka/flume/spark streaming part of exam skills? 2)no more avro tools   Many thanks!! ... View more

Dear community, if I want to run the teragen program with output compression, is this the correct command: sudo -u hdfs hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.13.0.jar teragen -D mapreduce.map.output.compress=true -D mapreduce.map.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D mapreduce.output.fileoutputformat.compress=true 1000 /user/dev/teragen   Is the following correct? First option sets intermediate compression, 2nd option specifies it needs to be zipped compression, third option would ensure also the output is zipped. I have seen multiple commands some of them deprecated. I notice that my output of the terragen is still not zipped so something is still not correct   Thanks! ... View more