I have an Oozie Workflow where I am have a job job which loads some data into a table, I refresh the table in Impala and then have an Impala query to export the most recent data in this table to a CSV File.   My Problem is that even after doing the Impala refresh I do not get the most recent data only the data for the previous load.   For Example I have a process that starts running at 1pm spark job finishes at 1:15pm impala refresh is executed 1:20pm then at 1:25 my query to export the data runs but it only shows the data for the previous workflow which run at 12pm and not the data for the workflow which ran at 1pm.   I am using Oozie and cdh 5.15.1.   Sample Warning Message Read 972.32 MB of data across network that was expected to be local. Block locality metadata for table '..' may be stale. Consider running "INVALIDATE METADATA ... Thanks ... View more

I want to export an impala query where it concats many columns into one without a delimeter and the extra quotes that come with it .   Here is my query  impala-shell --ssl -B -i ${load_balancer} -f ${sql_file_local_subscriber} -o ${file_path_edge_subscriber} --print_header; Basically it gives " ""A"",""B"",""C"" "   I want "A" , "B" , "C".   How can I accomplish this through impala.   Thank you ... View more

 I am trying to run a spark2 program from intellij but I am unable to access hive through spark   val sparkSession = SparkSession.builder.master("local[2]") .config("spark.hadoop.parquet.enable.summary-metadata", "false") .config("hive.exec.dynamic.partition", "true") .config("hive.exec.dynamic.partition.mode", "nonstrict") .config("parquet.compression", "GZIP") .config("spark.sql.warehouse.dir","thrift://localhost.localdomain:9083") .config("hive.metastore.warehouse.dir","thrift://localhost.localdomain:9083") .enableHiveSupport() .getOrCreate Where is my hive-metastore directory I though it would be the one from hive-site.xml   Error message here   Exception in thread "main" java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState': at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:981) at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110) at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109) at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.apply(SparkSession.scala:878) at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.apply(SparkSession.scala:878) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) at scala.collection.mutable.HashMap.foreach(HashMap.scala:99) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:878) at Test$.main(Test.scala:16) at Test.main(Test.scala) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:978) ... 12 more Caused by: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog': at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:169) at org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:86) at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101) at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:101) at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:100) at org.apache.spark.sql.internal.SessionState.<init>(SessionState.scala:157) at org.apache.spark.sql.hive.HiveSessionState.<init>(HiveSessionState.scala:32) ... 17 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:166) ... 25 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264) at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:353) at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:257) at org.apache.spark.sql.hive.HiveExternalCatalog.<init>(HiveExternalCatalog.scala:66) ... 30 more Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Lorg/apache/hadoop/hive/conf/HiveConf;Lorg/apache/hadoop/hive/metastore/HiveMetaHookLoader;Ljava/util/Map;Ljava/lang/String;)Lorg/apache/hadoop/hive/metastore/IMetaStoreClient; at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2935) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2954) at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3179) at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:210) at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:197) at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:307) at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:268) at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:243) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:512) at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:188) ... 38 more   The code works until I add the .enableHiveSupport() . Why????    Thank you ... View more

I am looking for    <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.10</artifactId> <version>1.6.0</version> </dependency> But the cloudera version how do I find while maintaning the spark version same for spark streaming   <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming_2.10</artifactId> <version>1.6.0</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming-kafka_2.10</artifactId> <version>1.6.1</version> </dependency> As well as for say different versions of spark    I found this but I am still a bit lost.   https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh5_maven_repo_57x.html   Thanks  ... View more

I have a kafka spark streaming job which runs a consumer which reads from a producer and the job runs fine for a while up to 25 hours but then for some reason it fails and it gives this error message.   I cant figure out why . I changed the spark submit to take the active and not stand by node.   Any suggestions would be greatly appreciated ... View more

Here is the code inside the streamset   val dfOlDDatta = hivecontext.sql(s"select * from $db where part_id=$partId") dfOlDDatta.persist() /*.withColumn("testDate2", unix_timestamp(col("timeStamp"), "yyyy MMM dd HH:mm:ss").cast("String")). withColumn("testDate3", from_unixtime(col("testDate2"), "yyyy-MM-dd HH:mm:ss").cast("String")) .withColumn("dt_skey1", from_unixtime(col("testDate2"), "yyyyMMdd").cast("Int")) .withColumn("date2", from_unixtime(col("testDate2"), "yyyy-MM-dd HH:mm:ss").cast("String"))*/ val unionDf = dfTransformed.unionAll((dfOlDDatta)) //maybe change append to an overwrite println("writing to hdfs new partition") unionDf.coalesce(1).write.mode("overwrite").partitionBy("part_id").parquet(tempHdfsLocation) val fullPartitionString = "part_id=" + partId //hivecontext.sql(s"load data inpath '$tempHdfsLocation/$fullPartitionString' overwrite into table $db partition ($fullPartitionString)") unionDf.coalesce(1).write.mode("overwrite").partitionBy("part_id").parquet(explLocation) dfOlDDatta.unpersist() This is all within a spark streaming context.   This fails after one run through the spark streaming context .   Always after one run.                     error in short   17/11/30 15:24:01 ERROR datasources.InsertIntoHadoopFsRelation: Aborting job. java.io.FileNotFoundException: File does not exist:    I am using spark 1.6 and cdh 5.7.2   Any suggestions greatly appreciated.   ... View more

I was trying to understand why I was receiving this error and if I need to take it seriously as I think that the spark job is still succeeding it says succeeded and the data is loaded into the table   Issue with setfacl: Permission denied. user=..   I am loading data from a temporary location on hdfs where files are say ownded by user A into the hive table where the files are owned by user B through spark code.   I read somewhere else that this did not seem like an important issue just wanted to confirm.   Thanks   ... View more

Looking a website or repository where I can easily just copy paste pom files with all dependencies ready and built for a specific version of cdh.   Like the pom file will spark scala kafka any other dependences for different versions of these software and different versions of cdh does such a repo exist.   Specifically for the quickstart vm     Thanks   ... View more

I receive this error when I try to overwrite a table in apache spark.   Machine generated alternative text: URL: agnostics: User class threw exceptiom org.apache hadoop securityAccessControIExceptiom Permission denied by sticky bit: user—...expl.db" at org.apache nadoop fidts server. namenode DefaultAuthorizationProvider c heckStickyBit(DefaultAuthorizationProvider java:3B7) at org.apache nadoop fidts server. name node. DefaultAuthcrizationProvider c heckPermission(DetaultAuthorizationProvider.java: 159) at org.apache nadoop fidts server. name node F S PermissionChecker c heckPermissicn(F S PermissionChecker.java: 152) at org.apache hadoop hdts.server_namenode_FSDirectory_checkPermission(FSDirectory java:352g) at org.apache hadoop hdts.server_namenode_FSNamesystem chec kPermission(FSNamesystem java:BE50) at org.apache hadoop hdts.server_namenode_FSNamesystem at org.apache hadoop hdts.server_namenode_FSNamesystem deletelnt(FSNamesystem.java:4070) at org.apache hadoop hdts.server_namenode_FSNamesystem delete(FSNamesystem.java:4054) at org.apache hadoop at org.apache hadoop hdts.server_namenode_AuthorizationProviderProxyCIientProtocoLdelete(AuthorizationProviderProxyClientProtocoLjava:30B) at org.apache hadoop hdts.protocoIPB.ClientNamenodeProtocolServerSideTransIatorPB_delete(ClientNamenodeProtocolServerSideTranslatorPB.java:603) at org.apache hadoop hdts.protocoI.proto.ClientNamenodeProtocoIProtosSClientNamenodeProtocolS2 callBIockingMethod(CIientNamenodeProtocolProtos.java) at org.apache hadoop ipc_ProtobufRpcEngineSServerSProtoaufRpclnvoker.calI(ProtoöutRpcEngine.java:E17) at org.apache hadoop 1073) at org.apache hadoop at org.apache hadoop at java.security.AccessController doPrivileged(Native Method) at javax_security auth at org.apache hadoop security_userGroupIntormation at org.apache hadoop 2214)     I am using spark 1.6 and cdh 5.7.2  I try to overwrite using a hiveContext and persisting the df   dfPartId.coalesce(nCoalesce).write.mode("overwrite").partitionBy(partitionCol).parquet(expl_hdfs) hiveContext.sql(s"msck repair table $capiTbl") Thanks ... View more

I am using spark 1.6 cdh 5.7.2 and sometimes when I run a spark job with saveAsTable where the dataframe is created from a hive query it just deletes the table but not always which is weird.   https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_spark_ki.html   I noticed a similiar issue is referenced by cloudera over here. The odd thing is I tried to emulate this issue on the vm and I was able to fix it but this did not work when we deployed it to the server maybe it has something to do with parallelism.     For example something like this. var dfLong = hiveContext.sql(s"select * from $tblName") dfLong.write.mode("overwrite").parquet(tblLocation) Would sometimes cause the table to be overwritten I tried to persist the df but that did not work either.   If I manually create the table and insert each time will that work using the hiveContext and my question is if so why?   Thanks   ... View more

To get a hive table to appear in impala I can do invalidate metadata on everything.   But that will be very memory intensive is there a way to invalidate metadata and just get this database invalidated.   Like say I have 100 schemas and I create a new one in hive say Ab.   Can I do something like invalidate metadata Ab.   I know this can be done on a table but what about a schema?   Thanks  ... View more

I am running the workflow in the cloudera quickstart vm 5.7.2 I am able to run the job from the command line but then in hue it just freezes gives the error  JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.   Here is my workflow.xml    <workflow-app name="sample-workflow" xmlns="uri:oozie:workflow:0.5"> <global> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> </global> <start to="sshScript"/> <action name="sshScript"> <shell xmlns="uri:oozie:shell-action:0.3"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <exec>hello.sh</exec> <argument>44 </argument> <file>/user/cloudera/hive-config.xml#hive-config.xml</file> </shell> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message>Failed script</message> </kill> <end name='end'/> </workflow-app>   and my job.properties   oozie.use.system.libpath=True security_enabled=False dryrun=False queueName=default nameNode=quickstart.cloudera:8020 jobTracker=quickstart.cloudera:8032 oozie.libpath=/user/hue/oozie/workspaces/sampleScript/ oozie.wf.application.path=/user/hue/oozie/workspaces/sampleScript/workflow.xml     Thanks ... View more

I have been able to run an oozie workflow from the actual cluster. But I was just looking for a tutorial that would walk me through doing it on my vm preferably with the neccessary sample properties and workflow.xml files. Make sure all the files are setup properly etc.. Where to place everything on the hdfs.  Where can I find this in the cloudera documentation or somewere else on the net.   Thanks ... View more

I tried downloading the kafka parcel but I received this error in the image parcel  not found on remote repository.    As well in shows in the second image that the cluster on the vm is not configured to use parcels.   My question is there any easy way for me to download kafka to my quick start vm either through the parcel or otherwise. Thanks   ... View more

I set up my workflow put it into hdfs as well and I try to run the conf directory the properties file with this syntax. I am really not sure why it is not working if I have a typo in my workflow.xml or job.properties or if I need to modify some config setting. Thanks   Error message    Here is the link to the error message, https://ibb.co/dkHnJv      ... View more

I just want to be able to run an oozie job anything from the quickstart vm. Really just the list of steps required to configure and set up everything. I have been able to run from hue but I had a lot of difficulty from the quickstart vm not sure why would rather just over.   Thanks ... View more

I receive the error database does not exist with this code in intellij spark maven project and on all new projects. But what is weird is that in an older project with the same code it works. Here is the code and below the error log. I have the hive-sites.xml in both the hive and spark conf and set all directories user/hive/warehouse to 777 what else could be the error. Thanks I am using the cloudera quickstart vm cdh 5.7.2 Spark 1.6 scala 2.10.5 ... View more

I saw this https://issues.cloudera.org/browse/IMPALA-1832 and I was wondering if there was any update on this. For example estimated memory used is x and actual memory used per node 3x or 4x. Talking about queries which are using gb of ram. Thanks ... View more

When I search in hue for running or completed jobs lets say abc it only shows me the 10 or 50 most recent on the page. My problem is that it doesnt the 10 or 50 which match the search ony 10 or 50 most recent so that I need to click next many times if I want to find a job more than a day old. My question is there a way to be able to see more on one page like 500 so that it would come up or to make my search search through all the jobs of all time. What I receive is a buch of empty pages and having to click next many times before I find the job that I am looking for. Thank you ... View more

I had tried to pip install the cloudera manager api after having installed pip. I executed the command  pip install -vvv cmd-api and I tried it as well with sudo and received the same error.  Here is a screen shot of the error I tried modifying my certificates thought might be a proxy related error but not sure. I am using python 2.6.6 any suggestions are greatly appreciated. It still seems to have not installed. Thanks   ... View more

I am a little bit lost with the core-sites.xml and hdfs-sites.xml and would like to know where are the namenode ,job tracker and queue locations so that I may ran an oozie workflow thanks. I am looking for basically an explanation of those 2 files for dummies what they do and what I need from them to run an oozie hive or spark job. Thanks again ... View more