@Plop564 I am not an expert in Spark, but my understand is below: 1. I will have 100 output files >>> this depends how many partitions you have in your original DF. "coalesce" can only reduce number of partitions, so if you have less than 100 partitions before, then it won't do anything, as "coalesce" does not do shuffling. If you want to guarantee number of output files, I believe "repartition" function is better. 2. Each single CSV file is locally sorted, I mean by the "date" column ascending >>> Yes 3. Files are globally sorted, I mean CSV part-0000 have "date" inferior to CSV part-0001, CSV part-0001 have "date" inferior to CSV part-0002 and so on .. >>> I believe it is also Yes, but will wait for other Spark experts to confirm. Cheers Eric ... View more

@sbn, /etc/spark2/conf should be a symlink to /etc/spark2/conf.cloudera.spark2_on_yarn, can you confirm by running: ls -al /etc/spark2 Cheers Eric ... View more

@Cl0ck , What errors are you getting? ... View more

@gimp077 , Did you mean that "REFRESH" takes time, and eventually you can see the update data, but just some delay? How big is the table? I mean in terms of number of partitions and number of files in HDFS? Eric ... View more

Hmm, I can now reproduce the issue.   After creating the privilege: kafka-sentry --config /etc/sentry/conf -gpr -r eric-test -p 'HOST=*->CLUSTER=kafka-cluster->action=clusteraction'   It is stored as "cluster_action": kafka-sentry --config /etc/sentry/conf -lp -r eric-test ... HOST=*->CLUSTER=kafka-cluster->action=cluster_action   And when try to drop it, it will fail with the error you are seeing.   Need a bit more time to look into why. ... View more

@Pit, I do not think anything will change for the CDH versions that have been released. New OS version will be added to future release of CDH. Currently CDH 6.3 has been release a while back and still only support CentOS 7. Cheers Eric ... View more

@bene, Looks like the valid value should be "clusteraction", not "cluster_action", see below code: https://github.com/cloudera/sentry/blob/cdh5-1.5.1_5.14.4/sentry-core/sentry-core-model-kafka/src/main/java/org/apache/sentry/core/model/kafka/KafkaActionConstant.java#L31 I am not sure how it gets in in the first place though. Cheers Eric ... View more

Sorry that I am not answering your question directly, but I am wondering why you want to run hive query through pyspark? Why don't you just use SparkSQL? Cheers Eric ... View more

@chhaya_vishwaka, Do you still see the same error? Have you checked if scm.CM_VERSION has been re-created correctly with data in there? Cheers Eric ... View more

@parthk , There is no current date locked in for the new impala release that will support Ranger at the moment. However, I would like to ask why you do not want to have kerberos? Authorization does not work properly without Authentication in the front. Think about an online application, you surely want users to be able to login first, before you can say what level of access they should have. Same applies in CDH world. Kerberos acts as the front end login, and Sentry/Ranger acts as the backend authorization control. So without Kerberos, you are allowing everyone to be able to access CDH. I strongly suggest you to implement Kerberos first before Sentry, Ranger is the same story regardless. Cheers Eric ... View more

@gimp077 , When you say you did "REFRESH" the table, did you run "REFRESH <tablename>" or "INVALIDATE METADATA", because those two are not identical in the way they work. Is your table partitioned? If yes, can you see the new partition from Impala by running 'SHOW PARTITIONS <tablename>"? Cheers Eric ... View more

@yukti, I see that there is a new line after UUID in the log, not sure if that is formatting issue on the post or it is there in the log? Can you please double check and also the content in UUID file? Cheers Eric ... View more

Hi Vinay, Can you search sessionid "0x26c4d39814f4b8e" in ZK server log to see if there any clue to see why the session got closed? Thanks Eric ... View more

Hi Vijay, Sorry, haven't able to nail down the cause yet, but can you collect EXPLAIN EXTENDED of the query and share the output as attachment to the post? EXPLAIN EXTENDED select * from tdb1.t1 where bs1_dt=2017-06-23; EXPLAIN EXTENDED select * from tdb1.t1; I would like to check how HS2 does the query plan and see if there is any clue. Cheers Eric ... View more

@wati, I can see that hivexmlserde has not been updated since 2015, see below link: https://mvnrepository.com/artifact/com.ibm.spss.hive.serde2.xml/hivexmlserde This is confirmed by checking the github repository: https://github.com/dvasilen/Hive-XML-SerDe It is unlikely that it will work with latest Hive version, given there are lots of changes. You might need to find alternatives. I have not used any, so I do not have recommendations. ... View more

I am a bit running out of ideas, but a couple of more questions: 1. what version of CDH are you using? I can't remember if I have asked 2. can you try to remove below directory in HDFS: hdfs://nameservice1/abc/xyz/efg/bs1_dt=__HIVE_DEFAULT_PARTITION__ and also drop the partition on the table against this path, just to make sure that it is clean and try again in in beeline with your problematic query? I remember this might cause issue sometimes, but can't remember which version though, need a bit more digging. Cheers ... View more

Before you try to connect from PyHive, please confirm if you can do so through beeline and make sure HS2 is functioning OK. Again, HS2 port is 10000 based on your config, so you should connect to 10000 from client side. Do you see any errors from HS2 log? Try beeline and what does it say? Cheers ... View more

hive.server2.thrift.port is the one you should connect to, which is 10000, NOT 10002, please try with 10000 port and see how you go. Cheers Eric ... View more

@ranger Did you change HS2's default ports? 10002 by default is HS2's web UI port, the thrift port should be 10000. Can you please check your HS2's hive-site.xml file and confirm the value for hive.server2.thrift.port and hive.server2.webui.port? Thanks Eric ... View more

@Karachay , It is difficult to know what is happening without seeing what you see from your terminal. Did the Sqoop job finished OK? Any errors? Did you mean the below command returned empty resultset? hadoop fs -ls /user/hive/warehouse/ Can you share the output of Sqoop command to confirm it finished without issues? Cheers Eric ... View more

That is Cloudera Manager Server then, not the agent. And it is requirement that you allocate a few GBs of memory for Cloudera Manager server, please refer to below documentation: https://www.cloudera.com/documentation/enterprise/release-notes/topics/hardware_requirements_guide.html#sizing_cm_server_reqs For a small cluster of less than 10 nodes, you still need at least 2GB of memory for CM server to function properly. So you 1.9GB is about right and you should not go anything less than that. Cheers Eric ... View more