Looking for some gudence on setting up alert using CM api for yarn application. duration>1h,memory > 100GB,cpu_millis>1M .Which resource in yarn api can be used for this. Any example would be highly helpful.   Current limitation of alert publisher is that we cannot set custome email recipient for different alerts. ... View more

Is it possible to create custom yarn application chart for specific pool usage using chart builder. If someone could help with the corresponding sample TS query  that would be great help.   I would like to combine below charts in to a dashboard. CM version 5.13   1) Number of application ran under specific pool for last one week. Separated by application type (MR,Tez,Spark etc) 2) Show number of applications which got killed /failed and their details. 2) Shows application which ran more than specified time . 3) Show applications which took more than specific number of mappers. 4) Shows applications which took more than specified number of cpu. ... View more

What is the best way to stream data from ES TO Hadoop (real time). Any suggestion examples. ... View more

Currently Cloudera spark 2 does not supports spark-avro or spark-xml , what are the alternate options available.  ... View more

Hi,   1) If we create a table (both hive and impala)and just specify stored as parquet . Will that be snappy compressed by default in CDH?   2) If not how do i identify a parquet table with snappy compression and parquet table without snappy compression?.   Also how to specify snappy compression for table level  whiel creating and also at global level, even if nobody specified at table level (all table stored as parquet should be snappy compressed).   Please help   ... View more

1) I want to create a dashboard in Cloudera Manager contanis yarn and hdfs charts. Yarn: weekly chart with daily numbers (y axis number and x axis date) 1) Total number of completed applications (succeeded+finished+failed+killed+error) and number based on application type (MR,SPARK,TEZ etc.)   HDFS: 1) hdfs usage growth per week for a month as a line graph How to achieve it ... View more

@Romainrexport csv/excel not working in chrome (error details below) but working fine in Firefox for hive query results. I have seen some earlier comments about the number of rows and result size and hue-2142. But nowhere noted abt browser specific issues. Is there something specific to browser .   CDH --> 5.8.3 Chrome --> 56.0.2924.87 Firfox -->51.0.1 (32-bit)     Traceback (most recent call last): File "/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 1215, in communicate req.respond() File "/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 576, in respond self._respond() File "/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 590, in _respond for chunk in response: File "/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/utils/six.py", line 535, in next return type(self).__next__(self) File "/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/http/response.py", line 292, in __next__ return self.make_bytes(next(self._iterator)) File "/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/lib/hue/desktop/core/src/desktop/lib/export_csvxls.py", line 96, in create_generator for _headers, _data in content_generator: File "/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/lib/hue/apps/beeswax/src/beeswax/data_export.py", line 106, in HS2DataAdapter results = db.fetch(handle, start_over=False, rows=fetch_size) File "/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/lib/hue/apps/beeswax/src/beeswax/server/dbms.py", line 282, in fetch return self.client.fetch(query_handle, start_over, rows) File "/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 1073, in fetch data_table = self._client.fetch_data(operationHandle, orientation=orientation, max_rows=max_rows) File "/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 799, in fetch_data results, schema = self.fetch_result(operation_handle, orientation, max_rows) File "/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 826, in fetch_result res = self.call(self._client.FetchResults, fetch_req) File "/opt/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 642, in call raise QueryServerException(Exception('Bad status for request %s:\n%s' % (req, res)), message=message) QueryServerException: Bad status for request TFetchResultsReq(fetchType=0, operationHandle=TOperationHandle(hasResultSet=True, modifiedRowCount=None, operationType=0, operationId=THandleIdentifier(secret='\xec\xb0f\xb8\xce\xe0H\x82\x90hf\xd39.\xcbg', guid='\x1d\xe5\x0e?\xf6\x9bF/\xbc\xa64\xca\x7f\xe0\x02\x04')), orientation=0, maxRows=1000): TFetchResultsResp(status=TStatus(errorCode=0, errorMessage='java.io.IOException: java.lang.NullPointerException', sqlState=None, infoMessages=['*org.apache.hive.service.cli.HiveSQLException:java.io.IOException: java.lang.NullPointerException:14:13', 'org.apache.hive.service.cli.operation.SQLOperation:getNextRowSet:SQLOperation.java:366', 'org.apache.hive.service.cli.operation.OperationManager:getOperationNextRowSet:OperationManager.java:277', 'org.apache.hive.service.cli.session.HiveSessionImpl:fetchResults:HiveSessionImpl.java:753', 'org.apache.hive.service.cli.CLIService:fetchResults:CLIService.java:438', 'org.apache.hive.service.cli.thrift.ThriftCLIService:FetchResults:ThriftCLIService.java:686', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1553', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1538', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor:process:HadoopThriftAuthBridge.java:746', 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:286', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1142', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:617', 'java.lang.Thread:run:Thread.java:745', '*java.io.IOException:java.lang.NullPointerException:16:2', 'org.apache.hadoop.hive.ql.exec.FetchTask:fetch:FetchTask.java:152', 'org.apache.hadoop.hive.ql.Driver:getResults:Driver.java:1798', 'org.apache.hive.service.cli.operation.SQLOperation:getNextRowSet:SQLOperation.java:361', '*java.lang.NullPointerException:null:18:2', 'org.apache.hadoop.hive.ql.exec.FetchOperator:pushRow:FetchOperator.java:425', 'org.apache.hadoop.hive.ql.exec.FetchOperator:pushRow:FetchOperator.java:417', 'org.apache.hadoop.hive.ql.exec.FetchTask:fetch:FetchTask.java:138'], statusCode=3), results=None, hasMoreRows=None)    ... View more

Hi,   Our set up has sentry enabled and recently added hive on spark as well.So after this it is very difficult to identify the application owner (always hive) and application query (always hive on spark)?   This is making life difficult in identifying and troubleshooting apps.   Any help on this or any alternate ... View more

We have a scenario where some third party will be loading xml files in one nas directory . There will around 100K files /day (all files are less than 1 mb)and they will be loaded randomly. Currently we are loading files in batches to HDFS (5000 /for each run/every hour) , using hadoop fs -put command. After loading it to HDFS we are doing transformation and loading it to Hive. It is taking lot of now.   What would the best alternate approches we can use to speed up things here? ... View more

When creating a new pool , is it mandatory to name the pool similar to AD group? If we want group_a members only to have submission control on that pool, is it mandatory the pool name also to be group_a?   Our requirement is to create a pool (with any name) and have specific groups members to have submission control privilege on that pool. is this possible?   This is how the placement rule looks like.     ... View more

Hi ,   I would like to set up alerts ( mail alert) for the below events . How to achieve this using cloudera manager?   Whenever a job fails. When a job is in Accepted state for more than 10 minutes. ... View more

Is it possible to set up alert when a new directory got created in HDFS? ... View more

I have few questions on granting access prvilege on hdfs uri as part of sentry integration (hive,hdfs).   1) Is it possible to grant read only privilege on hdfs uri? ( when i do grant select on hdfs uri it is giving all privilege (*) on the uri).    2) If i want to give full permission for a hive db on the hdfs uri do i need to specify  * at the end so that all the underlying tables will also have that permission ( grant all on uri 'hdfs://namespace/production/test_db/* to role ...) or is it ok if i grant only on the db path (  grant all on uri 'hdfs://namespace/production/test_db to role ... ), so that tables created under will also inherit this privileges?   ... View more