Q&A

Hello Team,We are using Impala to query data stored as parquet on s3. This has been an awesome feature. Recently Amazon S3 has announced a new feature called S3 Select  which helps in speeding up the column projection when querying data stored on S3. As of now Hive and Presto support  S3 select push down.Is Impala going to support S3 select push downs? Parth

Hello All, We are running Spark jobs via yarn and its failing with the below error, any help / pointer to fix is much appericated. Shell output: main : command provided 1main : run as user is TEST1main : requested yarn user is TEST1Writing to tmp file /data/8/yarn/nm/nmPrivate/application_1587389136999_0013/container_e56_1587389136999_0013_01_000477/container_e56_1587389136999_0013_01_000477.pid.tmpWriting to cgroup task files...Container exited with a non-zero exit code 1org.apache.spark.rpc.RpcTimeoutException: Cannot receive any reply in 120 seconds. This timeout is controlled by spark.rpc.askTimeout at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48) at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63) at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunc

Hi Cloudera Community,   I have just completed installation of CDSW 1.6.1 in my DEV cluster. I just want to know the initial "admin" account credentials.   I have tried below listed username/password combinations so far, but no luck: 1. admin/admin 2. cloudera/cloudera 3.admin/cloudera   Regards, Nanda    

Hi , From Ambari 2.7.5 documentation, one of the prerequisites mentioned is to set HOSTNAME in /etc/sysconfig/network , but as per RHEL 7, /etc/sysconfig/network file is deprecated. What is your recommendation on this? Thanks,Satish.

Dear Cloudera Community , I want to learn about cloudera data warehouse. I try to install to trial version, I have some questions : How to Install the CDP Trial Version ?What is the Hardware Requirement including the requiment of space, cpu dan memory ? What is the Operating System Requirement ?  Regards,  BSST

Hi,I'm trying to update the PostgreSQL from 9.2.24 to 9.6. I tried the followingTook backup of the DB with following command.pg_dumpall > outputfileOnce the backup is done I stopped the service and Rename the pgsql directory under /var/lib/pgsql/Updated the postgres with the following command :yum install postgresql96-server postgresql96.After the installation the new directory got created /var/lib/pgsql/9.6/. I ran the initdb command to create the cluster. Restored the old  pg_hba.conf and postgresql.conf files from the backup directory. Started the postgres service with the following command:/usr/pgsql-9.6/bin/pg_ctl start -D /var/lib/pgsql/9.6/dataOnce the service was up I restored the DB with the following command:/usr/pgsql-9.6/bin/psql -d postgres -f outputfileThe DB restoration done but still while starting the ambari server  I'm getting the error  Ident authentication failed for user "ambari" I have the doubt that is the upgrade properly done I mean the a

Hi All,My services are not starting and it failed after sometimes. CDH : 6.2.0Error:AttributeError: 'NoneType' object has no attribute 'get'[16/Jul/2019 10:31:20 +0000] 6528 MainThread agent ERROR Caught unexpected exception in main loop.Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 752, in __issue_heartbeat heartbeat = self.prepare_heartbeat() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1295, in prepare_heartbeat process.update_supervisor_data(supervisor_data.get(process.name, None)) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/process.py", line 1282, in update_supervisor_data self.deactivate_complete() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/process.py", line 974, in deactivate_complete LOG.info("[%s] Unregistered supervisor process %s", self, self.fixed_supervisor_state()) File "/opt/cloudera/cm-agent/lib/python2.7/si

I had question to write hive script to produce report. But location/file_name details were missing for the report.I wrote query to export report on HDFS location. In test result I got Incorrect solution SQL comment.Please let me know what is the expected result for this type of question.Thanks. 

-- I adapted from this example www.cloudera.com/documentation-- impala sideCREATE TABLE flat_struct_array (continent STRING, country STRING, city STRING);INSERT INTO flat_struct_array VALUES ('North America', 'Canada', 'Toronto') , ('North America', 'Canada', 'Vancouver') , ('North America', 'Canada', "St. John\'s") , ('North America', 'Canada', 'Saint John') , ('North America', 'Canada', 'Montreal') , ('North America', 'Canada', 'Halifax') , ('North America', 'Canada', 'Winnipeg') , ('North America', 'Canada', 'Calgary') , ('North America', 'Canada', 'Saskatoon') , ('North America', 'Canada', 'Ottawa') , ('North America', 'Canada', 'Yellowknife') , ('Europe', 'France', 'Paris') , ('Europe', 'France', 'Nice') , ('Europe', 'France', 'Marseilles') , ('Europe', 'France', 'Cannes') , ('Europe', 'Greece', 'Athens') , ('Europe', 'Greece', 'Piraeus') , ('Europe', 'Greece', 'Hania') , ('Europe', 'Greece', 'Heraklion') , ('Europe', 'Greece', 'Rethymnon') , ('Europe', 'Greece', 'Fir

Does Sentry have data in rest and data in motion encryption? or only HDFS and Cloudera Navigator have these features?

I am using CM 5.14.2, and from hive's configuration page, i can see the value for javax.jdo.option.ConnectionUserName, which is used to access the underneath metastore db like mysql, but i can't see the value of javax.jdo.option.ConnectionPassword (which is display as **** in the page). i have forgotten the password used to access hive metastore db. Is there any way to find it back?

Hi there, I added Atlas in my CDP cluster, and find can't connect the web UI. In the application.log, I found the error: Can not find the specified config set: vertex_index.I followed the instruction in the following links, but still couldn't get it to work. Any suggestions?https://issues.apache.org/jira/browse/AMBARI-18368https://community.cloudera.com/t5/Support-Questions/Atlas-UI-not-available-Service-Unavailable/td-p/132435 Here is the application.log:                              Atlas Server (STARTUP) project.name: apache-atlasproject.description: Metadata Management and Data Governance Platform over Hadoopbuild.user: jenkinsbuild.epoch: 1574150696506project.version: 2.0.0.7.0.3.0-79build.version: 2.0.0.7.0.3.0-79vc.revision: cbb309de67ee0215efa0763f03c2f32f40f1d8f3vc.source.url: scm:git:git://git.apache.org/atlas.git/atlas-webapp####################################################

Greetings, After enabling "Kerberos Authentication for HTTP Web-Consoles" for YARN the Resource Manager WebUI and the HistoryServer Web UI become inaccessible with a valid Kerberos ticket (without a ticket the UI correctly gives the "Authentication required" HTTP 401 error message). Navigating to either of the interfaces returns the following error:       HTTP ERROR 500Problem accessing /jobhistory. Reason: Server ErrorCaused by:java.lang.IllegalArgumentException: Empty key at javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:96) at org.apache.hadoop.security.authentication.util.Signer.computeSignature(Signer.java:93) at org.apache.hadoop.security.authentication.util.Signer.sign(Signer.java:59) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:587) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doF

Hello All,     I've installed CM manager 6.3.0 and agents in my 7 node cluster. While setting up the CDH 6.3.1 installation, i'm stuck at setup database step, where the test connection is running for more than 2 hours, with only info found in Cloudera manager log given below. There is no error or any other information. I can successfully connect to my databases from command line. But CM is unable to verify the connections. Any help will be appreciated.   2019-12-11 21:48:41,016 INFO scm-web-93:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/testConnection2019-12-11 21:48:41,036 INFO scm-web-93:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/testConnection, Status:2002019-12-11 21:48:43,071 INFO scm-web-93:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult2019-12-11 21:48:43,081 INFO scm-web-93:com.cloudera.en

I'm using CDH 6.3.2. When trying to create directories in hdfs, I'm having permission denied like this one:[devuser@master01 ~]$ hdfs dfs -mkdir /devmkdir: Permission denied: user=devuser, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-xHow can I add my user on hdfs? and how can my user run jobs as well?

Hi,  New to Cloudera search.   How do I get the Solr.xml from zookeeper using solrctl tool or any other tools Basically want to see the entire configuration for Solr service on the cluster   migrated cluster from 5.14 to 6.2 Cluster Info: CDH 6.2 Lucene 7.4.0        

I had the cloudera manager(cms) , management service (mgmt) and the database(mariadb) running on a single VM. In order to enable HA for cms, I separated the mgmt and database to independent servers. I shutdown scm-agent on the cms server and have it running on the mgmt server. I also deployed the HAproxy (v1.8) for load balancing as advised in the documentation. I migrated the databases from the current cms server to the new DB server. Ref# https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_cm_ha_hosts.html   The DB and NFS mounts for the management service directories are served by a single host. After following exactly as per the doc, management roles applied starts up fine in the UI without errors, however the status is always "Unknown Health". To the top it shows this message: The status of the services are "?" The logs on the management service VM has the below errors: cloudera-scm-agent LOG Errors:   Spoiler (Highlight to read)[30/Mar/2020 16:1

Hi All,  Downloaded ambari 2.7.0.0 from http://public-repo-1.hortonworks.com/ambari/sles12/2.x/updates/2.7.0.0/ambari.repo    Downloaded the tarballs from https://docs.cloudera.com/HDPDocuments/Ambari-2.7.1.0/bk_ambari-installation/content/hdp_30_repositories.html   Using Local Repository. Using postgresql for ambari, druid, hive, oozie, ranger and rangerkms.  SLES 12 SP5   In the Install phase, i get to Hive Client Install, it fails with this error: Traceback (most recent call last):File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_client.py", line 60, in <module>HiveClient().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in executemethod(env)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_client.py", line 39, in installself.install_packages(env)File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/scrip

I am encountering this error when I'm executing this command : /opt/cloudera/cm/schema/scm_prepare_database.sh mysql cloudera_manager_db cloudera_db Enter SCM password:JAVA_HOME=/usr/lib/jvm/java-openjdkVerifying that we can write to /etc/cloudera-scm-serverCreating SCM configuration file in /etc/cloudera-scm-serverExecuting: /usr/lib/jvm/java-openjdk/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.[ main] DbCommandExecutor ERROR Error when connecting to database.java.sql.SQLException: The server time zone value 'CDT' is unrecog

Hi guys   I just install a fresh cluster with CDP 7.0.3 (7.0.3-1.cdh7.0.3.p0.1635019) in "Trial period" to practice and probe the environment...   The cluster has 3 masters and 5 workers and only has HDFS (no HA mode), Hive, Impala, Zookeeper, Spark and Yarn services installed with integrated postgre database, no ldap, no tls...   All works fine except HIVESERVER2 that doesn't start... i try to debug the error but i can't   The messages from /var/log/hive/hadoop-cmf-hive-HIVESERVER2-hostname.log     2020-03-13 13:15:54,948 INFO org.apache.hadoop.hive.conf.HiveConf: [main]: Found configuration file file:/run/cloudera-scm-agent/process/1546333137-hive-HIVESERVER2/hive-site.xml2020-03-13 13:15:55,681 WARN org.apache.hadoop.hive.conf.HiveConf: [main]: HiveConf of name hive.query.redaction.rules does not exist2020-03-13 13:16:02,898 INFO org.apache.hadoop.hive.conf.HiveConf: [main]: Found configuration file file:/run/cloudera-scm-agent/process/154633313

Greetings,   I am looking for a way to enforce password complexity requirements, mandatory rotation of passwords and mitigation of brute force password cracking attacks on Cloudera Manager (CM) accounts.   I was researching user authentication options in the free version of CM and came to the conclusion that the only available option is Kerberos and SPNEGO, paired with Kerberos password policies.   In CM I enabled "Enable SPNEGO/Kerberos Authentication for the Admin Console and API", but I didn't notice anything different after restarting the service.   Are there other options (for the free version of CM 6.3) that I am missing? What are the options in the paid version?   Thank you, Kind regards, Julius

Hi, We are running Hue as part of CDH 6.2 (and currently upgrading to 6.3.3). We have an AD group - hue-users - which is the base_dn in the CDH Hue config. This group controls basic access to Hue. I would like to add another AD group. How do I do this? For context, this is so that we can split permissions between groups. One group (hue-users) is for user access (all modules). The other is for unattended service accounts, so we only want them be able to access/schedule/run Oozie jobs. Edit: they're all in the same AD realm, although not necessarily the same OU path. My current config is (the numbers are for reading nested AD groups)(memberOf:1.2.840.113556.1.4.1941:=CN=hue-users,OU=x,DC=x,DC=x,DC=x) I've tried (with various brackets etc):((memberOf:1.2.840.113556.1.4.1941:=CN=hue-users,OU=x,DC=x,DC=x,DC=x) || (memberOf:1.2.840.113556.1.4.1941:=CN=service-users,OU=x,DC=x,DC=x,DC=x)) (|(memberOf:1.2.840.113556.1.4.1941:=CN=hue-users,OU=x,DC=x,DC=x,D

Hi,Am getting below error from Ambari UI i.e., while starting or stopping or restarting the components present in Hadoop cluster. Could any one help please ______________________________________________________________________________Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.4.0.v20120608-r11652): org.eclipse.persistence.exceptions.DatabaseExceptionInternal Exception: java.sql.BatchUpdateException: Batch entry 0 INSERT INTO request (request_id, cluster_id, command_name, create_time, end_time, exclusive_execution, inputs, request_context, request_type, start_time, status, request_schedule_id) VALUES (820, 2, NULL, 1587733705827, -1, 0, NULL, 'Start History Server', 'INTERNAL_REQUEST', -1, NULL, NULL) was aborted. Call getNextException to see the cause.Error Code: 0Call: INSERT INTO request (request_id, cluster_id, command_name, create_time, end_time, exclusive_execution, inputs, request_context, request_type, start_time, status, request_schedule_id)

After a few months, my cluster is working properly. However, yesterday I've noticed that one of the services went down. I've tried to restart the service but it won't start. The action stuck in the background UI window seen in the image. I've tried to restart some services. However, the same issue encountered with the other service. I am thinking that there is something with Ambari UI sending the command to the cluster. Any suggestions would be helpful for me. Thank youI've already restarted the ambari-server and ambari-agent. H

I'm trying to discern what services will be available to us when we upgrade from CDH 5.16.2 to CDP-DC with CDF. From what I can tell, we should have access to any components part of CDF, as well as anything in the Cloudera Runtime. The thing is, I'm seeing some discrepancies.   In the maven artifacts for Cloudera Runtime 7.0.3, it mentions a lot of services, including Druid and Zeppelin: https://docs.cloudera.com/cdpdc/7.0/release-guide/topics/cdpdc-runtime-maven-703.html   But in the component versions list of Cloudera Runtime 7.0.3, Druid and Zeppelin are both not included: https://docs.cloudera.com/runtime/7.0.3/release-notes/topics/rt-runtime-component-versions.html   Additionally, in the documentation for CDF, I don't see Druid listed, even though in the past HDF has had Druid included: https://www.cloudera.com/content/dam/www/marketing/images/diagrams/cdf-diagram.png    Can someone verify for me whether Druid and/or Zeppelin is available for use by users