Q&A

I'm trying to install Cloudrea Manager. In the step of Install Agent, I got an error below:  Installation failed. Failed to receive heartbeat from agent.Ensure that the host's hostname is configured properly.Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).Ensure that ports 9000 and 9001 are not in use on the host being added.Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details).If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here. The log files in /var/log/cloudera-scm-agent/ shows that------------------------------------------------------------------------------------------------------------------------

I am trying to install Cloudera using VMware and docker image and after installtion I am not able to find the Cloudera Navigator and metadataserver running.Could you please quickly help on this.Version used:cloudera-quickstart-vm-5.13.0-0-beta-dockercloudera-quickstart-vm-5.13.0-0-vmware Thanks in advance. Regards,Rajesh

When using the Python SDK to create an InstanceTemplate if the bootstrap_scripts contain any special characters the API will reject it :     cloudera.director.common.rest.ApiException: (500)Reason: Server ErrorHTTP response headers: HTTPHeaderDict({'Connection': 'close', 'Content-Type': 'application/json;charset=utf-8'})HTTP response body: { "timestamp" : 1563826656470, "status" : 500, "error" : "Internal Server Error", "message" : "org.springframework.security.web.firewall.RequestRejectedException: The request was rejected because the URL contained a potentially malicious String \"%5C\"",   An example bootstrap script that fails is if you put:   wget https://www.google.com   Spring Security will reject this as a malicious URL since it is being encoded in the URL and not being put in the body it appears.   However, if you define the InstanceTemplate and don't create it in Director via the InstanceTemplatesAPI but rather just immediately pass it to the

Just wanted to bring up a concern about Director that it doesn't increase the non-java memory allocated to the host monitor (or at least not enough) when deploying a large amount of hosts to a deployment which will cause the host monitor to fail to start (due to CM specifying an error condition in 5.16.2). This doesn't fail the bootstrap (thankfully) but does require the user to go into the CM and manually set the recommendations for the Host Monitor before being able to start it and get metrics about the hosts.

I am trying to connect Java api to Hbase, I am using Ambari for Hbase and netbean for java. I am facing an error Exception in thread "main" java.lang.IllegalArgumentException: Not a host:port pair: PBUF#sandbox.hortonworks.com�}�����-.libraries of Hbase-client-0.95.0.jar and Hbase-server.0.95.0.jar have same version. please guide how can I resolve ? this is a code that I am using for connection. Configuration conf = HBaseConfiguration.create(); conf.set("hbase.master", "192.168.23.143:16020"); conf.set("hbase.zookeeper.quorum","192.168.23.143"); conf.set("hbase.zookeeper.property.clientPort", "2181"); conf.set("zookeeper.znode.parent", "/hbase-unsecure"); System.out.println("Connecting ..."); HBaseAdmin admin = new HBaseAdmin(conf); HBaseAdmin.checkHBaseAvailable(conf);System.out.println("Connecting ..."); boolean bool = admin.tableExists("transport"); System.out.println( bool); }Error:run:Connecting ...19/07/18 21:31:32 INFO zookeeper.ZooKeeper: Client environme

Hello,Whene executing a simple select on Ambari hive, i get "result fetch time out" error after one inute of execution.I follow the solution described on : https://community.hortonworks.com/articles/90768/how-to-fix-ambari-hive-view-15-result-fetch-timed.html but still facing the same error.I modified properties in "/etc/ambari-server/conf/ambari.properties" but after restarting the server i open this file and i can't see my modifications, is it normal ?Please can anyone help me.Thanks,Yassine

Good day. I`m now facing an issue converting Kafka`s message record of type long for nano-seconds (19 digits) to a string timestamp with milliseconds. The messages are coming in Avro format and contain different schemas (so we can`t statically define one schema) stored in Confluent Schema Registry. The current process is:1) ConsumeKafkaRecord_2_0 which reads the message and stores the Avro schema coming from Confluent Schema Registry into avro.schema attribute2) UpdateAttribute which is looking for a pattern of a timestamp record in the avro.schema and adds "logicalType":"timestamp-micros" (because i can`t find timestamp-nanos type in the Avro specification)3) ConvertRecord which converts the Avro flowfile using avro.schema into JSON. It uses the logicalType assigned in the previous step and converts the 19 digits long into the yyyy-MM-dd HH:mm:SS.SSSSSS. Here the issue is that 19 digits is a nano-timestamp type which is missing in Avro specification, so we only can use timestamp-micro

Performed the following steps to disable Kerberos Zookeeper -> enableSecurity (Enable Kerberos Authentication)-> false HDFS -> hadoop.security.authentication -> Simple HDFS -> hadoop.security.authorization -> false HDFS -> dfs.datanode.address -> from 1004 (for Kerberos) to default HDFS -> dfs.datanode.http.address  -> from 1006 (for Kerberos) to default HDFS -> Data Directory Permissions -> from 700 to 755 HBASE -> hbase.security.authentication -> Simple HBASE -> hbase.security.authorization -> false After doing so, all services started except Hbase.Found this in error logs Can't open /var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/supervisor_status: Permission denied. How to solve this problem?

i have created acid orc partition table in hive. By using spark i insert the data to that table . in table pointing location its showing data like/warehouse/tablespace/managed/hive/testdb.db/table__2/full_date=2019-07-17/delta_0000000_0000000_0000/ 0000_0 .when i query the table it showing only column names can any one give me a solution

Assuming the latest version of Altus Director... are there any documentated recommendations for the number of deployments/clusters that Director can reasonbly be expected to manage?   Our clusters are less than 21 nodes each and our Director server is running on an EC2 instance with 8 cores and 60GB of RAM and I have a dedicated RDS MariaDB DB sized at 2 cores and 16GB of RAM with 200GB of storage.  Thus far I'm managing around 14 clusters from ranging in size from 9 to 20 nodes.  No issues thus far.  I'm just curious if there is general guidance around this subject.

Hi,we have troubles to connect MicroStrategy(=MSTR) to HIVE on HDP3.1 over Knox, when trying to connect we receive: Unknow failed: Unable to connect.Error type: System call error. Error communicating with connection process.Find target process dead while waiting for reply.We installed ClouderaHiveODBC-2.6.4.1004-1.x86_64.rpm driver.MSTR Inteligent Server is a RHEL-7.4 machine, installation has been realized according to Cloudera-ODBC-Driver-for-Apache-Hive-Install-Guide.pdf:yum --nogpgcheck localinstall [RPMFileName]As we are not sure that yum resolves all dependencies, we install the sasl libraries as well:cyrus-sasl   x86_64    2.1.26-23.el7    rhel-7-server-rpmsodbc.ini looks like:[hdp.hive.lxprod]Driver=/opt/cloudera/hiveodbc/lib/64/libclouderahiveodbc64.soHost=hdp.lxprodPORT=443Schema=defaultServiceDiscoveryMode=0HiveServerType=2AuthMech=3ThriftTransport=2HTTPPath=/gateway/DM/hiveSSL=1AllowSelfSignedServerCert=1CAIssuedCertNamesMismatch=1Tr

Is it possible to use Kerberos ticket to login into Cloudera Manager Admin Console?   I can only see the instruction for enabling http web-console for Hadoop roles but not for admin console: https://www.cloudera.com/documentation/enterprise/6/latest/topics/cm_sg_web_auth.html Step 9: (Optional) Enable Authentication for HTTP Web Consoles for Hadoop RolesAuthentication for access to the HDFS, MapReduce, and YARN roles' web consoles can be enabled using a configuration option for the appropriate service. To enable this authentication:From the Clusters tab, select the service (HDFS, MapReduce, or YARN) for which you want to enable authentication.Click the Configuration tab.Select Scope > service name Service-Wide.Select Category > Security.Type Enable Kerberos in the Search box.Select Enable Kerberos Authentication for HTTP Web-Consoles.Enter a Reason for change, and then click S

Hi, We are testing heavy write on kudu-1.8 shipped by CDH 6.1.1. Now we have around 350 "hot replicas" on one t-server. When we are ingesting the data, the memory consumed by each tablet keep on increasing and eventally the total used memory end up being exceeding the memory limit we give to kudu.From the mem-tracker, some replica's memrowset use as much as 500 MB,does that mean we need to provide up to 350 * 500 MB= 175 GB for each t-server?  Is there any way/config to limit/throttle the memory consumed by memrowset(The limit shows in mem-tradcker UI is "none", see below screenshot)? Any suggestion will be appreaciated, thanks!

Hi, we need to decommission a host that the HDFS "Balancer" role resides on.  I don't see a way to migrate that role in CM.  Will the decommission process automatically assign it to a different host? Regards,Michael

hi allunder /var/log/zookeeper , we have the zookeeper logs , as the following-rw-r--r-- 1 zookeeper hadoop 104857712 Jul  5 15:54 zookeeper-zookeeper-server-master.sys87.com.log.1-rw-r--r-- 1 zookeeper hadoop       177 Jul 16 11:57 zookeeper-zookeeper-server-master.sys87.com.out-rw-r--r-- 1 zookeeper hadoop  66358998 Jul 16 23:15 zookeeper-zookeeper-server-master.sys87.com.logthe problem is that we get only the INFO severity from the logs!and when we set the TRCE severity , and restart the zookeeper service , then its not change the zookeeper log content to trace mode,can we get example of log4j for zookeeper ?we suspect that our log4j isn't configured properly

Hello everyone.I would really appreciate some help/advice with this.I run HDP Sandbox 3.1 on in a VM with 32GB memory, and plenty of disk space, for educational purposes. However, as soon as there more than two or three logged in users running Hive queries, the jobs get stuck in the queue, with one job listed as "Running", and everything else as "Accepted". And those are not even big jobs -- simply some exercises from HDP Sandbox tutorials.Can anyone suggest, is it possible to run HDP Sandbox with 25 to 50 concurrent users, working on some basic exercises from HDP Sandbox tutorials? What would I need to change in the configs for that? Many thanks!

Everytime I try to restart clean restart fails but nifi but nifi will be running. Also tried restarting from cli same issue. Because of this I am unable to apply any conf changes from Ambari. Attached screenshot of the exception. It reads asstderr: Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 982, in restart self.status(env) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 156, in status check_process_status(status_params.nifi_node_pid_file) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status raise ComponentIsNotRunning() ComponentIsNotRunning The above exception was the cause of the following exception: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 278, in Master().execute() File "/usr/lib/amb

I have enable stats using `COMPUTE STATS` on a table. After this when I re-run the query, it's giving me an error `A RIGHT OUTER JOIN type with no equi-join predicates can only be executed with a single node plan`. Dropping the stats is execuring the query. I'm trying to improve the query performance.

On a perfectly running HDP3.1 cluster, one time the ATSv2 Timeline Reader stopped responding. Following some guide I've tried refreshing the service by cleaning the configs and restarting it byyarn app -destroy ats-hbaseAfter this, restarting the service doesn't help, and starting the service by yarn app -start ats-hbasedoesn't work, since the config json is missing:ERROR client.ApiServiceClient: File does not exist:  <>/user/yarn-ats/.yarn/services/ats-hbase/ats-hbase.jsonWhat is this .json and how do I get it running again? Where are some example templates of a working .json that I could use? I remember this file existing somewhere on a fresh install, but sadly reinstall of the cluster is not an option. Another thing, trying a fresh install on my laptop VM does not create such json file too -- I guess the reason is that the laptop is too small for enforcing hbase use?Didn't find anything in the official documentation, so would be thankful for any advice!@Geoffrey Shelton O

I'm using Altus Director 6.2.    Is the following declaration in your latest Aluts Director documentation still accurate?   Changing the Instance Type is Not Supported in Azure Changing the instance type of an already-deployed VM is not supported in Azure. Changing the instance type of a VM through a tool external to Altus Director is not supported in Azure. You cannot, for example, use the Azure Portal to change the instance type. Altus Director is not updated when instance type changes are made with external tools, and your cluster will show errors in Altus Director. I have performed both of the actions described in the bullet points above after we decided our existing workers were not sized properly for the amount of data being ingested / processed.   Cloudera Director still shows the cluster in a "green" state.  No complaints / errors / warnings thus far.   Is this really not supported?   Reference: https://www.cloudera.com/documen

Hi Team, How to perform TPC-DS test/workloads on hive,impala,spark? how to analyze the results?  Thanks,Harish

Hello team,   Cloudera Navigator Metadata Server failed to start, no log file getting updated in /var/log/cloudera-scm-navigator for metaddata server.   Only Cloudera server shows below errors.   CDH version: 5.16.1   Kindly suggest?   Error:   2019-06-20 13:49:21,099 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Start this Navigator Metadata Server on role Navigator Metadata Server (a301-8744-7259)2019-06-20 13:49:21,127 INFO CommandPusher:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing role command Start BasicCmdArgs{args=[]}. Service: DbService{id=2, name=mgmt} Role: DbRole{id=324, name=mgmt-NAVIGATORMETASERVER-220e59e0745b944070fe360a2cc717c7, hostName=a301-8744-7259.ldn.swissbank.com}2019-06-20 13:49:21,127 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence2019-06-20 13:49:21,127 INFO CommandPusher:com.cloudera.cmf.command.flow.Cmd

Hello,   I need to improve speed for my select query currently I am testing how best I can get from Impala with given hardware.   I have created a parquet table with partitions and block size = 512mb. Here is some more details -   [quickstart.cloudera:21000] > show table stats tbl_parq_512m;Query: show table stats tbl_parq_512m+-----------+----------------+------+------------+-----------+--------+----------+--------------+-------------------+---------+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+| source_ip | destination_ip | year | event_date | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location |+-----------+----------------+------+------------+-----------+-------

Hi, I have a table with a lot of data,I want to create a new table based on some column values from this based which method is most efficient and cluster resources friendly Pseudo-Code 1. single job        insert into myNewTable         select * from myOldTable         where a=xxx etc. 2. two jobs:    job1. create datafame from select statement               select * from myOldTable         where a=xxx etc. as dataframe     job2 write dataframe as new table         insert into myNewTable select from dataframe

We have failed to start ambari-metrics-collector.The following error appeared in hbase-ams-master.log.I can not find another ERROR, what should I check?----------------/var/log/ambari-metrics-collector/hbase-ams-master-host.log2019-07-11 15: 34: 41,040 ERROR [main] master.HMasterCommandLine: Master exitingjava.lang.RuntimeException: Master not initialized after 200000ms at org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent (JVMClusterUtil.java: 229) at org.apache.hadoop.hbase.util.JVMClusterUtil.startup (JVMClusterUtil.java: 197) at org.apache.hadoop.hbase.LocalHBaseCluster.startup (LocalHBaseCluster.java:413) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster (HMasterCommandLine.java: 232) at org.apache.hadoop.hbase.master.HMasterCommandLine.run (HMasterCommandLine.java: 140) at org.apache.hadoop.util.ToolRunner.run (ToolRunner.java: 76) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain (ServerCommandLine.java