Solved

hi allwe have production cluster with HDP - 2.6.4 versionwe have 186 data-node machines ( DELL MACHINES WITH 10 disks )we try to re balance the data on the disks so disks will be with the same used size but without successwe feel that 2.6.4 version not have the tools that support re balance!!!as I mentioned on each data-node machine we have 10 disks while each disk is 1.8Tand some of the disks are 55% usedand some of them are only 1% usedso we have non balanced disks ( its like some disk are not useful ) , but why HDFS not balanced the data on all disks??my question - from which HDP version , we can re balance the data-node disks ?dose 2.6.5 version support re balance ?or from 3.X ?please advice , what we can do ?as I mentioned this is very huge cluster andwe get the bad feeling that the current HDP version ( 2.6.4 ) not support any re balance - is it true?example /dev/sdc                 3842878616 357409860 3485452372  10% /data_hdfs/sdc/d

Dear colleges and friendsI just reviewing on the HDP upgrade hortronworks documentation but not see the section about backup the folder - /var/hadoop/hdfs/namenode/currentis it necessary to backup the folder - current before HDP upgrade ?astar -zcvf current_bck.tar.gz /var/hadoop/hdfs/namenode/current-rw-r--r-- 1 root root 9062640 Jul 22 09:28 current_bck.tar.gz

Dear all , nice to be here , with all professional folkswe have ambari server , and all services shown from "Add Service Wizard"as click on actions button , and click on "+add Service"there we can found all available servicesso nowhow to get all these services list by API

Hi, I'm trying to deploy a few COORDINATOR_ONLY impalad on edge nodes in my cluster (CDH 5.16.1). HDFS short-circuit read are enabled on the cluster and works fine.When starting the COORDINATOR_ONLY nodes, I'm getting following error: Invalid short-circuit reads configuration: - Impala cannot read or execute the parent directory of dfs.domain.socket.pathAborting Impala Server startup due to improper configuration. Impalad exiting.Is there a way to run COORDINATOR_ONLY impalad with hdfs short circuit reads enabled. Thanks in advance!

hi allwe have small cluster ( ambari version - 2.6.1 )3 masters machines3 kafka machines 3 workers machinesis it possible to removed completely the HDP , for new fresh HDP version ?Now we have HDP - 2.6.5And we want to install new HDP - 2.6.4

We have a security requirement to be on TLS 1.2 by the end of this year, so we need to know is 1.2 compatible with the current version of CDH we are on 5.13. If not what is not supported ?And other part of my question is whether the TLS version 1.2 will run on CDH version 5.13 supporting all the applicable components/services ?Also we would like to confirm the Postgress of KTS supports TLS 1.2. ?

Ran into an interesting scenario in Director that I have not solved yet but will (hopefully) today.   Created a cluster via the python sdk and when attempting to repair a node in any instance group I get "A template must be specified". There is nothing in the logs and this instance group only contains gateway roles for HDFS and YARN.   This is Director v6.1 and there are no "templates" in the templates tab since everything was programmatically generated.   Screenshot for reference:

Hello all, I'm a little new in Haddop world, so, please forgive me any kind of questions that I'm doing.In first place, I installed a 7 node cluster in my environment, all of them has DataNode (I know now that's the worst scenario, but, it's already installed and no time to fix it).In this morning, a simple process that need to drop a partitioned table in Hive isn't working anymore, there was no modification on cluster.The simple command is "drop table ods.tims_declaracao;".Well, I think that I got the problem worse, I removed the directory "/apps/hive/warehouse/ods.db/tims_declaracao". I thought that I would drop the table, well, I realize that it was not.The table still hanging in drop the metadata. Can some good soul help me to find out what's the problem?Regards.

 Hi,   cloudera manager url is not opening. This is what I have checked so far. 1. All the services are running: Mysql, Cloudera-scm-agent, Cloudera-scm-server 2.  /etc/cloudera-scm-server/db.properties # Auto-generated by scm_prepare_database.sh on Wed Jul 10 05:13:24 EDT 2019 # # For information describing how to configure the Cloudera Manager Server # to connect to databases, see the "Cloudera Manager Installation Guide." # com.cloudera.cmf.db.type=mysql com.cloudera.cmf.db.host=servername com.cloudera.cmf.db.name=scm com.cloudera.cmf.db.user=scm com.cloudera.cmf.db.setupType=EXTERNAL com.cloudera.cmf.db.password=password 3.  /etc/my.cnf configuration:- [mysqld] datadir=/var/lib/mysql socket=/var/run/mysql/mysql.sock transaction-isolation = READ-COMMITTED # Disabling symbolic-links is recommended to prevent assorted security risks; # to do so, uncomment this line: symbolic-links = 0   key_buffer_size = 32M max_allowed_packet = 32M thread_stack = 256K thread_

Trying to access Ambari Files View, getting error when the View tries to load...As the error message is so terse, not sure what to do here for further debugging (new to HDP, but HDFS appears to be running normally and not sure which if any of the HDFS namenode logs would be relevant here). Any suggestions or fixes?

Often when provisioning clusters nodes will be cancelled due to Cloudera Director not being able to open more file handles:   [2019-07-23 17:31:18.585 +0000] ERROR [p-ebcce1c842e9-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.pipeline.ssh.SshJobFailFastWithOutputLogging - com.cloudera.launchpad.pipeline.util.PipelineRunner: Attempt to execute job failedjava.net.SocketException: Too many open files at java.net.Socket.createImpl(Socket.java:460) at java.net.Socket.connect(Socket.java:587) at net.schmizz.sshj.SocketClient.connect(SocketClient.java:126) at com.cloudera.launchpad.sshj.SshJClient.attemptConnection(SshJClient.java:343) at com.cloudera.launchpad.sshj.SshJClient.attemptConnection(SshJClient.java:318) at com.cloudera.launchpad.sshj.SshJClient.access$000(SshJClient.java:68) How to increase the file handles as ulimit and limits.conf do not seem to work?

I'm trying to install Cloudrea Manager. In the step of Install Agent, I got an error below:  Installation failed. Failed to receive heartbeat from agent.Ensure that the host's hostname is configured properly.Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).Ensure that ports 9000 and 9001 are not in use on the host being added.Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details).If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here. The log files in /var/log/cloudera-scm-agent/ shows that------------------------------------------------------------------------------------------------------------------------

I am trying to install Cloudera using VMware and docker image and after installtion I am not able to find the Cloudera Navigator and metadataserver running.Could you please quickly help on this.Version used:cloudera-quickstart-vm-5.13.0-0-beta-dockercloudera-quickstart-vm-5.13.0-0-vmware Thanks in advance. Regards,Rajesh

When using the Python SDK to create an InstanceTemplate if the bootstrap_scripts contain any special characters the API will reject it :     cloudera.director.common.rest.ApiException: (500)Reason: Server ErrorHTTP response headers: HTTPHeaderDict({'Connection': 'close', 'Content-Type': 'application/json;charset=utf-8'})HTTP response body: { "timestamp" : 1563826656470, "status" : 500, "error" : "Internal Server Error", "message" : "org.springframework.security.web.firewall.RequestRejectedException: The request was rejected because the URL contained a potentially malicious String \"%5C\"",   An example bootstrap script that fails is if you put:   wget https://www.google.com   Spring Security will reject this as a malicious URL since it is being encoded in the URL and not being put in the body it appears.   However, if you define the InstanceTemplate and don't create it in Director via the InstanceTemplatesAPI but rather just immediately pass it to the

Just wanted to bring up a concern about Director that it doesn't increase the non-java memory allocated to the host monitor (or at least not enough) when deploying a large amount of hosts to a deployment which will cause the host monitor to fail to start (due to CM specifying an error condition in 5.16.2). This doesn't fail the bootstrap (thankfully) but does require the user to go into the CM and manually set the recommendations for the Host Monitor before being able to start it and get metrics about the hosts.

I am trying to connect Java api to Hbase, I am using Ambari for Hbase and netbean for java. I am facing an error Exception in thread "main" java.lang.IllegalArgumentException: Not a host:port pair: PBUF#sandbox.hortonworks.com�}�����-.libraries of Hbase-client-0.95.0.jar and Hbase-server.0.95.0.jar have same version. please guide how can I resolve ? this is a code that I am using for connection. Configuration conf = HBaseConfiguration.create(); conf.set("hbase.master", "192.168.23.143:16020"); conf.set("hbase.zookeeper.quorum","192.168.23.143"); conf.set("hbase.zookeeper.property.clientPort", "2181"); conf.set("zookeeper.znode.parent", "/hbase-unsecure"); System.out.println("Connecting ..."); HBaseAdmin admin = new HBaseAdmin(conf); HBaseAdmin.checkHBaseAvailable(conf);System.out.println("Connecting ..."); boolean bool = admin.tableExists("transport"); System.out.println( bool); }Error:run:Connecting ...19/07/18 21:31:32 INFO zookeeper.ZooKeeper: Client environme

Hello,Whene executing a simple select on Ambari hive, i get "result fetch time out" error after one inute of execution.I follow the solution described on : https://community.hortonworks.com/articles/90768/how-to-fix-ambari-hive-view-15-result-fetch-timed.html but still facing the same error.I modified properties in "/etc/ambari-server/conf/ambari.properties" but after restarting the server i open this file and i can't see my modifications, is it normal ?Please can anyone help me.Thanks,Yassine

Good day. I`m now facing an issue converting Kafka`s message record of type long for nano-seconds (19 digits) to a string timestamp with milliseconds. The messages are coming in Avro format and contain different schemas (so we can`t statically define one schema) stored in Confluent Schema Registry. The current process is:1) ConsumeKafkaRecord_2_0 which reads the message and stores the Avro schema coming from Confluent Schema Registry into avro.schema attribute2) UpdateAttribute which is looking for a pattern of a timestamp record in the avro.schema and adds "logicalType":"timestamp-micros" (because i can`t find timestamp-nanos type in the Avro specification)3) ConvertRecord which converts the Avro flowfile using avro.schema into JSON. It uses the logicalType assigned in the previous step and converts the 19 digits long into the yyyy-MM-dd HH:mm:SS.SSSSSS. Here the issue is that 19 digits is a nano-timestamp type which is missing in Avro specification, so we only can use timestamp-micro

Performed the following steps to disable Kerberos Zookeeper -> enableSecurity (Enable Kerberos Authentication)-> false HDFS -> hadoop.security.authentication -> Simple HDFS -> hadoop.security.authorization -> false HDFS -> dfs.datanode.address -> from 1004 (for Kerberos) to default HDFS -> dfs.datanode.http.address  -> from 1006 (for Kerberos) to default HDFS -> Data Directory Permissions -> from 700 to 755 HBASE -> hbase.security.authentication -> Simple HBASE -> hbase.security.authorization -> false After doing so, all services started except Hbase.Found this in error logs Can't open /var/run/cloudera-scm-agent/process/758-hbase-REGIONSERVER/supervisor_status: Permission denied. How to solve this problem?

i have created acid orc partition table in hive. By using spark i insert the data to that table . in table pointing location its showing data like/warehouse/tablespace/managed/hive/testdb.db/table__2/full_date=2019-07-17/delta_0000000_0000000_0000/ 0000_0 .when i query the table it showing only column names can any one give me a solution

Assuming the latest version of Altus Director... are there any documentated recommendations for the number of deployments/clusters that Director can reasonbly be expected to manage?   Our clusters are less than 21 nodes each and our Director server is running on an EC2 instance with 8 cores and 60GB of RAM and I have a dedicated RDS MariaDB DB sized at 2 cores and 16GB of RAM with 200GB of storage.  Thus far I'm managing around 14 clusters from ranging in size from 9 to 20 nodes.  No issues thus far.  I'm just curious if there is general guidance around this subject.

Hi,we have troubles to connect MicroStrategy(=MSTR) to HIVE on HDP3.1 over Knox, when trying to connect we receive: Unknow failed: Unable to connect.Error type: System call error. Error communicating with connection process.Find target process dead while waiting for reply.We installed ClouderaHiveODBC-2.6.4.1004-1.x86_64.rpm driver.MSTR Inteligent Server is a RHEL-7.4 machine, installation has been realized according to Cloudera-ODBC-Driver-for-Apache-Hive-Install-Guide.pdf:yum --nogpgcheck localinstall [RPMFileName]As we are not sure that yum resolves all dependencies, we install the sasl libraries as well:cyrus-sasl   x86_64    2.1.26-23.el7    rhel-7-server-rpmsodbc.ini looks like:[hdp.hive.lxprod]Driver=/opt/cloudera/hiveodbc/lib/64/libclouderahiveodbc64.soHost=hdp.lxprodPORT=443Schema=defaultServiceDiscoveryMode=0HiveServerType=2AuthMech=3ThriftTransport=2HTTPPath=/gateway/DM/hiveSSL=1AllowSelfSignedServerCert=1CAIssuedCertNamesMismatch=1Tr

Is it possible to use Kerberos ticket to login into Cloudera Manager Admin Console?   I can only see the instruction for enabling http web-console for Hadoop roles but not for admin console: https://www.cloudera.com/documentation/enterprise/6/latest/topics/cm_sg_web_auth.html Step 9: (Optional) Enable Authentication for HTTP Web Consoles for Hadoop RolesAuthentication for access to the HDFS, MapReduce, and YARN roles' web consoles can be enabled using a configuration option for the appropriate service. To enable this authentication:From the Clusters tab, select the service (HDFS, MapReduce, or YARN) for which you want to enable authentication.Click the Configuration tab.Select Scope > service name Service-Wide.Select Category > Security.Type Enable Kerberos in the Search box.Select Enable Kerberos Authentication for HTTP Web-Consoles.Enter a Reason for change, and then click S

Hi, We are testing heavy write on kudu-1.8 shipped by CDH 6.1.1. Now we have around 350 "hot replicas" on one t-server. When we are ingesting the data, the memory consumed by each tablet keep on increasing and eventally the total used memory end up being exceeding the memory limit we give to kudu.From the mem-tracker, some replica's memrowset use as much as 500 MB,does that mean we need to provide up to 350 * 500 MB= 175 GB for each t-server?  Is there any way/config to limit/throttle the memory consumed by memrowset(The limit shows in mem-tradcker UI is "none", see below screenshot)? Any suggestion will be appreaciated, thanks!

Hi, we need to decommission a host that the HDFS "Balancer" role resides on.  I don't see a way to migrate that role in CM.  Will the decommission process automatically assign it to a different host? Regards,Michael