Solved

we installed small HDP cluster with one data-node machine HDP version is `2.6.5` and ambari version is `2.6.1` so this is new cluster that contain two name-node with only one data-node ( worker machine ) the interesting behavior that we see is that increasing of `under replica` on ambari dashboard , for now the number is `15000` under replica blocks as we know the most root cause of this problem is network issues between name node to data-node but this isn't the case in our hadoop cluster we can also decrease the under replica by the following proceduresu - <$hdfs_user>bash-4.1$ hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files-bash-4.1$ for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ; hadoop fs -setrep 3 $hdfsfile; done  but we not want to do it because under replica problem should not happens from beginning and maybe need to tune some HDFS paramet

Hi,  I'm not able to start the management service after the reboot of the manager node. I'm not sure what happened, everything was working fine until the reboot.  [21/Jan/2021 11:03:25 +0000] 15063 MainThread agent ERROR Heartbeating to ip-172-31-203-46.ec2.internal:7182 failed.Traceback (most recent call last):File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1399, in _send_heartbeatresponse = self.requestor.request('heartbeat', heartbeat_data)File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in requestreturn self.issue_request(call_request, message_name, request_datum)File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_requestcall_response = self.transceiver.transceive(call_request)File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceiveresult = self.read_framed_message()File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/

Hi All, I am trying to build JSON string using Ifelse condition in ReplaText processor but i am getting error as invalid Invalid Expression . Working Ifelse condition : ${status:contains('0'):ifElse('Success','Failed')} This below string is not working and it is throwing invalid expression . Please assist on this . ${status:contains('0'):ifElse('{"filename":"${overall_filename}","status":"${status}","message":"${overall_message}"}','{"filename":"${overall_filename}","status":"${exc_status}","message":"${overall_message}"}')} Thanks

Is Cloudera Manager an Open Source Software

We have a three node NiDi cluster with an F5 load balancer in front of it. The main flow start with a listenHTTP processor to wait for files from AWS. The F5 has a single pool that then sends files round robin to NiFi. How would I configure the flow to use NiFi load balancing instead of the F5 or can they be used together. What confuses me is I think I would send to a single NiFi node and the connection after ListenHTTP would load balance across the cluster. This is a single point of failure so I cannot see the benefit in NiFi load balancing. Can someone explain please Cheers.

Hi @ApacheNifi,i have i extracted the attributes from the filename and used AttributesToJson to create a json file to upload into elasticsearch (to create a catalog).It gives me many problems, what can I do?I also noticed that from updateattribute to attributetojson it adds "\ n" in two fields.This is my flow: UPDATEATTRIBUTE: I have correctly configured updateattribute and this is the result Setting:Result:ATTRIBUTESTOJSON:Configuration:Result: as you can notice in jsonattributes it adds a "\ n" at the end of the strings "giri" and "id-riduttore" fieldsPUTELASTICSEARCHHTTPRECORD:Configure:   Error:How can i do this? thank you so much 

Ranger service is failing to install with below error Error executing: CREATE FUNCTION `getXportalUIdByLoginId`(input_val VARCHAR(100)) RETURNS int(11) BEGIN DECLARE myid INT; SELECT x_portal_user.id into myid FROM x_portal_user WHERE x_portal_user.login_id = input_val; RETURN myid; ENDjava.sql.SQLException: This function has none of DETERMINISTIC, NO SQL, or READS SQL DATA in its declaration and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable)SQLException : SQL state: HY000 java.sql.SQLException: This function has none of DETERMINISTIC, NO SQL, or READS SQL DATA in its declaration and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable) ErrorCode: 14182021-01-19 10:13:39,544 [E] ranger_core_db_mysql.sql file import failed!

When I try to look at logs from command `yarn logs -applicationId xxx`, some errors raised: Response from the NodeManager:dp006:45454 WebService is not successful, HTTP error code: 500, Server response:Exception reading log file. Application submitted by 'xxx' doesn't own requested log file : syslogCan not get any logs for the log file: stderrResponse from the NodeManager:dp006:45454 WebService is not successful, HTTP error code: 500, Server response:Exception reading log file. Application submitted by 'xxx' doesn't own requested log file : stderrCan not get any logs for the log file: prelaunch.outResponse from the NodeManager:dp006:45454 WebService is not successful, HTTP error code: 500, Server response:Exception reading log file. Application submitted by 'xxx' doesn't own requested log file : prelaunch.outCan not get any logs for the log file: launch_container.sh The question is similar to https://community.cloudera.com/t5/Support-Questions/Cannot-read-log-files-on-ap

Hi there, I'm running hive with HDP 3.1.4, and with kerberos enabled.Currently there is a strange issue. Some sql could be success only if we run it with the system user `hive`, otherwise it fails with map error. A simple case is as below:1. Create a table named `test` with columns (id int, val string), and data [(1, '1'), (2, '2')].2. Run sql with user hive: `select * from test t join test t1 on t.id=t1.id;`, it succeeds as below:3. Run the same sql with some other user (e.g. test): `select * from test t join test t1 on t.id=t1.id;`, it fails with a vertex issue:```INFO : Compiling command(queryId=hive_20210119153558_48453ef9-4b06-48ed-b9f6-f64f82992d81): select * from test t join test t1 on t.id=t1.idINFO : Concurrency mode is disabled, not creating a lock managerINFO : Semantic Analysis Completed (retrial = false)INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:t.id, type:int, comment:null), FieldSchema(name:t.val, type:string, comment:null), FieldS

Hello,   I am new to CDH and excuse if its a simple or an easy mistake.   I have shell action in oozie which is failing to read a file. I have simple wrapper script, reading a file and passing the value in file to the next step, but its failing saying file or directory not exist. I could read/see the same file in server.   Really appreicate your help on this.    below is my script   #!/bin/bash #to test a sample sqoop script   while read line; do tablename=`echo $line | cut -d'.' -f2` sqoop import --connect "jdbc:sqlserver://ppppppp:1433;database=yyyy" --username userid --password '******' --table $tablename --target-dir hdfs://nameservice1/d/dl/rlsolutions/raw/{$tablename} -m 1 --as-avrodatafile done < /d/dl/rlsolutions/code/oozie-workflows/rl_solutions/hiveScripts/tables.txt   error: ./sample3.sh: line 10: /d/dl/rlsolutions/code/oozie-workflows/rl_solutions/hiveScripts/tables.txt: No such file or directory  

HiI want to upgrade our cluster to HDP3.However I can not access to "public" repo describe in doc (for instance https://archive.cloudera.com/p/ambari/centos7/2.x/updates/2.7.5.0/ambari.repo) because an authentication is now required.Does the community edition still exist or do we need to subscribe a paying subscription even if we do not need support ?Thx for the answer.Regards.

%sqlselect * from enrichedEvents order by hoursDriven desc limit 20 Table or view not found: enrichedEvents; line 1 pos 14set zeppelin.spark.sql.stacktrace = true to see full stacktrace Why this error coming in HDP 2.6.0 where no issues with HDP 3.0 sandbox. Any confugration changes in Zeppelin notebo

I am writing a script to remove a worker node. So I need Ambari API to uninstall the clients running on the node (ZOOKEEPER HDFS YARN MAPREDUCE2 TEZ SPARK2 HIVE)

Hello,  I've looked everywhere and can't seem to find any posts about resolving this issue. I set this same type of config in Google Cloud and it works fine. However, something is wrong with our configuration in our environment and I can't seem to figure it out but I think it's some type of bug.Any suggestions are appreciated. Here's the scenario:Overview:   We have a nifi cluster (2 nodes), we have enabled all of the Site-to-Site settings on both nifi nodes. # Site to Site propertiesnifi.remote.input.host=abcdnifi.remote.input.secure=falsenifi.remote.input.socket.port=1234nifi.remote.input.http.enabled=truenifi.remote.input.http.transaction.ttl=30 secnifi.remote.contents.cache.expiration=30 secs However, the minifi-app.log keeps getting the following errors when we try to start the flow.2020-12-14 19:19:19,595 INFO [main] o.apache.nifi.controller.FlowController Not enabling RAW Socket Site-to-Site functionality because nifi.remote.input.socket.port is not

I am writing a script to remove a worker node. So I need Ambari api to put the node in maintenance mode before removing host components and deleting the node from the cluster.

I use ExecuteSQLRecord to run a query and write to CSV format. The table has 10M rows. Although I can split the output into multiple flow files, the query is executed by only a single thread and is very slow.  Is there a way to partition the query into multiple queries so that the next processor can run multiple concurrent tasks, each one process one partition? It would be like:GenerateTableFetch -> ExecuteSQLRecord (with concurrent tasks)The problem is that GenerateTableFetch only accepts table name as input. It does not accept customized queries. Please advise if you have solutions. I am new to NiFi. So I would like your details. Thank you in advance.

Hi,I have HDP 2.6.5 running in VirtualBox with the networkmode "bridgenetwork".I have opened port 50010 in the sandbox vm and the port is accessible.Now I want to create a text file from my Windows machine with a Java programme. Configuration conf = new Configuration();conf.set("dfs.client.use.datanode.hostname", "true");FileSystem fs = new DistributedFileSystem();fs.initialize(new URI("hdfs://sandbox-hdp.hortonworks.com:8020"), conf);FSDataOutputStream fdos=fs.create(new Path("/testing/file02.txt"), true);fdos.writeBytes("Test text for the txt file");fdos.flush();fdos.close();fs.close(); The text file is also created, but there is nothing in it and I get the following error: 21/01/13 19:13:00 INFO hdfs.DFSClient: Exception in createBlockOutputStreamjava.io.EOFException: Premature EOF: no length prefix availableat org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2294)at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOu

Hello Experts, I have upgraded CDH 5.16 to CDP 7.2.4 Cloudera Manager only( not the runtime yet). After upgrade the Cloudera Manager server is running good including agent. But the Cloudera Agents on the rest of the nodes wont start when running $ sudo systemctl restart cloudera-scm-agentThe agents are still at 5.16 as the final step for upgrade does show the upgrade button for the agents. Before the upgrade the agents were running fine. Also the agents where shut down during the Cloudera Manager upgrade. How to start the agents and upgrade them to CDP 7.2.4 now. When trying to restart agents It just shows:$ sudo systemctl status cloudera-scm-agent● cloudera-scm-agent.service - LSB: Cloudera SCM AgentLoaded: loaded (/etc/rc.d/init.d/cloudera-scm-agent; bad; vendor preset: disabled)Active: active (exited) since Tue 2021-01-12 17:52:10 EST; 7s agoDocs: man:systemd-sysv-generator(8)Process: 14093 ExecStop=/etc/rc.d/init.d/cloudera-scm-agent stop (code=exited, status=0/SUCCE

I am new to NiFi, and I am trying to put some data into a postgresql database. I am trying to do it with the ConvertJSONtoSQL processor, unfortunately the JSON keyname and the columename of the table having not the same names (for example keyname = ServiceGroup and columnname = service_group).How can I map the key- and the columnname, so I can store the data in the tablename.

We want to deploy HDP 3.1.5 in production environmentWe have 3 server for masternode and 6 server for workernodeAnd we have plan component layout across 9 nodes above but we want to make sure where we need to place the service-client below 1. Yarn clientsFirst we've plan to install this to 9 nodes, does it okay or just install to 3 master nodes? Because as far as we know, yarn is needed for all nodes include resource managers and node managersOr is it just needed for launch yarn apps or anything else 2. Mapreduce2 clientsSame as above, we plan to install it to 9 nodes because it required for mapreduce jobsDo we need to install across 9 nodes? 3. Hive clientsWe've plan to install it to 3 master nodes, or we just need to install it to a master node?Is it just only needed for submit hive apps from beeline (cli)? 4. infra solr clientsWe just plan to install it to 9 nodes and we dont know enough to know how this client works 5. Kerberos clientsDoes all nodes need ke

HiWe are deploying on premise CDP 7.3 with CDSW 1.8 over RHEL 7.8 which comes with python 2.6 as default. Now we are planning to upgrade python to 3.6. Is there any specific steps I need to take in cluster config beside yum installation.  regards  

Hi All, I am new to solace and spark structured streaming. I would like to write a solace consumer using spark structured streaming and with the below code  var df = spark.readStream.format("solacestream").load I am getting java.lang.ClassNotFoundException: solacestream.DefaultSource Any hint of the format of the solace jms and how to pass in the connection parameters as option to the spark will be helpful. Thanks 

I want to obtain credentials by create support cacse in https://my.cloudera.com. But when i click the "Support caces" in https://my.cloudera.com/support.html , the next page show something : Access to page is restrictedAccessing the requested page requires special permissions. Someone can help me, thanks.

Hello Cloudera support. Is it possible to change the execution engine for hive in HDP-3.0.1.0 to spark?   Thank you,Amit  

Hello Community!Hope you're all doing fine. I'm beggining my Nifi studying and i'de like to replicate a process that i have running in other plataforms.My goal is to read a CSV and process it row by row to insert into a sql server table. So far so good, i was able to do this with the following: GetFile -> SplitRecord -> ConvertRecord (to json) -> ConvertJSONtoSQL -> UpdateAttribute (date conversion) -> PutSQL. But i need to validade if the record already exists in the destiny table. Can anyone help me with that? Already looked for but i wasn't able to find somethig like it. If you guys have a solution that is better and completely different, no problem, i'm open to any change. Thanks in advance!