Find answers, ask questions, and share your expertise
Recently active
Hello all, I have a problem with connection from hue to hive and hdfs. For hive i have this error : HTTPSConnectionPool(host='fqdn', port=10001): Max retries exceeded with url: /cliservice/ (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),)) for hdfs : Accès impossible : /user/svc_data.HTTPSConnectionPool(host='fqdn', port=14000): Max retries exceeded with url: /webhdfs/v1/user/svc_dataiku?op=GETFILESTATUS&user.name=hue&doas=svc_dataiku (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),)) Need help please. Regards
Hi Team, I'm trying to merge two files(.pdf format) and combine it to one file using execute streamcommand using jar file. at java side the merge is working using the same jar. but nifi side the command is not working , instead of merging files it pushing both individual files to outputstream and throwing the below error at execute streamcommand processor."Failed to write flow file to stdin due to broken pipe:java.io.IOException:Broken pipe"Below is flow which i'm using and the command Arguements which i'm using is-jar;pdfmerge-0.0.1-SNAPSHOT.jar;/opt/nifi/nifi-current/temppdf/${filename:startsWith('invoice_')};/opt/nifi/nifi-current/temppdf/${filename:startsWith('QRBill_')};/opt/nifi/nifi-current/temp2/Output.pdf and here is my configuration of executeStreamCommand:Could you please help me on this?
Hello All, I have just added a Ranger KMS service to a newly built CDP Cluster. When attempted to create key, got error stating "user not allowed to do create key". I thought it must be related to providing privs to the user in Ranger policies for KMS service and hence, logged in to the Ranger Admin UI where I can't see Ranger KMS service itself.I can see the plugin is enabled and responding in the plugins tab, cm_kms policy is syncing as per plugin status tab. However, service is not there. Any suggestions please. Thankssnm1523
After reboot clutser hadoop, a few services like yarn don't start anymore with the following message : resource_management.libraries.providers.hdfs_resource.WebHDFSCallException: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://hostname:50070/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs'' returned status_code=401. <html><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/><title>Error 401 Authentication required</title></head><body><h2>HTTP ERROR 401</h2><p>Problem accessing /webhdfs/v1/ats/done. Reason:<pre> Authentication required</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/> <br/> Any idea ?Thanks
Hi , NIFI was installed before with root user in our enviorment, now in order to switch it to non-root user, i'm wondering if you can help in the steps that i need to follow. Should i clean and re-install it again? The version is 2.4.2.0 Thanks SJ
Hello, I downloaded the latest Nifi version. i was able to connect nifi to an oracle database and it works fine. However, i need to connect nifi to hive, my problem is i cant find any processors related to hive. There is no SelectHiveQL processor and no PutHiveQL processor or any processor related to hive at all. not even the controller services for hive is found. Am I doing anything wrong? I just need to connect Nifi to hive and i can't find the processors responsible for this. Please help. Thanks in advance
I am trying to write the below query in getMongo processor {"updatedAt" : {$gt : 2022-12-10}}But it is showing an error stating that query is not a valid JSON representation. I tried adding double quotes as well but when I run it, it doesn't show any error nor any results in success, failure or original nodes. My end goal is to fetch incremental data i.e. only the new data added daily (day -1) using "updatedAt" field. Kindly help me with this
Hi folks, I'm new to CDP and I'm trying to figure out what roles I require to be able to view Data Hub Clusters as well as the associated Environment and Data Lake pages in CDP Public Cloud? I've been granted the following Account/Resource Roles:"IamViewer""EnvironmentUser" This allows me to see specific DataHub Clusters but when I click the links for either Environments or Data Lakes there's no information appearing. The EnvironmentUser Role has the following Policy attached: [ { "crn": "crn:altus:iam:us-west-1:altus:policy:EnvironmentUserPolicy", "policyStatements": [ { "rights": [ "datahub/read", "datahub/write", "datalake/read", "environments/read" ], "resources": [ "*" ] } ] }]Given the Policy has environments/read & datalake/read what am I missing here? Thanks.
I am using the parameter contest in that their is parameter, which is a sensitive parameter and i need to call in routeOnattribute for our requirement. how can i achieve this calling the sensitive parameter from parameterContests.Please help me out. Thanks
NiFi Version: 1.16.1Java 11 I noticed the logback.xml is the setting for the nifi logs. However, the file size of the nifi-app.log is exceeding the maxFileSize set on logback.xml. This is my logback.xm file:
is TEZ UI working on CDP 7.1.7 ?if yes, how can we enable it ? Regards ;
Hi Team, We had lost the master passphrase for one of the navencrypt server. Can you help us on resetting the master passphrase to new one.Thanks,Ashok
in zeppelin i run show databases; and show tables works but when pyspark to select * from tablesget error permission denied user = anonymous java.security.AccessControlException: Permission denied: user=anonymous, access=READ, inode="/warehouse/tablespace/managed/hive when i check getfacl in /warehouse/tablespace/managed/hive user zeppelin can rwx the directory why does zeppelin read as user anonymous ? anybody can help ? thanks
Hello Experts, I have custom processor code where I am using 2 controller services as input property.DBCPService (jdbc connection pool) works fine without any issue,but when I try to take input as MongoDBClientService (mongodb connection pool controller service) I get MongoDBClientService NoClassDefFoundError error during project build. public class TrackRequestStatus extends AbstractProcessor { public static final PropertyDescriptor DBDC_SERVICE = new PropertyDescriptor.Builder() .name("JDBC Connection Pool") .description("JDBC Connection Pool") .required(false) .dynamic(true) .identifiesControllerService(DBCPService.class) //.addValidator(StandardValidators.NON_EMPTY_VALIDATOR) .build(); static final PropertyDescriptor CLIENT_SERVICE = new PropertyDescriptor.Builder() .name("mongo-client-service") .displayName("MongoDB Client Service") .description("If configured, this p
I set up a Haproxy service:listen ranger_admin bind 0.0.0.0:6080 mode tcp server ranger_node1:6080 server ranger_node2:6080 Then I refer to the document,https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/fault-tolerance/content/configuring_ranger_admin_ha_with_ssl.htmlfrom 32 steps to the end. I found the Ranger authorization invalid.Namenode log error:WARN client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(182)) - Error getting policies. secureMode=true, user=nn/node1@TEST.COM (auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0}, serviceName=Captain_hadoopWARN client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServiceTagsIfUpdated(365)) - Error getting tags. secureMode=true, user=nn/node1@TEST.COM (auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0}, serviceName=Captain_hadoop Ranger admin access log error:"GET /service/plugin
Hello, I would like to write a log file to /var/log/cdp/hdfs/audit/solr/spool directory.I added the following settings to Cloudera Manger's HDFS Advanced ranger-solr-audit.xml.xasecure.audit.destination.solr.batch.filespool.enabled=truexasecure.audit.destination.solr.batch.filespool.dir=/var/log/cdp/hdfs/audit/solr/spoolxasecure.audit.destination.solr.batch.filespool.archive.dir=/var/log/cdp/hdfs/audit/solr/spool/archivexasecure.audit.destination.solr.batch.filespool.archive.max.files=50/var/log/cdp/hdfs/audit/solr/spool directory was missing, creating spool and archive directories.Should I modify install.properties or ranger-hdfs-plugin/install/conf.templates/enable/ranger-hdfs-audit.xml directly?Or, I would like to know if there are any additional parts that need to be set up in Cloudera Manager. Any help would be highly appreciated.
Hi All,I am running 4 node Nifi cluster with each node having 8 core and 32 GB or RAM. it was running fine until I introduced QueyRecord processor to process huge metric data (14 GB per min). the CPU now consistently remaining above 85% and memory above 90% (buffering too much). Can someone let me know how to diagnose the issue and find the root cause. Thanks in advance
Hello, We're looking for some advice on how to identify possible improvements on our current NiFi flow that is a copy of what we had implemented in Flume, both running on AWS with CDP Private Cloud. Our use case begins with an HTTPS listener that in its simplest form receives GET requests and processes them and writes events into Kafka. Between those two ends, we perform some validations and transformations:- we convert the request parameters into JSON format (using 5 processors with AttributesToJSON and ReplaceText)- convert JSON format to rename properties using a JoltTransformJSON- validations using a couple of RouteOnAttribute + ReplaceText to confirm mandatory fields and transform null values- mapping of numeric values into strings using reference data (using UpdateAttribute, RouteOnAttribute and 2 LookupAttribute)- convert JSON into Avro format with ConvertRecord- returning the HTTP response + publishing each individual record to Kafka In total, we have around
How can I download Cloudera Manager 7.x Installer
I am fetching data from table for employees and each employees has maximum 7 Records. I am splitting data based on the IDs and week number column. I need to merge the data in such a way that each flow file contains 100 records and all the records of each employees comes into that flow filefor exampleMy flow file already has 98 records and next employee has 7 records so, that employee's data should not be the part of that flow file. It should come in next flow file likewise..... While merging the FFs order is also important.How can I do that. I am not familiar with all the processors of NiFi
Hello, I am unable to launch Yarn Queue Manager Web UI and get error - Failed to load the Yarn Queue Manager UI. Reload the UI after fixing the issue. Reason - No active RM Found. We have 2 RM (Active/ Standby) yet facing this error. Reques any help / guidance to resolve this. CM 7.6.1Cloudera Runtime - 7.1.7
Hi guys, I am facing a problem with Nifi I have about 80G data in queues that make backlogs and I need the data to be ingested faster and but I can't clear the queues in case of missing files, Please help me solve this issue, thanks a lot. NIFI full queues:((((
I read a Document guide of Cloudera on this Schedule Job link. The problem is I don't have access to "Cloudera Data Platform (CDP) management console" (which looks like below):. I only have access to Cloudera Web server UI on xxx.xxx.xxx.xxx:7180, which looks like this:Please note that we run Cloudera on a host machine that runs centos, and I have to ssh to that machine there isn't any UI like the first picture, there is only a black window with line commands. I only have the webserver. Same problem for many other guides on the website, they're always require you have access to CDP management console UI
Hi AllWe have our hive database on Postgress and wanted to migrate it to MySQL,Is there any easy way to do it? Thanks
we want to find the **approach / test /cli / API** that gives us the results about HeartBeat Lost between Ambari agent to Ambari serverHeartBeat Lost could be as results of poor connection between Ambari agent to Ambari server or because Ambari server was down for along time , etc Note - from Ambari GUI the machine with **HeartBeat Lost** state usually colored by yellow state clarification: the case as described here appears when `ambari-agent status` is in running state as the followingambari-agent statusFound ambari-agent PID: 119315ambari-agent running.Agent PID at: /run/ambari-agent/ambari-agent.pidAgent out at: /var/log/ambari-agent/ambari-agent.outAgent log at: /var/log/ambari-agent/ambari-agent.log
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.