Hello @KSKR ,   thank you for your inquiry about how to profile Spark workloads.   Apart from the Spark Web UI (which parses the event log file), you can use the same event log file along with the Spark application log (container logs) to compare the same or similar applications run with each other. It can reveal tuning opportunities, like: "my application was running much faster a month ago, what changed?"   Cloudera has a product that does it for you along with other goodies, and it is called Workflow XM.   Please let me know if you need more information on this topic.   Thank you: Ferenc ... View more

Hello @xinfengz ,   thank you for your interest in the installation of CM6.3.3.   Based on the documentation under "Managing Licences", you have the below options: When you install Cloudera Manager, you can select among the following editions:  Cloudera Express  (no license required), a 60-day  Cloudera Enterprise   Cloudera Enterprise  trial license, or  Cloudera Enterprise  (which requires a license).  To obtain a   Cloudera Enterprise   license, fill in this   form   or call 866-843-7207.   Please let us know if it addresses your inquiry.   Thank you: Ferenc ... View more

Hello @Love-Nifi  and @vchhipa ,   Thank you for posting your inquiry about timeouts. Without the full log, I can provide only some "if you see this, do that" kind of instructions.   If you see an ERROR message with: org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow, then follow the below is the steps to resolve the issue: 1. Go to NIFi UI > Global Menu > Cluster 2. Check which host is the coordinator and login to that host on the shell. 3. Go to flow.xml.gz file location. [default location is /var/lib/nifi/conf/] 4. Copy flow.xml.gz on the disconnected node and replace the original flow.xml.gz with copied flow.xml.gz file. 5. Check permissions and ownership of newly copied flow.xml.gz file and then restart Nifi on the disconnected node only.   If you are suspecting purely timeout issues, please attempt to tweak the below values in nifi.properties and restart the service: - nifi.cluster.node.protocol.threads=50 (Default 10) - nifi.cluster.node.connection.timeout=30 sec (Default 5 sec) - nifi.cluster.node.read.timeout=30 sec (Default 5 sec) Please find below a set of configurations that worth tuning on larger clusters based on https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html The below are some example values for larger clusters (you need to tune it based on your unique setup): nifi.cluster.node.protocol.threads=70 nifi.cluster.node.protocol.max.threads=100 nifi.zookeeper.session.timeout=30 sec nifi.zookeeper.connect.timeout=30 sec nifi.cluster.node.connection.timeout=60 sec nifi.cluster.node.read.timeout=60 sec nifi.ui.autorefresh.interval=900 sec nifi.cluster.protocol.heartbeat.interval=20 sec nifi.components.status.repository.buffer.size=300 nifi.components.status.snapshot.frequency=5 mins nifi.cluster.node.protocol.max.threads=120 nifi.cluster.node.protocol.threads=80 nifi.cluster.node.read.timeout=90 sec nifi.cluster.node.connection.timeout=90 sec nifi.cluster.node.read.timeout=90 sec   Please check if you notice any certificate related exception, like: WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message due to: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate In this case, create a new keystore and truststore and add client auth in the keystore.   Best regards: Ferenc ... View more

Hello @unkedeuxke ,   thank you for your enquiry about upgrading Hive 3.1.0 to 3.1.2 on HDP 3.1.4 without upgrading other components.   I would personally re-phrase the question: - why is it only supported to upgrade the entire HDP stack instead of the services individually?   HDP is a distribution of various components bundled into a product. These components are tested against each other, so the distribution works reliably. Should you upgrade a single component only, you lose these guarantees that your HDP distribution will work. For instance think about API changes, new features or bugs being introduced.    In short: you might able to upgrade a specific component however, it is not recommended because it might break things.   Hope this helps!   Best regards: Ferenc ... View more

Hello @ameya ,   thank you for your question about how to upgrade your cluster from HDP 2.6.5 to CDP 7.1.1. This quick start guide details the steps to follow.   If you’re a Cloudera Subscription Support customer, we can connect you with your Account team to explore a possible Services engagement for this request. Let us know if you’re interested in this path, we’ll private message you to collect more information.   Please let us know if you need any further assistance!   Best regards: Ferenc ... View more

Hello @emeric , the "kinit: KDC has no support for encryption type while getting initial credentials" usually occurs after configuring encryption types that do not match the ones present in the tgt principal (such as krbtgt/CLOUDERA@CLOUDERA) in the KDC. This can also happen while starting a service where the enctypes in the krbtgt principal do not match those used in service keytab.   From an earlier Community post: Please compare the Kerberos server and client configurations and reconfigure krb5.conf on all your nodes to explicitly use the supported encryption type.   The documentation describes: " Kerberos client OS-specific packages must be installed on all cluster hosts and client hosts that will authenticate using Kerberos."   Wondering if some missing packages might be the issue?   Kind regards: Ferenc ... View more

Hello @meiravR ,   thank you for confirming that you ensured your AWS account is configured accordingly to our documentation. Should you have a Cloudera Support Subscription, please file a support case with us to assist you further, as we reached the limit of what can be addressed efficiently via Community.   Thank you: Ferenc ... View more

Hello @md186036 ,   The error message you pointed out [1] seems to be a known issue and is looked at by the below internal JIRA ticket: NAV-7272 - NPE in getEpIdentitiesForMissingRelations As per the JIRA ticket: "An NPE is being caused by getEpIdentitiesForMissingRelations() during Spark extraction. The condition that causes it is rare, however, once the condition exists, because of the NPE, it will continue forever. The code is trying to detect ep2Ids for linked relations that are missing so they can be added. However, the code fails to check for null in the case that this is true."   The fix is not available yet in any currently released CDH distribution. The fix might be available in CDH6.4.0, 5.16.3, 6.2.2, 6.3.4, 7.1.1, 5.17.0.   My understanding is that this can cause no new metadata is produced. Should you have a Cloudera Support Subscription, please kindly file a support ticket with us to assist you further, as there is no workaround identified for this bug.   Thank you: Ferenc   [1]  ERROR SparkPushExtractor [qtp1810923540-17908]: com.cloudera.nav.pushextractor.spark.SparkPushExtractor Error extracting Spark operation. java.lang.NullPointerExceptio   ... View more

Hello @meiravR ,   thank you for your feedback. I have checked the credentials page and I can see a tooltip on the UI that explains what is the "Enable Permission Verification" for. Although I cannot see the "question mark tooltip" in your screenshot.   Is your AWS environment satisfies the prerequisites detailed here, please?   Thank you: Ferenc   ... View more

Hello @meiravR ,   I have discussed your question internally and it seems to be that the regions are not being listed when the credential you’re using doesn’t have permissions to list the regions. When you are creating a credential, there’s a checkbox to verify permissions. When enabled it should check the permissions prior to creation and throw errors is perms are missing from the CB policy.   The 504 Gateway Timeout is displayed e.g. when the creation of an environment fails. We are currently working on returning a more meaningful exception (internally tracked DWX-4799).   Kind regards: Ferenc ... View more

Hello @md186036 ,   I have checked the Navigator logs in a test cluster for CDH5.16 (for our purposes this minor version difference should not matter much) and found the entries mentioned in this doc for CDH5.15: 2020-07-29 17:41:18,151 INFO com.cloudera.nav.server.NavServerUtil [main]: Found 885 documents in solr core nav_elements 2020-07-29 17:41:18,155 INFO com.cloudera.nav.server.NavServerUtil [main]: Found 916 documents in solr core nav_relations   I was navigating in CM -> Cloudera Management Service -> Navigator Metadata Server and then clicked on Log Files -> Role log file. I have downloaded the file by clicking on the "Download Full Log". I have observed that I have log entries saying INFO messages.   Do you see log entries with INFO in the log too, please? I would like to rule out that the log level was not applied since your config change (e.g. was not restarted) or the log level was changed in other means (e.g. without restart).   Thank you: Ferenc ... View more

Hello @meiravR ,   thank you for asking for clarification. I was just wondering if e.g. between step 3. and 5. you pasted the data somewhere (e.g. to a notebook, an editor, a text file, a chat window, a google doc, an MS Word editor) and then copied it out again?   Thank you: Ferenc ... View more

Hello @md186036 ,   do you see INFO level messages in the prod cluster log? I am suspecting that your log level is set to e.g. WARN or ERROR level/threshold and maybe this is the reason you do not see the "nav_" elements. It can be set via CM's service configuration too   Thank you: Ferenc ... View more

Hello @md186036 ,   thank you for your questions on How to disable heap dump, so your tmp folder is not filling up on OOM How to calculate the optimal heap size for Navigator Metadata Server if you cannot locate the nav_elements and nav_locations pattern in cloudera-scm-navigator log file   In CM -> Cloudera Management Service -> Configuration -> Category: Advanced -> Uncheck "Dump Heap When Out of Memory" To assist you more efficiently, can you please specify the CDH version you are using, please?   Thank you: Ferenc ... View more

Hello @meiravR ,   thank you for reporting your experience having difficulty adding a new environment to the CDP public cloud.   During the process of creating CDP credential, we need to copy and paste texts that contain special characters. Just to narrow down the possible cause of the issue, can you please confirm if you used any text editor to paste these values (e.g. for documentation purposes) and copied it back to the browser? I would like to rule out the possibility that encodings changed between copying and pasting, hence the characters displayed represented some other characters (that might be not obvious to differentiate visually, e.g. control characters are not displayed).   Do you face the same issue when you are using only a plain character-based editor to save your work, please?   Thank you: Ferenc ... View more

Hello @MeenaK ,   thank you for reaching out.   This community thread mentions, that " The ConvertJsonToAvro processor was removed from the default NiFi distribution bundle because of space limitations as of the Apache NiFi 1.10 release.".   this thread points to a repo of the Apache Nifi.   Hope it helps!   Kind regards: Ferenc ... View more

Hello @agsumeet , @direcision , @Dorown and @abinanths ,   thank you for reaching out! I have checked internally and the issue described in this thread looks like HIVE-16683 / SPARK-26932   As of now, there is no fix backported to any CDH release (at the time of writing it means CDH6.3.3 and earlier).   Should you have a Cloudera Support Subscription, please file a support case with us for further assistance!   Thank you: Ferenc ... View more

Hello @Pr1 ,   thank you for your enquiries about Graphql API access via REST and NiFi REST interactions.   Found this article about GraphQL via REST and the NiFi REST API Reference documentation.   Have you seen these already?   Thank you: Ferenc   ... View more

Hello @Jua ,   thank you for reporting the issue of CM not being able to connect to it's embedded database after installing manually postgres client rpm.  I understand that you already tried to start the scm server db.    Do you still face the issue, please or was it resolved since?   Please verify if: the postgres database is running and can be accessed? the db is running... once you kill the db process and restart the cloudera-scm-server-db.service, does it fix the issue?   Thank you: Ferenc   ... View more

Hello @jpetro416 ,   thank you for reporting the behavior that CM installation fails when it picks up a wrong hostname. In the log segment you've shared it seems to be there is a dot after localhost.localdomain: localhost.localdomain.   To alleviate the issue once hit it, please: update the /etc/hosts file with the correct hostnames copy this corrected file to all hosts update /etc/cloudera-scm-agent/config.ini to point to the CM server hostname hard restart the agent: # sudo systemctl stop supervisord # sudo systemctl start cloudera-scm-agent add the hosts to the cluster install Management Services Steps were verified for CM6.2.   Please let me know if I've addressed your observation.   Thank you: Ferenc ... View more

Hello @gss2020 ,   thank you for exploring our HDP Sandbox. For the default passwords, please see our "Learning the Ropes of the HDP Sandbox" tutorial page. Within this, please have a look at "Appendix A: Reference Sheet" for the login credentials.   Please let me know if you need any further input from us.   Thank you: Ferenc ... View more

Hello @Nagamalleswara ,   please make sure you are using the yarn user when collecting the thread dumps.   I've found useful this description too for figuring out what is going on with the JVM.   Kind regards: Ferenc ... View more

Hello @pauljoshiva ,   my understanding is that MySQL master-slave configuration's downside is that you need to fail over manually. You can load balance between active nodes, however in my read the master-slave means that the master node is active and the slave is only follower, therefore the slave node is not serving requests, hence nothing to load balance.   Kind regards: Ferenc ... View more

Hello @Nagamalleswara ,   What is odd is that the issue happens on large AWS nodes only and intermittently. Did you check if ports are open bi-directionally on the affected nodes for NMs? Can you access the NM WebUI on the affected node, please?   Thank you: Ferenc ... View more

Hello @netapp1 ,   thank you for your deep dive into the CDH6.x clusters CM Express version's 100 nodes limitation.   The intent from Cloudera is to limit Cloudera Express 6.x usage to 100 nodes. The implementation on how it was enforced from a User Experience point of view is different between minor versions (e.g. 6.0. vs 6.1. vs 6.2.). It means that earlier versions might let you install the agents across the nodes and once the installation completes it's blocks most of the functionalities until you decrease the number of nodes to 100.   We recognised that it would be best to block installation until the nodes are not exceeding 100 nodes and in later version it's been gradually improved.   Hence the discrepancy between the descriptions in the documentation.   In short: 100 node limitation is enforced across Cloudera Express 6.x versions.   Please let us know if we addressed your enquiry on this topic by pressing the "Accept as Solution" button, which helps other members to find the answer of a similar question.   Thank you: Feren ... View more