Resources

The file "cdswCluster.runs.jsonlines" in the output contents of "cdsw logs" contains session information of all projects. But there should be a regular archiving operation, that is, it may only keep a fixed number of records/ records within a fixed time range. I can't confirm this now. I have sorted out how to query this information through the curl command: On the CDSW Master node:   authHeader="Service-Authorization: Basic $(printf cdsw-metrics-service:$SERVICE_ACCOUNT_SECRET | base64 -w 0)"domain=$(kubectl get secrets internal-secrets --namespace=${CDSW_NAMESPACE} -o jsonpath="{.data.domain}" | base64 -d)apiUrl="${protocol}://${domain}/api/v1"datasets_json=$(curl -sSf -H "$authHeader" "${apiUrl}/metrics/datasets/names")if [ 0 -eq "$?" ]; then datasets=$(echo "$datasets_json" | python2 -c "import sys, json; print '\n'.join(json.load(sys.stdin))") for dataset_name in $datasets; do curl -sSf -H "$authHeader" "${apiUrl}/metrics/datasets/values/${dataset_name}" >$(pwd

In past years many companies have come up with (Hybrid) Cloud strategies, and there is no shortage of recommendations for when and how to use Cloud providers such as AWS, Azure, GCP, and others. However, most advice is geared towards a generic setting, and may not translate well to a Big Data context.    And here is the main reason why:   BIG Data requires BIG resources that have a BIG cost impact   This does not mean that any specific vendor is expensive, and it is not limited to a single deployment form such as PaaS or SaaS. It comes down to the fundamental economics that infrastructure provides must cover the costs they make, and if you use their solution for a big amount of data, these costs will be larger than for a small lightweight app.    In this context, hopefully, the following advice makes sense.   Avoid Network UsageFlexible Cloud Infrastructure vs Economical On-Prem infrastructureCostValueStay in control of your dataSecurityAccessibi

In CDP Public Cloud, Data Hub is a service for launching and managing workload clusters. Data Hub clusters strongly resemble traditional (CDH/HDP) clusters. This makes Data Hub clusters an attractive destination for migrating existing workloads to the cloud. In CDP Data Hub, clusters are secure by default.  Among other things, this means that perimeter security is provided by Apache Knox Gateway. For remote clients, this means that the available cluster services are only accessible by way of secure endpoints.  In this article, I will walk through how to submit Spark jobs to a Data Hub cluster from a remote client, and I will also provide examples demonstrating how to use the API. Apache Spark provides a spark-submit script, which is used to launch applications on a cluster. The spark-submit script depends on having network access to the cluster manager (YARN). A remote client is one that is not deployed on a target Data Hub cluster node, such as a Gateway node. T

In Kafka, topic/partition data is stored in data directories. These directory locations are configured using "log.dirs" config property. We can configure one (or) more directory locations. Kafka balances the partition data directories across these given directory locations. Normally we start with one directory location. With in increase in data size, we may need to add more disks. We can append new directory location to existing "log.dirs" config property. After server restart, Kafka uses new directory location for new partitions. Kafka does not automatically move existing partitions directories to new directory locations. (i.e) It does not auto balance the partitions across directory locations. Some times we want to move some partition data to different location. We have two approaches for this Approach 1: Just delete existing data directory contents and configure new data directories locations In this approach, Kafka replicates the partition data from other members of the cluster

Some people use the Boto3 library to browse their Amazon bucket from Python, and I was searching the same for Azure. This is far from being optimized, but it could be a starting point. First things first, we need to find the Azure access token. Notice those keys are supposed to rotate so you have to have that in mind. In Azure portal, let's get to the Storage Account in the ResourceGroup defined for your account, and click on Access keys There are two keys (for rotation without interruption), let's copy the first one.    In my CML project, I'm defining an AZURE_STORAGE_TOKEN environment variable with that key:   As you see above, 'STORAGE' variable has been populated. If you want it to be automatically populated, here's some code:   !pip3 install git+https://github.com/fletchjeff/cmlbootstrap#egg=cmlbootstrapfrom cmlbootstrap import CMLBootstrap# Instantiate API Wrappercml = CMLBootstrap()# Set the STORAGE environment variabletry : storage=os.environ["STORAGE"

In this article, we will learn to pass atlas-application.properties configuration file from a different location in spark-submit command. When Atlas service is enabled in CDP, and we run Spark application by default, atlas-application.properties file is picked from /etc/spark/conf.cloudera.spark_on_yarn/ directory. Let's test with SparkPi example:   spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client /opt/cloudera/parcels/CDH/jars/spark-examples*.jar 10   We can see the following output in the application log.   21/08/23 06:12:03 INFO atlas.ApplicationProperties: Looking for atlas-application.properties in classpath21/08/23 06:12:03 INFO atlas.ApplicationProperties: Loading atlas-application.properties from file:/etc/spark/conf.cloudera.spark_on_yarn/atlas-application.properties   If we want to pass the atlas-application.properties configuration file from a different location, for example /tmp directo

Some NiFi Expression Language (EL) expressions can be fairly complex, or used in a large flow, or both. These can make it difficult to test an EL expression on a running NiFi system. Although an excellent feature of NiFi is being able to adapt the flow while the system is running, it may not be prudent to stop a downstream processor, reroute a connection to something like UpdateAttribute, then list the queue in order to see attributes, content, etc.To make EL testing easier, I wrote a Groovy script called testEL.groovy that uses the same EL library that NiFi does, so all functions present in the specified NiFi version are available to the test tool. The following is the usage:usage: groovy testEL.groovy [options] [expressions] Options: -D <attribute=value> set value for given attribute -help print this messageAs an example, the following tests an expression that appends "_world" to the "filename" attribute:> groovy testEL.groovy -D filename=hello '${filena

This video describes a way to hide the plain text passwords in Knox topology. The passwords are basically the Bind User passwords when used in ShiroProvider and LDAP/AD authentication.  

Use Case: Ingesting energy data and running an Apache Spark job as part of the flow.We will be using the new (in Apache NiFi 1.5 / HDF 3.1) ExecuteSparkInteractive processor with the LivyController to accomplish that integration. As we mentioned in the first part of the article, it's pretty easy to set this up.Since this is a modern Apache NiFi project, we use version control on our code:On a local machine, I am talking to an electricity sensor over WiFi in a Python script. This code is processed, cleaned and sent to a cloud hosted Apache NiFi instance via S2S over HTTP.In the cloud we receive the pushed messages.Once we open the Spark It Up processor group, we have a flow to process the data.Flow OverviewQueryRecord: Determine how to route based on query on streaming data. Converts JSON to Apache AVRO.Path for All FilesUpdateAttribute: Set a schemaMergeContent: Do an Apache AVRO merge on our data to make bigger files.ConvertAvroToORC: Build an Apache ORC file from merged

Following is the criteria: ORC and Managed table ACID enabled HDFS directory should be owned by Hive According to the shouldTableBeExternal method in HiveStrictMigration.java, a table is external if it is a StorageHandler table, or if it is an Avro/Text/Parquet table, or if it is a list bucketed table, or the directory is not owned by Hive.    Following is the codebase that describes the same: String reason = shouldTableBeExternal(tableObj, ownerName, conf, hms, isPathOwnedByHive); if (reason != null) { LOG.debug("Converting {} to external table. {}", getQualifiedName(tableObj), reason); result = TableMigrationOption.EXTERNAL; } else { result = TableMigrationOption.MANAGED; Reference: HiveStrictManagedMigration (GitHub) It should be converted to an EXTERNAL table, in case it does not fit the criteria for the upgrade.

This video explains feasible and efficient ways to troubleshoot performance or perform root-cause analysis on any Spark streaming application, which usually tend to grow over the gigabyte size. However, this article does not cover yarn-client mode as it is recommended to use yarn-cluster for streaming applications due to reasons that will not be discussed on this article.   Open the video on YouTube here   Spark streaming applications usually run for long periods of time, before facing issues that may cause them to be shut down. In other cases, the application will not even be shut down, but it could be facing performance degradation during certain peak hours. In any case, the amount and size of this log will keep growing over time, making it really difficult to analyze when they start growing past the gigabyte size.It's well known that Spark, as many other applications, uses log4j facility to handle logs for both the driver and the executors, hence it is recommended to tun

It may often come to pass that you need to utilize data that does not exist in your Environment's Data Lake. This article will cover 2 scenarios where you need to access data that is outside of your Data Lake (either in a Storage Account in the same resource group as your CDP Public Cloud or in a different resource group, albeit in the same subscription). UPDATE: Scenarios 3 and 4 below cover RAZ! Scenario 1 - Adding External Storage Account Access to your Data Access Identity (non RAZ) In this scenario, you may want a single managed identity to be able to access both your Data Lake Storage Account as well as your External Storage Account. For this, I will choose to grant access to only a single container within the storage account (but, as you will see in the following scenario, you could grant access to the entire storage account - I'm showing both purely for illustration). Find your DataAccessIdentity in the Azure Portal, and note it's Object ID Find your External Storage Account

Hello Team, I have a situation here.When I created my support profile using my company email address, it gave me an option to select a company as my company (i.e. xxxxx) have multiple support accounts. I selected the correct and relevant company i.e. (xxxx Group) for me and since then it was all working fine. This morning when I logged in to the support portal, it automatically updated my company information to (xxx India) and now it does not allows me to go back to my original company selection. Having company as (xxxx India) is irrelevant to my profile and due to this I have lost tracking and visibility to all the support cases raised with Cloudera under company (xxxxx Group). Please help me to get my company information on my profile reverted to my original selection. P.S.: All the other team members of my team still has their company selected as (xxxx Group). ThanksSunil M.

Summary of this article This article attempts to explain the data sources of the Overview and Usage tabs and their meanings in the Admin page of CDSW. Background I have encountered several customers asking me the same questions. It is how to evaluate the resource usage of the CDSW cluster. Below, I will record some of the specific questions asked by customers with a representative meaning, so that I can respond to customers more quickly in the future, and for other people's reference. Why is the usage graphs I observed on Grafana of CDSW not consistent with the resource usage rate observed on the host chart s on CM? For your confusion, from the conclusion: On the CM Web UI, find the page of the specific host. The charts you see under the Chart Library are all accurate. Please use this as a benchmark to judge the true situation of the host's resource usage. The data source for this part is the Host Monitor of Cloudera Management Service, a monitoring service designed and developed by

As per HIVE-12192, by default, Hive displays timestamp in UTC. This article lists the Hive query commands that can be used to change the timestamp. To display the current timestamp in Asia/Kolkata:   SELECT from_utc_timestamp(cast(from_unixtime(cast(current_timestamp() AS bigint)) as TIMESTAMP),"Asia/Kolkata")   To display a UNIX epoch number in Asia/Kolkata:    SELECT from_utc_timestamp(cast(from_unixtime(cast(1623943533 AS bigint)) as TIMESTAMP),"Asia/Kolkata") ;  

To list all the tables that are ACID(TRANSACTIONAL)/ NON_TRANSACTIONAL, perform the following steps: Connect to the backend database Execute "use hive" as all the tables are in the Hive databaseNote: Please check if the database name is different. This is specified at the time of creating the Hive service. Execute the following to check for ACID table:select PARAM_KEY,PARAM_VALUE,TBL_NAME from TABLE_PARAMS,TBLS where PARAM_KEY='transactional' and PARAM_VALUE='true' and TABLE_PARAMS.TBL_ID=TBLS.TBL_ID;​ OR, execute below to check for NON-TRANSACTIONAL  table. select PARAM_KEY,PARAM_VALUE,TBL_NAME from TABLE_PARAMS,TBLS where PARAM_KEY='transactional' and PARAM_VALUE='false' and TABLE_PARAMS.TBL_ID=TBLS.TBL_ID;​    

Note: Cloudera does not support antivirus software of any kind. This article contains recommendations for excluding HDP components and directories from AV scans and monitoring. It is important to note that these recommendations do not apply to each service, and further, some services will have additional items to exclude which are unique to them. These details will be addressed in individual articles dedicated to the service in question. The three primary locations you will want to exclude from Antivirus are: Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the AV holds up writes. Log directories: These are write-heavy. Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the AV holds up writes. In general, consider excluding the following directories and all o

When a Hive replication policy executes in Replication Manager, the admin can monitor the details of the policy and its execution by checking the underlying command details on the source CDH Cluster's CM. In order to export the data and metadata related to the specified Hive tables, multiple steps are executed in the source Cloudera Manager. The Export Hive Metastore section has multiple steps that are explained below:   The first subcommand exports the Hive Metadata for the input database and tables. The next step transfers the HDFS files from the CDH cluster to the cloud storage, which is used by the Data Lake cluster. The next step exports the relevant Sentry permissions. Once the above steps are executed successfully, the workflow resumes on the Data Lake cluster as explained in the previous section.  With these steps, the administration now has a detailed behind-the-scenes view of the steps involved in running a Hive replication policy.

When a Hive replication policy executes in Replication Manager, the admin can monitor the details of the policy and its execution by checking the underlying command details on the Datalake cluster's CM. An admin can determine further details about the policy by checking “All Recent Commands”. The first step is the Export Metadata step that is performed on the source cluster. This setup in turn exports the data and metadata of the input Hive databases and tables specified in the policy. From the command details, the admin can find logs of the check metadata step. Next is the Import metadata step that updates the Hive database (and table) metadata for a given replication policy. Finally, the last step imports the Exported Sentry permissions into Ranger policies on the Data Lake cluster. If the above steps ran into an error, the admin can determine the specific error by checking the stderr logs of the specific step that ran into an error to troubleshoot the actual error.

A step-by-step guide that explains the process to register a CDH Cluster as a Classic Cluster is outlined below. This process currently outlines the CDH clusters that aren’t directly visible to the CDP control plane.    Admin needs to select the Classic Cluster tab in the CDP control plane and proceed further with the next steps: CDH Cluster is discovered by its internal IP address which is not visible to the outside world. The wizard provides step-by-step instructions on how to proceed further with the classic cluster registration process. Once the administrator installs the required packages, the admin can verify whether the service is installed and started properly by logging into the CM server host Run the status command to verify whether the service is running properly. Proceed further with the classic cluster registration process. Provide the Cloudera Manager login credentials here. Provide further details such as location, data center name (which is arbitrary) about t

1. Get the output from Ambari serverhttp://AMBARI-HOST:8080/api/v1/stacks/HDP/versions?fields=repository_versions/operating_systems/repositories/*,repository_versions/operating_systems/OperatingSystems/*&repository_versions/RepositoryVersions/repository_version=3.1.0.0-782. Copy the json of "operating_systems" outer block of step1 output and create repo.json file -it would looks like repo.json3. update repo.json file to reflect your changes/modifications.4. now run below API call to update.curl -insecure  -H "X-Requested-By:ambari" --user admin:admin --data @repo.json  -X PUT  http://AMBARI-HOST:8080/api/v1/stacks/HDP/versions/3.1/repository_versions/1Note: please make sure you include correct repo version id at the end. in this case it is 15. Now go and check in UI to see if those changes reflects.Note: This tried and tested in Ambari2.7.x

The Qualys tool reports vulnerabilities in ZooKeeper, even when the ZooKeeper security configuration is applied (HDP doc, CDP doc). There are two kinds of reports Qualys makes that are not addressable by Cloudera: The security guidance keeps several znodes in the affected services as world-readable, like so:/zookeeper/quota sasl:zookeeper:cdrwa,world:anyone:rAny world-readable znode will appear on the scanner, and will require an exception filed for it. This is the position of Qualys, as reported by our customers who use Qualys. The security guidance does not cover several services. The following components require no action according to our documentation: Calcite Knox MapReduce Spark Tez Zeppelin The Qualys tool will still report znodes owned by these services. Note: it is possible to harden the ACLs beyond the Best Practices recommendation in the documentation, and to harden the ACLs of services not covered in the Best Practices. However, Cloudera cannot provide what the

This article will cover the following:   The purpose of this article Problem Statement Troubleshooting Step-1: Use the kubectl describe pod command to view the details of the Pod of the Session. Step-2: Use kubectl describe node to view the status of all the nodes. Step-3: Verify the kubelet process argument. Step-4: Check the nodefs and imagefs recognized by kubelet. Summary The purpose of this article This article records the troubleshooting about CDSW unable to start the Session. The reason for the problem involved is that the remaining capacity of nodels and imagefs used by Kubelet has reached the threshold of evicting the node, resulting in the node being labeled as NoSchedule taint by Kubernetes. Problem Statement One of my customers reported that they found their CDSW failed to start sessions. The following is a screenshot of the failure to start a session on CDSW. Troubleshooting Use the Kubectl describe pod comman

This article will explain the following:   How to connect to CDSW database using kubectl exec? How to forward the database service of CDSW to the host? Change user's password if you cannot set up smtp service to change password First of all, we need to know that in CDSW/Kubernetes, we use SDN (Software Defined Network) to create a network that is only used by Pod/Service within the cluster. For example, the IP CIDR used by the CDSW nodes is 192.168.0.0/24. Then the Pod IP CIDR used by CDSW/Kubernetes will generally not overlap with 192.168.0.0/24 such as 100.66.0.0/16.    kubectl cluster-info dump | grep -m 1 cluster-cidr# "--cluster-cidr=100.66.0.0/16",   How to connect to CDSW database using kubectl exec? For how to connect to CDSW database, you can refer to this official documentation. CDSW_NAMESPACE="default" # All CDSW internal applications are running in "default" NameSpace.db_pod=$(kubectl get

Purpose of this article Our purpose is to replace the nodes of Kafka Broker. We have two nodes newly added to the CDP PvC Base cluster. We will migrate two Kafka Brokers that were originally in use to these two new nodes. This article uses Cloudera CDP official documentation as a guide. Summary Environment Steps of Broker Migration Step-1: Startup the new broker as a member of the old cluster Step-2: Stop both the new broker and the old broker that it is replacing Step-3: Change broker.id of the new broker to the broker.id of the old one both in Cloudera Manager and in data directory/meta.properties. Step-4: Startup the new broker. Environment Here are the currently in use Kafka Brokers: Broker Host IP 1546345005 c3669-node3.coelab.cloudera.com 172.25.36.133 1546345009 c3669-node4.coelab.cloudera.com 172.25.36.207 1546345001 c3669-node2.coelab.cloudera.com 172.25.36.206 Steps of Broker Migration Startup the new broker as a member of the old clus