Member since
08-27-2018
4
Posts
1
Kudos Received
0
Solutions
08-21-2020
02:32 PM
1 Kudo
Accessing CDP Endpoints using Postman or cURL
When automating, testing, or validating APIs within the Cloudera Data Platform within our AWS, Azure, or GCP infrastructures, most developers utilize either curl or postman. In this article, we will highlight the proper API endpoints, proper credentialing through Knox, and the setting of workload passwords.
Details Needed to Access Endpoints
There are several details to access specific application endpoints within CDP:
The Endpoint URL for the Specific Application
Your specific username
Your Workload Password
The Endpoint URL for the Specific Application
Each application within a CDP environment will generate a unique URL for both GUI and API development. As an example for DataHub Clusters, you can view the API endpoints by clicking on the Cluster details, scrolling to the bottom, and selecting "Endpoints."
Within the Endpoints section, you'll find the appropriate API endpoints for each of the applications on that specific cluster.
Your Specific Username
As you connect into CDP using an identity provider, you need to fully verify your username. To verify your username, click on your name and view your profile. Within the profile, it will list your username.
In this example, my workload user name is 'jkibler'.
Your Workload Password
On the same profile screen, you can manage your workload password. Click "Set Workload Password" and assign a password for your account. You will need to wait for the password to sync completely before proceeding.
Using cURL
Within your command line, you can use standard curl with a username and command-line password. As an example, I'm querying one of my SOLR endpoints with my username:
curl -u [workload user name] -L -i "https://jkibler-solr-eval-....cloudera.site/jkibler-solr-eval/cdp-proxy-api/solr/jkibler/select?q=*%3A*"
When you execute this command, a command-line password prompt will appear for you to enter your password.
Using Postman
Within postman, generate a GET request with the full endpoint. Within Authorization, place your [username] and [workload password] into the Basic Authentication type. Then, select "Update Request."
At this point, you can "send" the request.
... View more
11-02-2018
01:52 PM
Hi @siddharth pande - Can you also expand on the type of duplication -- do 4 concurrent tasks equate to duplication of 4x of the originating data on Kafka? Alternatively, are you getting one partition duplicated while others not? Does the issue happen all of the time? Are we sure the duplication isn't happening before it gets into Kafka (say, a producer sending duplicate data?) Per Raffaele's suggestion, please send over the configuration of the Kafka Processor within Nifi. Also, if you have fake data (or a schema that we could follow that you can easily reproduce), you can share that you can get this to duplicate by publishing to a topic, I'd like to try to reproduce it. Thanks, Jeff
... View more