About snm1523

snm1523 · ‎01-18-2023

Thank you for the response @tj2007. We have ensured that the required permissions are assigned to the account that is provided to Cloudera to create principals. we further tweaked some settings and also after a quick modification to gen_credentials_ad.sh script (post discussion with Cloudera support) got through with error. However, now getting below error: We have scheduled a call later today with Cloudera once again to discuss this. However, if you may be able to suggest something would be helpful. Thanks snm1523

snm1523 · ‎01-17-2023

Hello, I am in process of setting up a CDP 7.1.7 cluster. At the moment, CM 7.6.1 is installed and integrated to AD on LDAPS protocol (Had to select authentication type as LDAP for the integration to work even though we have AD being used). Next step is I have added few basic services i.e. HDFS, YARN and Zookeeper and now I am enabling Kerberos. At the step of Generating credentials it fails with attached screenshot. However, in the same window I noticed that CM is trying to connect to AD on LDAP protocol on port 389. Ideally it should be connecting via LDAPS on 636 as we have TLS also configured and enabled. Not sure if this is even relevant. From where does CM gets the LDAP URL? I tried to understand gen_credentials_ad.sh script at /opt/cloudera/cm/bin, however, did not completely interpret. Please help as this is bit urgent. Thanks snm1523

snm1523 · ‎11-23-2022

Hello All, I have just added a Ranger KMS service to a newly built CDP Cluster. When attempted to create key, got error stating "user not allowed to do create key". I thought it must be related to providing privs to the user in Ranger policies for KMS service and hence, logged in to the Ranger Admin UI where I can't see Ranger KMS service itself. I can see the plugin is enabled and responding in the plugins tab, cm_kms policy is syncing as per plugin status tab. However, service is not there. Any suggestions please. Thanks snm1523

snm1523 · ‎11-18-2022

Hello, I have recently built a CDP 7.1.7 SP1 cluster and have all the services running except YARN QueueManager WebApp. QM Store is running with no errors. Even in logs for WebApp, only entry i see is: WARN WebApp Properties Reading conf/webapp.properties. I am almost clueless what to do to bring this up. Any suggestions please. Thanks snm1523

snm1523 · ‎11-07-2022

Hi All, We are in process of migrating HDP cluster to CDP. As we would like to retain Atlas lineage in CDP from HDP, we have exported them from HDP cluster which gave us a .zip file. Extracted it and remediated the contents of .json files to match the CDP cluster name and server names, then zipped it back to a new_name.zip file. Now when running Atlas import API command to import remediated Atlas data to CDP Atlas, we are getting below error for most of the Atlas imports. {"errorCode":"ATLAS-404-00-007","errorMessage":"Invalid instance creation/updation parameters passed : type_name.entity_name: mandatory attribute value missing in type type_name"} Also, have some more errors, which certainly need to be looked into but not super priority. {"errorCode":"ATLAS-400-00-08A","errorMessage":"Attempting to import empty ZIP file."}: {"errorCode":"ATLAS-500-00-001","errorMessage":"java.lang.NullPointerException","errorCause":null} Having tried to figure this out since a week, but still no luck. Any suggestions / guidance will be really helpful. Thanks snm1523

snm1523 · ‎11-06-2022

Sent

snm1523 · ‎11-02-2022

Hello, I am working on importing multiple Atlas entities using Atlas API calls. After completing execution of each curl command, the message I get is "connection is left intact". Is there a way to kill / terminate this connection to Atlas server? Thanks snm1523

snm1523 · ‎10-17-2022

Hi, We are in process of setting up a CDP 7.1.7 SP1 cluster. We have got Knox enabled and configured across all services which works fine for all except Livy. When we attempt to submit a spark session via Livy service (Knox URL), it does not recognise the user session is getting submitted from and so returns "unauthorised error). There is a log entry in Knox gateway which clearly shows that the user name is not detected. However, when we submit a spark session directly via Livy URL with the same user, it passes through. This confirms definitely something is not correct in the Knox / Livy configuration. In this attempt, at the same place in Knox gateway logs, it has a POST entry which clearly displays the user name who has submitted this job. Digging it further, found that there are is no entry in simplified topology configuration for Livy. Not sure if this is an issue? Attached is the configuration. Secondly, we also found that in KNOX_DATA_DIR/services, we have a folder called Livy and it has 3 version folders. What sense this makes, not sure? Honestly, I do not understand what is the actual significance of these folders. It has rewrite and services.xml in it. Further referring to below article, did mentioned about creating these files in services folder, however, we are not sure how exactly that would help. Add custom service to existing descriptor in Apache Knox Proxy | CDP Private Cloud (cloudera.com) Is there a documentation that actually helps to understand all the steps that are involved in configuring Livy to work with Knox and explains the communications / interactions within these services? Also, any help in getting this setup correct would be really great. Additionally, how exactly topologies in Knox have an impact in this. Thanks snm1523

snm1523 · ‎06-30-2022

Hello, Is there a straight documentation available that would help to side car migrate Oozie jobs (50+) from HDP to CDP PB? I know of the properties file that might need some modification to point to CDP RM and relevant servers, however, facing it hard to understand / map properties file as we have 50+ workflows to be migrated. Thanks snm1523

snm1523 · ‎06-30-2022

Thank you @araujo, Do we also get a field to enter these details while installing Cloudera Manager at the page were we add custom repositories for Hadoop parcels? I don't remember it hence, asking. Thanks snm1523

snm1523 · ‎06-24-2022

Thank you for the response. I was able to find a way out to fix this.

snm1523 · ‎06-16-2022

Hi, I just completed setting up CDP Private Base in a POC environment. I was in process of attempting AD (LDAP based) integration so that users get authenticated via Active Directory. I am unsure if there was a mis configuration, however, after restarting Cloudera Manager, I can't login to it via "Admin" account (local). Thought it has got AD integrated, tried multiple accounts from AD, however, none of them are working. Please help to re-enable admin (local) account. Thanks snm1523

snm1523 · ‎06-16-2022

Hello, I am in process of setting up a CDP cluster (private base) and in step of configuring local repositories. I have got the Cloudera repos configured via out standard repository solution, Artifactory since servers will not have access to public internet to access Cloudera archives. Now the URL given to me to access those repos has to be authenticated via a User ID and API key. So URL ultimately turns out something like this: https://<user>:<API Key>@Repo URL/ Questions are: 1. Is there a way I can configure a authentication based YUM repository for Cloudera Manager packages, without having the need to enter credentials in a plain text in base URL field of YUM repo? 2. Once we have Cloudera Manager installed, we can provide a custom repo link in Cloudera Manager to fetch Cloudera Runtime Parcels. At that stage, will Cloudera Manager accept Environment variables in the URL or that again has to be a plain text? Else, is there any other way to setup authentication based Cloudera runtime repo which can be used in Cloudera Manager so we don't have to provide these creds in a plain text. Thanks snm1523

snm1523 · ‎04-08-2022

Hi All, I was able to get my script tested on my 10 nodes DEV cluster. Below are the results: 1. All HDP core services started / stopped okay 2. None of Hive Service Interactive service started and hence, Hive service was not marked as STARTED though HMS and HS2 were started okay 3. None of the Spark2_THRIFTSERVER was started Any one can share some thoughts on points 2 and 3? Thanks snm1523

snm1523 · ‎03-09-2022

Additional info @steven-matison, I checked further an API call for SPARK2 service and found a difference. Spark Thrift Server is reported as STARTED in the API output, however, on Ambari UI it is stopped. See the screenshots. Ambari UI: API Output: Any thoughts / suggestions. Thanks snm1523

snm1523 · ‎03-09-2022

@steven-matison Something like this. This time Spark, Kafka and Hive got stopped as expected, but since YARN took a longer to Stop (internal components were still getting stopped), moment script got service status of YARN as INSTALLED, it triggered HDFS and HDFS was waiting. Please refer to screenshot: What I want is to ensure previous service is completely started / stopped before the next one is triggered. Not sure if that is even possible. Thanks snm1523

snm1523 · ‎03-09-2022

Thank you for the response @steven-matison. I have 3 different scenarios so far: 1. Kafka, at times one or other Kafka Broker doesn't come up quickly. 2. Spark, Spark Thrift server is always not starting at the first place when bundled in an All services script. However, if called individually (only SPARK) works as expected. 3. Hive, we have around 6 Hive Interactive servers. HSI by nature takes a little long to start (they do start eventually though). In all the scenarios mentioned above, the moment there is an API call to start or stop service, the status in ServiceInfo field of API output changes to what is needed (i.e. Installed in case of Stop and Started in case of start), however, the underlying components are still doing their work to start / stop. Since, I am checking the status at the service level (reference below), the condition is passed and moves ahead. Ultimately I am in a situation of one service still starting / stopping and other is already triggered. str=$(curl -s -u $USER:$PASS http://{$HOST}/api/v1/clusters/$CLUSTER/services/$1) if [[ $str == *"INSTALLED"* ]] then finished=1 echo "\n$1 Stopped...\n" fi So far I have noticed this only for those 3 services. Hence, I am seeking suggestions on how do I overcome this OR if there is a way I could check status of each component of each service. Hope I was able to explain this better. Thanks snm1523

snm1523 · ‎03-09-2022

Thank you Andre. Wasn't aware. Actually I had used the steps mentioned in my reply during installation of Nifi. So thought might be useful in this case. Thanks Sunil

snm1523 · ‎03-08-2022

Hi, I am trying to perform a scripted start / stop of HDP services and its components. I need to get this done service by service, because there are few components like Hive Interactive / Spark Thrift server / Kafka broker, which does not get started / stopped in proper time. Most of the times it is observed that even if HSInteractive is still starting (visible in the background operations panel on Ambari), command moves to next service to start / stop and ultimately HSI fails. Hence, I also want to ensure that previous service is completely stopped / started, before attempting to stop / start next service in the list. To achieve, below is the script I have written (this is to stop, I have similar for start). However, when it reaches to the point for Hive or Spark or Kafka services, even though the the internal components like Hive Interactive or Spark Thrift server or Kafka broker are not started / stopped, it moves to the next service. Most of the times the start of Spark Thrift server fails via this script. However, if same API call is sent only for Spark or Hive individually it works as expected. Shell script for reference: USER='admin' PASS='admin' CLUSTER='xxxxxx' HOST='xxxxxx:8080' function stop(){ curl -s -u $USER:$PASS -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Stop '"$1"' via REST"}, "Body": {"ServiceInfo": {"state": "INSTALLED"}}}' http://$HOST/api/v1/clusters/$CLUSTER/services/$1 echo -e "\nWaiting for $1 to stop...\n" wait $1 "INSTALLED" maintOn $1 } function wait(){ finished=0 check=0 while [[] $finished -ne 1 ]] do str=$(curl -s -u $USER:$PASS http://{$HOST}/api/v1/clusters/$CLUSTER/services/$1) if [[ $str == *"INSTALLED"* ]] then finished=1 echo "\n$1 Stopped...\n" fi check=$((check+1)) sleep 3 done if [[ $check -eq 3 ]] then echo -e "\n${1} failed to stop after 3 attempts. Exiting...\n" exit $? fi } function maintOn(){ curl -u $USER:$PASS -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo":{"context":"Turn ON Maintenance Mode for $1 via Rest"},"Body":{"ServiceInfo":{"maintenance_state":"ON"}}}' http://$HOST/api/v1/clusters/$CLUSTER/services/$1 } stop AMBARI_INFRA_SOLR stop AMBARI_METRICS stop HDFS stop HIVE stop KAFKA stop MAPREDUCE2 stop SPARK2 stop YARN stop ZOOKEEPER Any help / guidance would be great. Thanks Snm1523

snm1523 · ‎03-08-2022

Hi @CookieCream, Should guide you on how do we get certificates generated. Apache NiFi Toolkit Guide Once you have this followed and certs generated, you will have a new nifi.properties created that will include truststore and keystore related properties. I have not tried anything on MacOS, but, I did saw there are some specific instructions for MacOS. Have a look Thanks snm1523

snm1523 · ‎03-02-2022

Here's your actual error: 2022-02-27 23:08:20,716 WARN [main] o.a.nifi.security.util.SslContextFactory Some truststore properties are populated (./conf/truststore.p12, ********, PKCS12) but not valid 2022-02-27 23:08:20,717 ERROR [main] o.apache.nifi.controller.FlowController Unable to start the flow controller because the TLS configuration was invalid: The truststore properties are not valid Ensure that you have generated certificates (SSL in your case, i assume). Add them to the truststore.jks of your Nifi instance (default location: ./conf/). Also, ensure the truststore and keystore properties in nifi.properties is accurately updated. This should help i guess. Thanks snm1523

snm1523 · ‎02-24-2022

Have you got a company wide CA cert or a SSL cert created and added that to the Nifi truststore? Also, if it is clustered, ensure that certificate of all cluster nodes are added to each others truststore. Additionally, also ensure that .keystore and .truststore properties in Nifi.properties are updated properly i.e. correct password and locations are entered. Thanks snm1523

snm1523 · ‎05-24-2021

Hello Vos, Please share an email address to send the doc. Thanks Snm1523

snm1523 · ‎05-14-2021

@tusharkathpal , got hit to another issue, working on that. Will revert with these results by Monday.

snm1523 · ‎05-14-2021

Hello @vidanimegh, We are not changing any permissions to users (including me) on default DB. They are just default (whatever they get once created). We are managing permissions using Sentry on each DB that is created. We have verified permissions are all okay in Sentry as users (including me) are able to see / query tables in Impala via Hue but not in Hive. Thanks snm1523

snm1523 · ‎05-14-2021

Hello @tusharkathpal, Thank you for the reply. Have verified all the users are in correct groups and same on all nodes. Please suggest what can be checked further. Thanks snm1523

snm1523 · ‎05-10-2021

Any suggestions please?

snm1523 · ‎03-24-2021

Hello @Bandit , Have sent you an email. Thanks snm1523

snm1523 · ‎03-23-2021

Hello All, We have a around 22 databases and their respective tables which are accessible via Impala in Hue, but not via Hive for 3 newly added users. We get below error which is related to permissions to databases via Sentry, however, this looks strange to me since the permissions are managed at DB level and not specific to service. So if permissions are not correct, we should not have been able to access them via Impala as well. Error message: Error while compiling statement: FAILED: SemanticException No valid privileges User xxxxxx does not have privileges for SWITCHDATABASE The required privileges: Server=server1->Db=*->Table=+->Column=*->action=select->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=insert->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=alter->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=create->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=drop->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=index->grantOption=false;Server=server1->Db=*->Table=+->Column=*->action=lock->grantOption=false; Query that works fine in Impala but not Hive: select * from dbname.tablename limit 5; Please suggest what can be checked / done to fix this. Thanks snm1523

snm1523 · ‎02-08-2021

Hello, We have a strange issue here. We are on CDH 6.3.0 and have sentry in place for authorizations. A user is trying to execute queries from Hue on Impala Editor and fails with Authorization errors. However, same query works fine via Hive and also via Impala shell. It fails only from Hue editor. We have: 1. Refreshed metadata 2. Invalidated metadata 3. Verified permissions are in place in Sentry 4. There are no logs generated (I checked in Hue, Sentry Server and Catalog server). Please suggest if any other place should also be checked. Example query: select * from DB.table limit 5; Kindly help to diagnose and resolve this issue. Thanks Snm1523

Online	Offline
Last Visited	‎01-31-2023 03:57 PM

Date Registered	‎10-29-2015 07:36 PM
Last Visited	‎01-31-2023 03:57 PM
Total Messages Posted	85
Total Kudos Received	10

Re: Enable Admin account for Cloudera Manager

Re: Datanode not starting: SIGTERM error

Re: MKDirs failed to create file

Re: URGENT: Enabling AD KDC on CDP 7.1.7

URGENT: Enabling AD KDC on CDP 7.1.7

Ranger KMS service not visible in Ranger Admin UI

Unable to start QueueManager WebApp

Atlas import is failing : HDP to CDP

Re: Integrating Apache Griffin to CDH Clusters

Kill Atlas API session after import command is com...

Knox / Livy communications / configuration

Migrating Oozie jobs from HDP to CDP PB

Re: Auth based local repository configuration for ...

Re: Enable Admin account for Cloudera Manager

Enable Admin account for Cloudera Manager

Auth based local repository configuration for CDP ...

Re: Scripted start / stop of HDP services

Re: Scripted start / stop of HDP services

Re: Scripted start / stop of HDP services

Re: Scripted start / stop of HDP services

Re: Nifi starts without error but cannot connect

Scripted start / stop of HDP services

Re: Nifi starts without error but cannot connect

Re: Nifi starts without error but cannot connect

Re: Setting up Nifi 1.15.3 on plain vanila linux b...

Re: Integrating Apache Griffin to CDH Clusters

Re: Strange permission Hive issue in accessing DBs

Re: Strange permission Hive issue in accessing DBs

Re: Strange permission Hive issue in accessing DBs

Re: Strange permission Hive issue in accessing DBs

Re: Integrating Apache Griffin to CDH Clusters

Strange permission Hive issue in accessing DBs

Authorization errors in Impala