Member since
06-26-2019
68
Posts
8
Kudos Received
6
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1836 | 04-20-2018 09:51 PM | |
2366 | 02-08-2018 01:27 AM | |
653 | 01-31-2018 06:39 PM | |
2798 | 01-31-2018 07:27 AM | |
797 | 01-17-2018 09:37 PM |
03-05-2020
11:43 PM
How would anyone with an on-premise HDP 2.6.5 opensource version be able to upgrade to CDP7 without a subscription ?
... View more
03-05-2020
11:41 PM
@SushantRao where is the source code for Cloudera Manager and all the other CDH7 components ? Is there documentation available to locate and build the packages ?
... View more
03-05-2020
11:38 PM
1 Kudo
@Tim Armstrong that is great ! Is the source now available ? and the documentation to build the source ? I am pretty sure the community will simply rally around and build something up with enough pointers. I agree with @PieterB hiding the base binaries is silly. Did the HDP model have any failures ? Enterprises which needed support bought it anyway. Keeping it free increased adoption.
... View more
12-05-2019
09:15 AM
Yes thanks. Hopefully as part of the first release you would have documentation available on how to build the binaries as well.
... View more
12-03-2019
03:50 PM
@SushantRao Can this be explained in plain english ? I am a bit confused with the legal terms here. https://www.cloudera.com/products/faq.html The document states both that a subscription will be required for access to cloudera-hosted components and that the source code will be open-source. It states that the open-source license will come into effect around February 2020. For the set of companies or developers who don't need maintenance, support , training and consultancy i.e they do not need subscription but want to deploy the software to production where will the open-source code be hosted ? on apache ? So does it imply that the open-source binaries will NOT be cloudera-hosted and we have to wait till February 2020 ? Or are you suggesting that a subscription is needed to access the "open-source" code from which the binaries would then need to be built ?
... View more
12-03-2019
11:21 AM
1 Kudo
@SushantRao @Cloudera Can we have a response on this ? We are planning for future deployments and this information will help us scope out and plan our architecture rather than pivoting to a different model.
... View more
12-02-2019
05:17 PM
@SushantRao as @Shelton mentioned we are talking about the open-source CDP distribution that we can download and install on bare-metal as per cloudera's commitment to open-source.
... View more
11-21-2019
03:08 PM
@LakshmiR thanks. You mention HDFS is expected to be local for WALs. So are you suggesting that there could be another non-local HDFS configured for storing the hfiles ? So hfiles can be decoupled from the regions and essentially regionServers. If this is possible then one can potentially size a region server to serve the compute requests from a storage layer, the non-local hdfs and HBASE can then essentially boot up from scratch from the non-local hdfs and keep scaling independently. This does introduce latency in-terms of the region servers serving requests from a non-local hdfs to the client but as long as the reads on hdfs hold and the network as well .. it may be fine. Which version of CDH supports this .. is it starting from CDP ?
... View more
11-21-2019
02:51 PM
1 Kudo
@sagarshimpi HDP already supported Node Labels. However CDH thought that it was not production ready. https://community.cloudera.com/t5/Support-Questions/Node-Labels/m-p/37275/highlight/true# The question is whether node lable support is in the roadmap for CDP as CDH and HDP have joined. It does open up interesting use-cases for mixed cluster use.
... View more
11-19-2019
10:07 PM
Will Node labels be supported in the new CDP ?
... View more
Labels:
- Labels:
-
Cloudera Data Platform (CDP)
11-19-2019
10:03 PM
In the latest webinar for accelerate your time-to-insight with CDP Data Center it showcased the potential of compute clusters.
Is HBase also supported as a compute cluster i.e will it be able to use the shared DataContext of HDFS and have non-local region servers .. does that even work as a concept ?
Or would the idea be for HBase to have its own local hdfs because region servers are better of being data-local. The problem for scaling HBase is that the region servers and region capacities are tied to the datanodes but then it is possible to have a lot of storage on less data nodes which does not equate well for hbase profile datanodes.
The next question would probably then be for YARN could that be a separate compute cluster say for MR2 which uses the HBase Compute Cluster as a DataContext ?
... View more
Labels:
- Labels:
-
Cloudera Data Platform (CDP)
09-24-2019
11:41 PM
https://www.cloudera.com/products/pricing.html
Where is cloudera's promise and committment to open-source ?
How are we supposed to use the CDP stack on an on-premise solution ?
When is the release which supports on-premise deployments going to be available ?
... View more
Labels:
- Labels:
-
Cloudera Data Platform (CDP)
09-28-2018
04:15 AM
This helped resolve our issue of spiking on a migration from oracle jdk 8 to open jdk 8 on centos 6.9
... View more
04-20-2018
09:51 PM
1 Kudo
When in doubt; doubt SSL. After several errors and trials, the core issue was that the alias in keystore for ranger was incorrect. What was really surprising that tomcat did not throw any errors whatsoever but just failed to start listening on port 6182. Increasing the debug level logs for several pacakges in the log4j for ranger-admin-env and even in /usd/hdp/current/..../ews/WEB-INF/ .... did inot show any error. Usually we have seen errors in a normal tomcat ssl setup. It was very surprising that no error was thrown. The only error was that it did not boot up to listen on port 6182. strange.
... View more
04-20-2018
09:47 PM
@Felix Albani Thanks, that did help us (i.e we were able to create the amb_ranger_admin user). However none of the plugins were registered i.e because the plugins kept complaining of wrong password. Wanted to add for a future user, that the core reason was amb_ranger_admin password has some requirements on what the password should be essentially alphanumeric and a length about 8 i think. It should probably not have special characters. That was the reason why plugins did not work.
... View more
04-20-2018
03:02 PM
Thanks will give that a shot. In our case the amb_ranger_admin ussr is not created automatically. Going through a few other posts have checked enabling plugins restarting hdfs. Ranger etc. The user is created as part of ranger install or firat startup ? What does it mean of no iser is created ? Can i create the user manually by logging into ranger as admin under user/group s...how do i force the creation of the user ?
... View more
04-20-2018
06:57 AM
Is it possible to update the amb_ranger_admin password ? currently my cluster is a fresh install and all services state that the amb_ranger_admin as provided is incorrect. The documentation states that the password is only provided during install time. So if I update in the ranger config (and am not using kerberos) .. then the documentation states that I will need to go to each individual componet which has the ranger plugin and udpate the password .. where exactly ? https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_Ranger_Install_Guide/content/updating_ranger_admin_passwords.html Also is the password store in a .jceks file ? can it be viewed ? Going through other posts it seems that restarting ranger server / a component with plugin enabled would force create the amb_ranger_admin user .. but that does not seem to be happening ..
... View more
Labels:
- Labels:
-
Apache Ranger
04-19-2018
02:54 PM
@Felix Albani the formatting was off .. have cleaned it up. Thanks ! $ sudo ps -ef | grep rangeradminranger
2009 1 0 07:10 ? 00:01:42 java -Dproc_rangeradmin -XX:MaxPermSize=256m -Xmx1024m -Xms1024m -Duser.timezone=UTC -Dservername=rangeradmin -Dlogdir=/var/log/ranger/admin -Dcatalina.base=/data1/hdp/2.6.4.0-91/ranger-admin/ews -cp /data1/hdp/2.6.4.0-91/ranger-admin/ews/webapp/WEB-INF/classes/conf:/data1/hdp/2.6.4.0-91/ranger-admin/ews/lib/*:/data1/hdp/2.6.4.0-91/ranger-admin/ews/ranger_jaas/*:/data1/hdp/2.6.4.0-91/ranger-admin/ews/webapp/WEB-INF/classes/conf/ranger_jaas:/usr/java/latest/lib/*:/*: org.apache.ranger.server.tomcat.EmbeddedServer $ sudo netstat -anp | grep 2009
tcp 0 0 127.0.0.1:6085 0.0.0.0:* LISTEN 2009/java
tcp 0 0 10.108.10.112:44131 10.128.30.110:5432 ESTABLISHED 2009/java
tcp 0 0 10.108.10.112:44132 10.128.30.110:5432 ESTABLISHED 2009/java
tcp 0 0 10.108.10.112:44130 10.128.30.110:5432 ESTABLISHED 2009/java
tcp 0 0 10.108.10.112:44138 10.128.30.110:5432 ESTABLISHED 2009/java
tcp 0 0 10.108.10.112:44139 10.128.30.110:5432 ESTABLISHED 2009/java
unix 2 [ ] STREAM CONNECTED 9339763 2009/java</property> $ grep https -C2 /data1/hdp/2.6.4.0-91/ranger-admin/conf/ranger-admin-site.xml
<property>
<name>ranger.externalurl</name>
<value>https://myserver:6182</value>
</property>
<property>
<name>ranger.https.attrib.keystore.file</name>
<value>/path/to/key/keystore.jks</value>
</property>
<property>
<name>ranger.service.https.attrib.client.auth</name>
<value>want</value>
</property>
<property>
<name>ranger.service.https.attrib.clientAuth</name>
<value>want</value>
</property>
<property>
<name>ranger.service.https.attrib.keystore.credential.alias</name>
<value>keyStoreCredentialAlias</value>
</property>
<property>
<name>ranger.service.https.attrib.keystore.file</name>
<value>/path/to/key/keystore.jks</value>
</property>
<property>
<name>ranger.service.https.attrib.keystore.keyalias</name>
<value>my_wildcard_alias</value>
</property>
<property>
<name>ranger.service.https.attrib.keystore.pass</name>
<value>_</value>
</property>
<property>
<name>ranger.service.https.attrib.ssl.enabled</name>
<value>true</value>
</property>
<property>
<name>ranger.service.https.port</name>
<value>6182</value>
</property>
<br>
$ ls -l /data1/hdp/2.6.4.0-91/ranger-admin/ews/
total 36
drwxr-xr-x 2 ranger ranger 4096 Apr 19 07:09 lib
lrwxrwxrwx 1 ranger ranger 21 Apr 19 00:23 logs -> /var/log/ranger/admin
-r-xr--r-- 1 ranger ranger 2192 Jan 4 10:47 ranger-admin-initd
-r-xr--r-- 1 ranger ranger 6347 Jan 4 10:47 ranger-admin-services.sh
lrwxrwxrwx 1 ranger ranger 58 Apr 19 00:23 ranger-admin-start -> /usr/hdp/2.6.4.0-91/ranger-admin/ews/start-ranger-admin.sh
lrwxrwxrwx 1 ranger ranger 57 Apr 19 00:23 ranger-admin-stop -> /usr/hdp/2.6.4.0-91/ranger-admin/ews/stop-ranger-admin.sh
drwxr-xr-x 2 ranger ranger 4096 Apr 19 00:23 ranger_jaas
-r-xr--r-- 1 ranger ranger 971 Jan 4 10:47 start-ranger-admin.sh
-r-xr--r-- 1 ranger ranger 969 Jan 4 10:47 stop-ranger-admin.sh
drwxr-xr-x 10 ranger ranger 4096 Apr 19 00:24 webapp
drwxr-xr-x 3 ranger ranger 4096 Apr 19 00:33 work
<br>
... View more
04-19-2018
07:03 AM
Hi We have gone through the entire process of automating our cluster using blueprints and have had several successful deployments using wild cart certs in all our environments. We recently hit a snag in one of our larger environments where the ranger-admin though successfully installed with no errors whatsoever does not init the embedded tomcat server to listen on port 6182 when configured for ssl. on a similar environment this is from the catalina.out on /var/log/ranger/admin Apr 19, 2018 6:02:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/data1/hdp/2.6.4.0-91/ranger-admin/ews/webapp] .....
Apr 19, 2018 6:02:07 AM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Apr 19, 2018 6:02:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/data1/hdp/2.6.4.0-91/ranger-admin/ews/webapp].
Apr 19, 2018 6:02:08 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-6182"]
Apr 19, 2018 6:02:08 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Tomcat
Apr 19, 2018 6:02:08 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.81
Apr 19, 2018 6:02:08 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/data1/hdp/2.6.4.0-91/ranger-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
but on the environment on which we have the problem the logs are as such ava HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
log4j:WARN No appenders could be found for logger (org.apache.tomcat.util.IntrospectionUtils).
log4j:WARN Please initialize the log4j system properly.log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementationSLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Apr 19, 2018 6:09:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/data1/hdp/2.6.4.0-91/ranger-admin/ews/webapp
Apr 19, 2018 6:09:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/data1/hdp/2.6.4.0-91/ranger-admin/ews/webapp, webAppName = /
Apr 19, 2018 6:09:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/data1/hdp/2.6.4.0-91/ranger-admin/ews/webapp] .....
Apr 19, 2018 6:09:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/data1/hdp/2.6.4.0-91/ranger-admin/ews/webapp].
log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.
Apr 19, 2018 6:10:10 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Apr 19, 2018 6:10:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
Note the missing logs Apr 19, 2018 6:02:08 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-6182"]
Apr 19, 2018 6:02:08 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Tomcat The environments are similar in setup and there is no difference. Tried to enable debugging by setting debug level for apache.ranger and spring.frameworks in log4j.xml and from ranger-admin-log4j.xml in the ui. Still there were no errors. Tomcat does not listen on port 6182 and the work directory under /usr/hdp/.../ranger/admin/ews/ ...doesnt have anything ? Any suggestions on how to further debug this ? (apart from removing the service and re-installing) from the xa_portal.log it looks like the spring application context gets initialized ..which is wierd .. all service install but can't connect to ranger on port 6182 because it is not listening on port 6182 !
... View more
Labels:
- Labels:
-
Apache Ranger
03-03-2018
05:42 AM
As a hack I am specifying the port in the host name and that seems to work. Note the db_host below {
"kms-properties" : {
"properties" : {
"KMS_MASTER_KEY_PASSWD" : "foo",
"DB_FLAVOR" : "POSTGRES",
"db_name" : "rangerkms",
"db_user" : "rangerkms",
"db_password" : "foo",
"REPOSITORY_CONFIG_USERNAME" : "keyadmin",
"db_host" : "mydb.server.com:7432"
}
}
}
... View more
03-03-2018
01:45 AM
As part of this design I want ranger-kms to be separated from the other databases. So obviously if I just switch ports then ranger-kms gets installed but ranger admin does not because it uses the same library without the port !
... View more
03-03-2018
01:06 AM
Hi I have managed to have a successful deployment of a full blown ambari-stack using blueprints. As part of the next level of our security requirements I am attempting to make rangerkms run in its own database instance of postgres on a non-default port. The corresponding blueprint entry {
"dbks-site" : {
"properties_attributes" : { },
"properties" : {
"ranger.ks.hsm.enabled" : "false",
"hadoop.kms.blacklist.DECRYPT_EEK" : "hdfs",
"ranger.ks.jpa.jdbc.credential.alias" : "ranger.ks.jdbc.password",
"ranger.ks.jpa.jdbc.url" : "jdbc:postgresql://mydb.server.com:7432/rangerkms",
"ranger.ks.jpa.jdbc.driver" : "org.postgresql.Driver"
}
}
}
During blueprint installation the jisql commands attempt to verify the connection to the database but it doesnt seem to take into account the port that i have configured resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-python-wrap /usr/hdp/current/ranger-kms/db_setup.py' returned 1. 2018-03-03 00:22:47,004 [I] DB FLAVOR :POSTGRES
2018-03-03 00:22:47,005 [I] --------- Verifying Ranger DB connection ---------
2018-03-03 00:22:47,005 [I] Checking connection
2018-03-03 00:22:47,005 [JISQL] /usr/java/latest/bin/java -cp /usr/hdp/current/ranger-kms/ews/webapp/lib/postgresql-jdbc.jar:/usr/hdp/current/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://mydb.server.com/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
SQLException : SQL state: 28000 org.postgresql.util.PSQLException: FATAL: no pg_hba.conf entry for host "192.168.10.21", user "rangerkms", database "rangerkms", SSL off ErrorCode: 0
2018-03-03 00:22:47,182 [E] Can't establish connection However on the same host where kms is being attempted to install say foo.server.com if i specify the port the connection is successful /usr/java/latest/bin/java -cp /usr/hdp/current/ranger-kms/ews/webapp/lib/postgresql-jdbc.jar:/usr/hdp/current/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://mydb.server.com:7432/rangerkms -u rangerkms -p 'rangerkms' -noheader -trim -c \; -query "SELECT 1;" 1 | How do i get the db connection verifier to use the specified port in the blueprint ? I do have an entry in my pg_hba conf for foo.server.com to access rangerkms db as user rangerkms ..
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Ranger
02-19-2018
06:16 PM
Download the vdf file wget http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.3.0/HDP-2.6.3.0-235.xml -O /tmp/HDP-2.6.3.0-235.xml Post it curl -v -k -u admin:admin -H "X-Requested-By:ambari" -X POST \
http://ambari-server:8080/api/v1/version_definitions \
-d '{
"VersionDefinition": {
"version_url": "file:/tmp/HDP-2.6.3.0-235.xml"
}
}' the response would be {
"href" : "http://ambari-server:8080/api/v1/version_definitions/1",
"VersionDefinition" : {
"id" : 1,
"stack_name" : "HDP",
"stack_version" : "2.6"
}
} version definitions http://ambari-server:8080/api/v1/version_definitions/1
# can use id or version in the blueprint
## id "repository_version_id": "1"
## version "repository_version" : "2.6.3.0-235",
... View more
02-16-2018
06:43 PM
I would need a knox gateway to set this up ? a simple start may be just 1 way ssl between ambari-server and a ssl enabled postgres db .. where do i provide the configurations for the postgres jdbc driver so that the ssl handshake happens ?
... View more
02-15-2018
11:27 PM
1 Kudo
Cannot find relevant documentation which shows 2-way ssl setup between the ambari-server-database (e.g postgres) and other Ambari (ambari-server)/ HDP(ranger,hive etc) components ... is this supported ?
... View more
Labels:
- Labels:
-
Apache Ambari
02-15-2018
09:44 PM
When you install blueprint you will need to register a vdf file https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-release-notes/content/ambari_relnotes-2.6.0.0-behavioral-changes.html so whatever version u have in there for the repos will be the one which will be installed.
... View more
02-08-2018
01:27 AM
The root cause of the issue was that the intermediate AND the root certificates were not imported into the server keystores. Took a bit of debugging the source to figure it out but it worked in the end. There were a couple of hiccups in terms of what ambari blueprints automates in terms of policy configurations vs what it does not. Also need to ensure that commonNameForCertificate is set appropriately to the alias of the certificate.
... View more
02-08-2018
01:25 AM
Thanks, was able to solve our problem as well. It was related to not importing the intermediate AND the root certificates into the server keystores. Took a bit of debugging the source to figure it out but it worked in the end.
... View more