Q&A

We have a SolrCloud service in Kerberos security enabled CDH 6 cluster. There a lot of collection in SolrCloud and Metric monitoring configuration is enabled, but there are NO DATA in all charts and collection name is not displaying under Collections dropdown in Collection Statistics tab.   How do we fix this situation?    

Hi All,   I've been using CM API V32 to update various service configuration and I noticed it works as long as we set service level attributes. However, while trying to update attributes at roleType level, the API responds with error code 400, stating "'Role type configuration is supported only up to API v2' in CM API V32.   Interestingly, the model schema provided by Cloudera recommends using the same attribute for the service config update.   I am using PUT `/clusters/{clusterName}/services/{serviceName}/config` API.   Anyone else facing a similar issue or knows how to deal with this issue?   Thanks.

Followed Hive metadata importing to atlas from the following link: https://atlas.apache.org/Hook-Hive.html Eventhough hive-site.xml contains  <property><name>datanucleus.schema.autoCreateTables</name><value>true</value></property> .    and I ran ./schematool -initSchema -dbType derby --verbose .  to initialize the db schemas, but still during importing the hive-dbs in atlas (./import-hive.sh) , I am getting this following error. But hive-db creation , deletion on its own works good, no issues. More so schematool info gives:Booting Derby (version The Apache Software Foundation - Apache Derby - 10.14.1.0 - (1808820)) instance a816c00e-016d-1feb-fbec-00000aae9110on database directory /opt/hive/bin/metastore_sql in READ ONLY mode with class loader sun.misc.Launcher$AppClassLoader@574caa3f.Loaded from file:/opt/hive/lib/derby-10.14.1.0.jar.java.vendor=Oracle Corporationjava.runtime.version=1.8.0_222-b

I am trying to install agent using Cloudera Express edition on different host. But getting following error   [12/Sep/2019 09:53:43 +0000] 20745 MainThread agent        ERROR    Heartbeating to omhs61p383.nbo.com:7182 failed.Traceback (most recent call last):  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat    self.cfg.max_cert_depth)  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__    self.conn.connect()  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect    sock.connect((self.host, self.port))  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect    ret = self.connect_ssl()  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL

Hi, I'm trying to get all entries from a redis cache. When i try to remove, getbykey etc that work but when i call myCache.keySet, i face an issue with following exception: UnsupportedOperationEception on null. I check it in git source repo and keySet method is implemented. May be there is a mistake in my deserializer ? I would like to know the point of view of Nifi community . I'm currently stuck as i need to list all entries that are in redis in order to split them in other processor for a specific action in Nifi context. BElow some piece of code : - The deserializer   public static class StringDeserializer implements Deserializer<String>{ @Override public String deserialize(byte[] input) throws DeserializationException, IOException { if (input == null || input.length == 0) { return null; } return input.toString(); } }    and piece of code of my custom processor code:   final DistributedMapCacheClient cache = cont

Hello Everyone,We have a Nifi production cluster made of 4 nodes from HDF 3.1.2.0, using our HDP Zookeeper. We had a major breakdown and had to restart all cluster and since the restart one of the node is unable to join the cluster again. Here is the log I have on it :2019-08-06 17:56:32,385 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave uporg.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:838) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.ut

NiFi processor updateAttributes runs too many tasks, but I just have only one flowfile transfers to it. I wonder why so many tasks are running.

Hello,   I've installed the sandbox-hdp docker-container on my ubuntu-server using Kitematic.The deploy process was quick and easy. The ports 22, 4200 and 8080 are forwarded to localhost on different ports (32785-32783).When trying to access these with my browser the page can't be reached.   Any tips? Do I need to use some other network adapter? Are other ports needed?   regards muffex

Using Cloudera Altus Director 6.3 to deploy CDH 6.3 to AWS. I submitted a similar post back in April, thinking that it had since been resolved - but apparently not.  We are ramping up CDH6 deployments again since our initial testing in April and again are seeing Hue fail to start during first run of CDH6 deployments due to Python lib folders being too restrictive.  Interestingly we do not see this issue when using the same CentOS 7.6 based AMI to deploy CDH 5.16.2 clusters.  The Python lib permissions differ on between the CDH5 and CDH6 deployments: CDH5:ls -l /usr/lib/python2.7/site-packages/six*-rw-r--r--. 1 root root 29664 Jan 2 2015 /usr/lib/python2.7/site-packages/six.py-rw-r--r--. 1 root root 29708 Nov 20 2015 /usr/lib/python2.7/site-packages/six.pyc-rw-r--r--. 1 root root 29708 Nov 20 2015 /usr/lib/python2.7/site-packages/six.pyo/usr/lib/python2.7/site-packages/six-1.9.0-py2.7.egg-info:total 16-rw-r--r--. 1 root root 1 Nov 20 2015 dependency_links.txt-rw

Can I access HDP Atlas without Ranger?I am new to HDP and so far I can use Hive, HBase, and Sqoop without Ranger installed. Now I am trying Atlas and following this link.https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/configuring-atlas/content/configure_atlas_security.htmlIt looks like I have to use Ranger foraccessing Atlas, and communication between Ranger and other services like Hive and HBase.Is this correct?Thanks.

while running a normal spark job SparkContext is not getting initialized giving below error: [Thread-6] ERROR org.apache.spark.SparkContext - Error initializing SparkContext.org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. I believe it is pointing to standby NN.This error is coming after we enabled Namenode HA in our cloudera's cluster.I have checked hive-site.xml, core-site.xml and spark-defaults.xml.These all are havinf correct configs pointing to nameservice id. 

I have the following configuration in hive-site.xml <configuration><property><!-- this should eventually be deprecated since the metastore should supply this --><name>hive.metastore.warehouse.dir</name><value>${WAREHOUSE_DIR}</value><description></description></property><property><name>hive.exec.scratchdir</name><value>${TMP_DIR}/scratchdir</value><description>Scratch space for Hive jobs</description></property><property><name>javax.jdo.option.ConnectionURL</name><value>jdbc:mysql://localhost:3306/metastore_db?createDatabaseIfNotExist=true</value></property><property><name>javax.jdo.option.ConnectionDriver</name><value>com.mysql.jdbc.Driver</value></property><property><name>javax.jdo.option.ConnectionUserName</name><value>root</value></property><property><name

I have an event  {"a":"avc123.abc.com","b":"24 - ny","c" : "21321kodi"}i would like to use extracttext to filter/extract only the "b" however, it is picking up the first match. How can I specify to extract text from field "b" only not "a" neither "c"

how can I extracttext specific key and value from json data ?{"key1" : "value1","key2" : "value2","key3" : "value3",}I would like to apply the extraxttext on Key2 and its value only. 

Does anyone know why the "database/table filter" and the "search saved documents" fields automatically populate with my username every time the page is reloaded? It takes me 3 button clicks to get rid of it every time just to be able to browse databases and tables, making my work incredibly frustrating.

Does anyone know if it is possible to exclude the Kafka service from the primary "Cluster Start/Stop" sequence in Cloudera Manager?  We need to stop the CDH cluster to perform maintenance on the MySQL metadata server, which affects Hive, Impala, Sentry, Hue, HBase?, Oozie, Navigator, etc... all of which we use.  Kafka is part if this CDH cluster.To stop the cluster, I would like to use the cluster "start/stop" menu item on the main CM page just for peace of mind in maintaining the proper shut down and startup sequence, but that would include Kafka going down as well, which we would like to avoid if possible.  Any thoughts are appreciated.  Thanks!

Hello! I have a student attempting to install Mongo into Cloudera.  The following error messages are being received.  I am not a coder and am unsure how to assist.  I am learning the language as I go along! Any ideas? Thank you! 

I have a JSON data coming as {"x":"avc123.abc.com"} I would like to have a processor in nifi to extract only the integer (123) from the entire value and output it from the process secondly, I would like that integer value (123) to be added in the JSON event with different keys such as the final JSON event would be {"x":"avc123.abc.com" "y":"123" } I already tried routetext and extract text but it didn't work.

Hi, I have e simple txt file for test: 1;God of War 12;God of War 23;God of War 34;God of War 45;God of War Chains of Olympus6;God of War Ghost of Sparta7;God of War Ascension8;Batman Arkhan Asylum9;Batman Arkhan City10;Batman Arkhan Knight I want get the values and split with a ExtractText processor. I create two properties in processor, id and desc id value -> (.*?); desc value -> .*?;(.*?); When the process goes to matched relationship, it get only id value, and desc property comes blank

Hi, Just wanted to know if we can setup and use KNIME (A visual data analytics tool). I've tried to install in HDP 3.0 with the help of link "https://docs.knime.com/2018-12/server_installation_guide/index.html". Installation was successful but when I was trying to use link sandbox-hdf.hortonworks.com:8090, it was not connecting. In error log file I could see that "tomcat" was not able to connect localhost:8005. Not sure if this was the reason. Please let me know if anyone used KNIME in HDP sandbox. Or any Visual Data Analytics tool you have used.Thanks.

Hello,   I'm just starting off and was wondering if there's any concrete way of setting the compression when writing to a file in Spark?   I used to use option when writing files:   exampleDF.write.option("compression", "snappy").avro("output path")   but when I go to check where the Avro files are saved I can't tell from the name of the files whether they've been compressed or not. Also just to say this is after I've imported "com.databricks.spark.avro._" so I'm not having any trouble using Avro files.   Another way I've seen is to use "sqlContext.setConf" and these would be the commands I'd use in this instance:   import org.apache.spark.sql.hive.HiveContext val sqlContext = new HiveContext(sc) sqlContext.setConf("spark.sql.avro.compression.codec", "snappy") exampleDF.write.avro("output path")   Neither way causes any errors so I was wondering which would be the better way and if there are any other more reliable ways of setting the compression when

hi, what data type should we use for general home address in hive???

Hello All, First of all, I would like to thank community. I am trying my hand in NiFi and I came across a situation that confuses me. I am trying to read a csv file in NiFi using ExecuteScript(yes, I want to use this) processor. All the good records must be collected in one flowFile and the bad records are collected in one flowFile each (this way, I will easily know how many bad records exist.) and then route them to SUCCESS and FAILURE relations accordingly. The script works fine when all the records are good. I deliberately inserted one bad record to test exception and it is not working. As per my knowledge, NiFi should throw ValueError but it throws NullPointerException. Not sure what I am missing. I am new to this and learning. any guidance will be much appreciated. Input:1,2,3,41,A,3,41,2,3,4 Code import jsonimport java.iofrom org.apache.commons.io import IOUtilsfrom java.nio.charset import StandardCharsetsfrom org.apache.nifi.processor.io import StreamCal

I have configured Ranger with Ranger-admin and Hadoop , HDFS  plugins.  I can access the basic url,curl -u rangeradmin:rangeradmin -i -X GET http://127.0.0.1:6080HTTP/1.1 302 FoundSet-Cookie: RANGERADMINSESSIONID=6A1E7311C911AE876D02359679222E24; Path=/; HttpOnlyX-Frame-Options: DENYCache-Control: no-cache, no-store, max-age=0, must-revalidatePragma: no-cacheExpires: 0X-XSS-Protection: 1; mode=blockX-Content-Type-Options: nosniffContent-Security-Policy: default-src 'none'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; connect-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline';font-src 'self'Location: http://127.0.0.1:6080/login.jspContent-Length: 0Date: Thu, 05 Sep 2019 09:02:16 GMTServer: Apache Ranger but on accessing other urls it fails:  curl -u rangeradmin:rangeradmin -i -X GET http://127.0.0.1:6080/service/plugins/policies/service/1HTTP/1.1 401 UnauthorizedSet-Cookie: RANGERADMINSESSIONID=96A517FCD3CDC23354BFDFE3D22E6FE4; Path=/; HttpOnl

Yarn Application status was failed, and displayed follows.Diagnostics: Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143.And Yarn Log Entries includes following message.ERROR CoarseGrainedExecutorBackend: Executor self-exiting due to : Driver XX.XX.XX.XX:46869 disassociated! Shutting down.INFO DiskBlockManager: Shutdown hook calledCould you tell me the checkpoint to solve this problem?Best Regards.