Q&A

Hello,I would like to filter some activities for 15 days, but not the last 15 (for example: from the 1st to the 15th and today is the 28th). how would I filter this in Cloudera Navigator?

Hello Team, Could you please help with the list of tools that are available in the exam and link of the documentation which can be used in the exam. ThanksSarath

I want the json split on the "DATA_MESSAGE", but note that some of them have multiple logEvents and should be included regardless of the number of them.  There are no outer "[ ]" in these flowfiles so $ or $.* doesn't work properly.{    "messageType": "DATA_MESSAGE",    "owner": "540804524916",    "logGroup": "/awg-p-deploy/aws/ec2/syslogs",    "logStream": "540804524916/i-01918034325752bbd/ip-10-170-16-11.xxxxxxx.xxx.org/var/log/audit/audit.log ",    "subscriptionFilters": ["awg-p-deploy-kinesis-destination-subscription-SubscriptionFilter-SOC692HYHJN8"],    "logEvents": [{            "id": "35604816732673909304179019342583609767225770179215556608",            "timestamp": 1596575200322,            "message": "type=SYSCALL ms

Hi everyone,Let say my existing Cloudera manager is managing a 10 node cluster. What happen if my cloudera manager server is down?(my cloudera management service is on diffrerent server and I'm using external database for the setup).  Can I immediately setup a new server, reinstall cloudera manager here, and manage the existing cluster without too much effort on the configuration and setup? I'm worried about the TLS setting, keberos setting,etc. Can it be preserve and work the way it was before the cloudera manager is replaced I believe the data in the HDFS should be safe right?  Thanks

While trying to install HDP 2.6.5 sandbox on windows 10 with wmare, after starting the vm with 4cpu and 10g ram, i can login the ambari but all service can't be started I have tried "restart all" service or restart zookeeper via the account: raj_ops, but no luck.  

I have installed apache metron version 0.7.2 using cloud installation, it created 6 nodes  and everything running fine but when i move to Metron Management UI it asks me for the password. I tried "Metron/Metron", "metron/metron", "admin/admin" etc but still unable to login. I tried metron user guides to find default password for metron management UI & metron alerts  UI but not able to find any document that gives or guides to set default password for metron.Is there anyone who tried this? Can anyone help me for default password for metron management UI & metron alerts  UI.

I have 2 NIFI instance on differents servers with 1 NGINX reverse proxy in front.It should go like this:NGINX Reverse proxy > proxy.lan/nifi1 goes to the NIFI 1 instance on nifi1.lanNGINX Reverse proxy > proxy.lan/nifi2 goes to the NIFI 2 instance on nifi2.lan NGINX reverse proxy runs on 192.168.1.114:80 NIFI runs on 192.168.1.153:8080 My configuration file nginx.conf:server {listen 80server_name 192.168.1.114;location /nifi1 {proxy_set_header X-ProxyScheme "http";proxy_set_header X-ProxyHost "192.168.1.114";proxy_set_header X-ProxyPort "80";proxy_set_header X-ProxyContextPath "/nifi1";proxy_pass http://192.168.1.153:8080;}} This is my NIFI nifi.properties file:nifi.web.proxy.content.path=/nifi1nifi.web.proxy.host=192.168.1.114:80 When accessing http://192.168.1.114/nifi1 , I will get the message "Did you mean /nifi" but the page looks off. It's missing the styling of the page. I get redirected after 5 secs to http://192.168.1.114/nifi1/nifi  an

Hello everyone,   we setup a Cloudera Environment which inherits a DataHub of type "7.1.0 - Data Engineering: Apache Spark, Apache Hive, Apache Oozie". We managed to connect to HIVE via a JDBC connection from our local machines. But so far we were not able to connect from CML to HIVE via JDBC I use the JayDeBeApi as follows:       conn_hive = jaydebeapi.connect('org.apache.hive.jdbc.HiveDriver', 'jdbc:hive2://dataengineering-master0.......:443/;ssl=1;transportMode=http;httpPath=dataengineering/cdp-proxy-api/hive;AuthMech=3;', \ {'UID': "user_name", 'PWD': "password"}, '/home/cdsw/drivers/HiveJDBC41.jar',)       The error message is       TypeError: Class org.apache.hive.jdbc.HiveDriver is not found       I set the environment variable CLASSPATH to '/home/cdsw/drivers/HiveJDBC41.jar' which is were the jar actually rests. Hence I wanted to check if JAVA_HOME is set correctly and yes, there the env. variable is set to '/us

Dear All,I am encountering issue with LDAP integration.I have completed LDAP (ldap-provider) and Certificate configurations according to documentation.I added IU certificate in NiFi (keystore, truststore etc.) and configured other pieces of the MS AD LDAP integration (authorizers.xml, login-identity-providers.xml and nifi.properties).I logged on NiFi from HTTPS UI with initial admin (admin1) and assigned the policies one of the LDAP users (nifiadmin) which is located on MS AD LDAP.I checked LDAP user (nifiadmin) from NiFi UI it is exist in the NiFi. It seems Ok. I added all screenshots (nifi_policies.jpg) about that.When I try to login initial admin (admin1) there is no error:nifi-user.log:2020-08-13 10:46:43,544 INFO [main] o.a.n.a.FileUserGroupProvider Users/Groups file loaded at Thu Aug 13 10:46:43 MSK 20202020-08-13 10:46:43,684 INFO [main] o.a.n.a.FileAccessPolicyProvider Authorizations file loaded at Thu Aug 13 10:46:43 MSK 20202020-08-13 11:21:28,051 INFO [NiFi Web Server-22] o.

Hello all,   We are seeing the next error in our infra-solr server logs (hortonworks HDP-3.1.4.0, infra-solr 0.1.0).   2020-09-23T16:33:00,258 [qtp2039810346-189141] ERROR [c:audit_logs s:shard2 r:core_node10 x:audit_logs_shard2_replica_n9] org.apache.solr.common.SolrException (SolrException.java:148) - org.apache.solr.common.SolrException: ERROR: [doc=697665cf-bc6b-4ada-a94e-e14ede1ba18c] Error adding field 'result'='true' msg=For input string: "true"at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:218)at org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:102)at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:967)at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:341)at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:288)at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:235)at or

Hi everyone, How do you guys install/manage Druid through Cloudera Manager? Been trying to find the parcel or CSD file for installation without any luck.    Is this option coming soon or you just install druid directly on the servers. I understand that Druid was one of the components that can be install on Ambari previously. Is this going to be the same on Cloudera Data Platform or Cloudera CDH version 5 or 6.   Thank you very much.

If I do echo $JAVA_HOME I get /usr/lib/jvm/java-8-openjdk-amd64 as a response, yet I cannot get nifi to see it when it starts, always gives this:nifi.sh: JAVA_HOME not set; results may varyJava home:NiFi home: /opt/nifi-1.12.0 This is on a clean Ubuntu 18.04 installation, having followed these instructions: https://techexpert.tips/apache-nifi/apache-nifi-installation-ubuntu-linux/At this point really not sure what to do.

I am running the command from hdfs user hdfs@master>sqoop import-all-tables --connect jdbc:mysql://10.11.11.15:6306/siki_asmet?serverTimezone=Asia/Jakarta --username micronics -P --hive-import --warehouse-dir /warehouse/siki --hive-database siki_ods --exclude-tables "Sheet1$" --m 1; When i run the above command, i get the following error:  FAILED: SemanticException Line 1:17 Invalid path ''hdfs://master.lpjk.com:8020/warehouse/siki/_asdamkindo_personal_ska_pendidikan'': No files matching path hdfs://master.lpjk.com:8020/warehouse/siki/_asdamkindo_personal_ska_pendidikan (state=42000,code=40000) But when i try to run the command again, it said file already exists. 

I'm using a PutSNS processor which puts JSON (from API call) into SNS, in some cases JSON is more than 256KB, and I get "Size exceeds AWS SNS Limit of 256KB". Is there a way to chunk this into smaller sets?

Hi, I have below JSON file with multiples image's details{    "images": [{        "id": "C2934",        "ds_folder_id": "759838458734"    },    {        "id": "C8343",        "folder_id": "3942879137986"    }]}. I have id of the image and have to update the folder_id with some value. For Example:id is C8343 and the new folder_id is 2984584735then I have to update folder_id of the JSON file with new one. New JSON would be{    "images": [{        "id": "C2934",        "ds_folder_id": "759838458734"    },    {        "id": "C8343",        "folder_id": "2984584735"    }]}. Please help me to perform above in NiFi.

Hi Community, I'm looking for the training materials for CDH.I think there's no public materials about CDH. They seems to be for CDP, new products.(https://www.cloudera.com/about/training.html )I also tried to check the partner portal but it's 404 link. https://www.cloudera.com/partner-portal/training.html I have to upgrade CDH v5.16 to v6.3.3 and I'd like to learn the whole functions about CDH.(the ones written in Japanese are better if possible but I'm OK with them in English) Best regards,Takuya Kohama

Hello.I have activity monitor in green state.In logs I have smth like this:ps Sorry for quality, have no internet access in my cluster.But when I open navigator web ui I see this:

we want to insert flowfile attributes(Filename,FileSize,Time) into Postgres Table Using NiFi

Exception in thread "main" java.lang.reflect.InvocationTargetExceptionat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:294)at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)Caused by: java.io.FileNotFoundException: Log directory specified does not exist: hdfs://nameservice1/user/spark/applicationHistoryat org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:268)at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:212)at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHisto

Hi ,We are currently upgrading from HDP 2.3.4 to HDP 2.6.5 following the following reference document as suggested in the following link.https://docs.cloudera.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-upgrade/content/ambari_upgrade_guide.htmlOurs is an environment without internet connection so we downloaded and created the offline repository.We already upgraded the ambari from 2.4.2 to Ambari 2.6.2.2 successfully and performed all the prerequisites for HDP upgrade. We are currently got stuck in the "Register and Install Target Version" phase. We have registered the local repository for HDP 2.6.5 and from ambari>stacks and versions>manage version we are trying to install the new softwares which is failing with the following error message.This issue is only happening for only one of the hosts in cluster.Rest all other hosts the installation was successful."Package Manager failed to install packages. Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Noth

Hello!   Please help me about minimal hardware requirements for out small cluster.   We decided to make very small production cluster with high availability for archiving purposes based on Cloudera CDH 6.3.3 (community version) Storage size planning as about of 10-20 TiB Workload planning: - every 2 minutes ETL from external oracle to local parquet about of 500-1000 rows of data - periodically (very rarely) analytic queries to hive about search through all of parquets - periodically (very rarely) run spark ad-hoc tasks with goals same as above Components: - Cloudera manager - HDFS - Hive - Hue - Impala - Spark - Yarn with MR2 - ZooKeeper - Streamsets parcel (as a part of cloudera) We want to use only 3 hosts (not more) and disaster of any of this host must not crash all system. So we plan to place all of above components to all of the hosts. In another words, each component will be on each host. Is it normal and available or someone may advise another alignment? We a

Dear experts,I am running HDP 2.6.4. Recently i have installed Ranger and integrated Hive. When i am trying to change Ambari Cluster Name and Ranger Hive Service Name, i am receiving the below errors in the hiveserver2.log. Could you please help ? 'cluster1' is the original cluster name,I want to change the cluster name to a different name. ERROR [Thread-13]: util.PolicyRefresher(PolicyRefresher.java:loadPolicyfromPolicyAdmin(285)) - PolicyRefresher(serviceName=cluster1_hive): failed to find service. Will clean up local cache of policies (-1)org.apache.ranger.plugin.util.RangerServiceNotFoundException: cluster1_hive   

Hello.I need advice there I can see log(audit) of hdfs operations.For example(mkdir, putFromLocal, delete file,change file).I tried use command from cmd "hdfs dfs -mkdir /tmp/mydir" - and don't see this log.There can I find it? 

I upgraded Cloud Manager from 5.10.0 to 6.3.0, then i tried to run 'wget http://10.0.0.209:7180/api/v13/clusters/{clusterName}/services/{serviceName}/clientConfig', got a error 404. Checked the log file of cloudera-scm-server, got the following:2020-09-15 04:11:57,977 INFO scm-web-16747:com.cloudera.api.ApiExceptionMapper: Exception caught in API invocation. Msg:Service 'yarn' not found in cluster 'platform-pod10-eason-cdh'.java.util.NoSuchElementException: Service 'yarn' not found in cluster 'platform-pod10-eason-cdh'.at com.cloudera.api.dao.impl.ManagerDaoBase.findService(ManagerDaoBase.java:484)......but I checked the services via accessing API /clusters/{clusterName}/services/{serviceName}, actually the service 'yarn' is in this cluster, ref:[root@rhel76x64 ~]# curl http://<user>:<password>@10.0.0.209:7180/api/v13/clusters/platform-pod10-eason-cdh/services/yarn/{"healthChecks" : [ {"name" : "YARN_JOBHISTORY_HEALTH","summary" : "GOOD","suppressed" : false}, {"name"

CDP 7.0.3 version does it have any free version like cloudera express? do we have that here? also how many days is the trial version of it?