Find answers, ask questions, and share your expertise
Recently active
Dear all, I have a question on using a new set of data sources on the currently installed cloudera CDP platform in Ooredoo network. Currently the OPEX optimizer uses Telco core and network access along with Charging System Data sources. Now, we have an opportunity to add a new set of IT (non Telco) data sources that may contain offline batch data sources as well as streaming data (ex. For Fraud Management and Revenue Assurance depts), shall we : Install a new Cloudera CDP platform fully dedicated to the new data sources (IT data source).Or use the existing CDP platform ? If the second option is chosen, what are the components and tools that will be used for ingestion, store and process the new data sources, and what are the nodes that will be implied to implement them ? can we dedicate some worker nodes working only for IT data sources for the same HDFS ? how is the configuration of the components and tools then ? Best Regards, mamoune,
Hi, I have to work on nifi data flow to migrate data from one database to another database from multiple schemas. I have to migrate only specific set of tables with a filtering applied to each of the tables. I came up with a sample flow where i used one ExecuteSql Processor for each table that has to be migrated. In the Sql query for the processor i applied the filter criteria and then connected that processor to an individual PutDatabaseRecord processor for each destination table. Just want to understand is that the right approach to have separate processors for each source and destination table? Also would appreciate some guidance on what are the best processors to be considered for data migration from one db to another? This will be a continuous migration to be used in Production environment that will keep running for a couple of months. Appreciate any help. Thanks
Hi,I am using a custom python script (image attached below) in ExecuteProcess processor and it generates flowfiles as result of stdout. The problem I am facing is that for all stdouts, it generates only one flowfile. My requirement is to generate seprate flowfile for each stdout.I tried putting a pattern in stdout and feeding the output flowfile to SplitContent as input. But I keep getting "A flowfile is currently penalized and the data cannot be processed at this time" and it doesn't resolve even after waiting for so long. SplitContent doesn't show any error either, it just doesn't process the splitting.Trying to split the flowfiles based on pattern - "#;#" [PFA the images below]I would really appreciate any help! PS.: Also tried to use ExecuteScript to run custom python script, but I guess since it works on jython, it didn't allow me to import some python packages.
Is there a way to use SSO without enabling HTTPS on NIFI?
Hi, I'm getting this message when running process with more than 10000 flowfiles.unable to write flowfile content to content repository container default due to archive file size constraints I googled and read more about content repository and content claims, but still don't get it.My archive folder is like 100MB. Can you please help me understand how to configure NiFi to get rid of this message? Thanks a lot. My settings which could affect this is:nifi.flow.configuration.archive.max.time=30 daysnifi.flow.configuration.archive.max.storage=500 MBnifi.flow.configuration.archive.max.count= nifi.queue.backpressure.count=10000nifi.queue.backpressure.size=1 GB nifi.content.repository.archive.max.retention.period=12 hoursnifi.content.repository.archive.max.usage.percentage=50%nifi.content.repository.archive.enabled=true
Hello all,I am installing Data Services with Embedded Container Service. There are 2 dedicated servers: 1 ECS server, 1 ECS agent. First installation run not good, always stuck at this task:Waiting for process rke-wait+ kubectl taint node -l ecs_role=master node-role.kubernetes.io/control-plane-taint "node-role.kubernetes.io/control-plane" not foundtaint "node-role.kubernetes.io/control-plane" not found+ true+ sleep 10+ /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get nodes -o 'jsonpath={range .items[*]}{@.metadata.name}:{range @.status.conditions[*]}{@.type}={@.status};{end}{end}'+ grep Ready=True++ hostname -f+ grep instance-20220615-1150.subnet06081614.vcn06081614.oraclevcn.com+ echo ...+ [[ ! -z true ]]+ kubectl taint node -l ecs_role=master node-role.kubernetes.io/control-plane-taint "node-role.kubernetes.io/control-plane" not foundtaint "node-role.kubernetes.io/control-plane" not found+ true Anyone face this familiar issue before? how can i fix
I am trying to import an *.xml template made with NiFi version 1.13.2 into a 1.14 version client (in docker).When trying to upload the *.xml file with the Upload Template button under the Operate panel, I get the following error message: <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 403 Forbidden</title> </head> <body><h2>HTTP ERROR 403 Forbidden</h2> <table> <tr><th>URI:</th><td>/nifi-api/process-groups/8f8e29ed-0181-1000-d7d0-76467ebfce01/templates/upload</td></tr> <tr><th>STATUS:</th><td>403</td></tr> <tr><th>MESSAGE:</th><td>Forbidden</td></tr> <tr><th>SERVLET:</th><td>jerseySpring</td></tr> </table> <hr><a href="https://eclipse.org/jetty">Powered by Jetty:// 9.4.42.v20210604</a><hr/> </body>
Hi, We are having an issue reading tables in Impala with datatype Decimal (1,1). The error is: column 'DECIMAL_1_1' has an invalid type length. Expecting: 1 len in file 4 We have two Cloudera clusters (CDH 6.3.3). This error only occurs in the 2nd cluster. We write data into Parquet files through IBM Datastage. I run the same job in both clusters to examine the parquet files schema using parquet-tools schema , and I found this: In the 1st cluster, in the parquet file the schema of the column causing the issue is:optional int32 DECIMAL_1_1 (DECIMAL(1,1)); However, in the 2nd cluster:optional fixed_len_byte_array(4) DECIMAL_1_1 (DECIMAL(1,1)); We need to understand why the type is fixed_len_byte_array(4) in this parquet while it's int32 in the other. Apparently the fixed_len_byte_array(4) is causing the issue with Impala.
hi We have recently upgrade from CDH to CDP 7.1.7. Now we want to disable Kerberos on CDP cluster. Is it feasible to do so. I know it is not recommended by Cloudera to disable Kerberos. But still we want to do that to implement one use case. Could someone tell me how to disable kerberos for whole cluster in CDP 7.
I am trying to learn how to use the RestLookupService to make REST API calls. But since it is a controller service, not a processor.. so I need some guidance on which processor I should use.. where I can configure this service. I want to save the response that this service returns to somewhere else..let's say to the local file system...using PutFile.
eating bean with name 'auditDataSource': FactoryBean threw exception on object creation; nested exception is org.h2.jdbc.JdbcSQLNonTransientConnectionException: File corrupted while reading record: null. Possible solution: use the recovery tool [90030-199] at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:342) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:113) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1707) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1452) at org.springframework.beans.factory.support.AbstractAutowir
Hello,I'm facing a problem with HDFS is in bad state because of Canary test failed. ERROR com.cloudera.cmon.firehose.polling.hdfs.HdfsCanary: (9 skipped) com.cloudera.cmon.firehose.polling.hdfs.HdfsCanary@70164e31 for hdfs://nameservice1: Failed to write to /tmp/.cloudera_health_monitoring_canary_files/.canary_file_2022_06_29-15_20_26.3f6b5657894eb2c0. Error: {}java.io.IOException: Could not get block locations. Source file "/tmp/.cloudera_health_monitoring_canary_files/.canary_file_2022_06_29-15_20_26.3f6b5657894eb2c0" - Aborting...block==null at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1491) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1271) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667)WARN org.apache.hadoop.hdfs.DataStreamer: Could not get block locations. Source file "/tmp/.cloudera_health_monitoring_canary_files/.canary_file_2022_06_29-15_24_31.b
Hi I am having a challenge with the access control for credentials while trying to execute the spark process in Zeppelin, can anyone help? ERROR idbroker.AbstractIDBClient: Cloud Access Broker response: { "error": "There is no mapped role for the group(s) associated with the authenticated user.", RegardsLakshmi Segu
Hello,I want to graph number of HBase regions per regionserver. In Ambari there was a metric regionserver.Server.regionCount, that provided just that. In CM the closest query I could find is "select regions_across_regionservers" that shows me an average number of regions per server. Average is useful, but I want to track per server data.I know how to get it from command line, or from master web gui. I am looking specifically for datasource query so I can graph it with grafana.Thank you!
Is there any way to e.g. keep the ttl for the service HDFS at 90 DAYS, and set the ttl for the other services to 14 DAYS? ---Related articles:Support Question "How to define the retention period for Ranger Audits in CDP INFRA Solr?"https://community.cloudera.com/t5/Support-Questions/How-to-define-the-retention-period-for-Ranger-Audits-in-CDP/td-p/318759 solrctl Referencehttps://docs.cloudera.com/runtime/7.1.0/search-managing/topics/search-solrctl-ref.html
HelloI'm using Postgresql with NIFI, how can I specify the schema I want to use from Postgresql There is no option for the schema in the DBCPConnectionPoll. and there is no option for schema also in the lookup service
Hello everyone, i need your help. I am relatively new to nifi and need your help. I am using the processor ConsumeKafkaRecord_2_6 1.15.3 with the record writer json set record writer. The output is then an array without object name. Which processor or setup can I use to add an array name to the json output? I get this Output [ { "moving_id" : 17078440, "store_id" : 9, "gtin" : 4064813288877, "qty" : -1.0, "condition_indicator" : 40, "issue_date" : 1656585240000, "received_date" : 1656547200000, "update_date" : 1656578057133}, { "moving_id" : 17078441, "store_id" : 9, "gtin" : 4064813288907, "qty" : -1.0, "condition_indicator" : 40, "issue_date" : 1656585240000, "received_date" : 1656547200000, "update_date" : 1656578057133} ] and i need json file in this structur {"stocks":[ { "moving_id" : 17078440, "store_id" : 9, "gtin" : 4064813288877, "qty" : -1.0, "condition_indicator" : 40, "issue_date" : 1656585240000, "received_date" : 165654720
Hi Team,We are using guava-15.0.jar as part of cloudera-manager-api dependency version 6.3.0 in build.gradle file of cloudera-manager client module.But while deploying it is giving the below error -java.lang.NoClassDefFoundError: com/google/common/base/Objects$ToStringHelperWe faced the above error even with guava-30.0-jre.I tried deploying by excluding the above guava jar but that again gave error -Exception in thread \"main\" java.lang.NoClassDefFoundError: com/google/common/cache/CacheLoader Please find below the build.gradle file dependencies-dependencies {compile group: 'com.cloudera.api', name: 'cloudera-manager-api', version:'6.3.0'compile group: 'org.slf4j', name: 'slf4j-log4j12', version:'1.7.21'compile group: 'com.github.stefanbirkner', name: 'system-rules', version:'1.3.0'compile group: 'org.eclipse.persistence', name: 'eclipselink', version:'2.4.2'compile group: 'org.apache.cxf', name: 'cxf-rt-transports-http', version:'2.7.7'compile group: 'org.json', name: 'json',
Hello Experts, I am trying to upgrade nifi from 1.12.1 version to 1.16.3.I have a script 'ECMAScript' which was working fine in 1.12.1and now behaving different in 1.16.3.Below is the portion of code giving error in 1.16 version when 'xyzId' is null.But it just logs null as expected in 1.12 version var xyzId = flowFile.getAttribute('xyzId');log.info(xyzId); Below is the error in 1.16 version when null value received for above line (log.info() line throws below error)imer-Driven Process Thread-8] o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=bae38131-0181-1000-29af-196312cdd9d9] ExecuteScript[id=bae38131-0181-1000-29af-196312cdd9d9] failed to process due to org.apache.nifi.processor.exception.ProcessException: java.lang.RuntimeException: java.lang.NoSuchMethodException: Can't unambiguously select between fixed arity signatures [(org.apache.nifi.logging.LogMessage), (java.lang.String)] of the method org.apache.nifi.controller.TerminationAwareLogger.info for arg
Hi, I have successfully installed Cloudera Manager on RHEL 7 Azure VM but I'm not able to access the web ui. I have tried after adding 7180 port as inbound port in Network Security Group in azure portal.
Hi Team, I have the below flow to get the ticket details through Rest Api call i want to execute it will atomatically execute after every 60 min but scheduling not working.Please support how can execute below flow after every 60 min. Note:API's working fine through postman able to execute flow I have set below property but flow is not executed after 60 min.
Hi , getting this error while connecting with secured nifi instance.I tried adding certs to Keystore of java. /bin/cli.sh nifi current-user -p nifi.propertiesERROR: Error executing command 'current-user' : PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Hi, I took the CDP generalist exam few weeks ago,and I can't find a place to download its invoice to hand it to my company for a refund.anyone can help? Regards,
Hi Everyone, I'm trying to schedule a impala query using Oozie scheduler but it keeps failing or gets stuck at a percentage for hours. I created a workflow and then scheduled which did not work. Does anyone any expierence scheduling a impala query using Oozie? Could you please provide step by step directions or share a link that would walk me through the process? Thank you in advance.
The following attached error during Kerberos setup with Ambari 2.7.5
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.