Find answers, ask questions, and share your expertise
Recently active
Hello All, I am novice to Kafka. I am receiving SNMP traps in a particular server 'X', but my Kafka is installed in a different server 'Y'. How can I publish the traps using the kafka-console-producer.sh script to load to topic ?. Any help is appreciated
Hello,is it possible to alter compression.type on a kafka topic while having applications on the topic running? What should be expected hard drive saving factor for text records like json (2KB size) and compression.type=gzip. In CM, I see a global setting. Does it apply to KafkaMirrorMaker only? Are producer and consumer applications somehow affected, when the global parameter changes.Best regardsJaro
Hi All, I have come across a issue where my disk is filling up because of topic __consumer_offsets as i see the size as below.24G __consumer_offsets-4429G __consumer_offsets-3342G __consumer_offsets-13How do i reduce the size of these partitions, from the logs below is the exception what i am seeing.2022-04-05 03:03:30,041 ERROR kafka.log.LogCleaner: [kafka-log-cleaner-thread-0]: Error due tojava.lang.IllegalArgumentException: inconsistent rangeCDH version is 5.16 and kafka is KAFKA-3.1.1-1.3.1.1.p0.2Assuming that log cleaner can be crashed i tried referring below link but that did not help.https://my.cloudera.com/knowledge/Disk-utilisation-growing-because-Kafka-LogCleaner-stopped?id=86234 Need your help and advise to fix the issue.Thanking in advance
Hi, I am getting error when I am querying my table in impala editor.The simple query:select * from table GeoIP2_ISP_Blocks_IPv4;The error:The table creation query is give below:CREATE External TABLE GeoIP2_ISP_Blocks_IPv4 (Client string,ID int,Number int,Name string,)ROW FORMAT DELIMITED FIELDS TERMINATED BY ','WITH SERDEPROPERTIES ("separatorChar" = "\,","quoteChar" = "\"")LOCATION 'hdfs:///user/hive/warehouse/tips/'tblproperties ("skip.header.line.count"="1"); The same error is coming when I am creating external tables from tab delimited files. All this is happening after latest CDP Update. Please someone help me it would be really helpful.
Hi,is there a way to find and remove duplicate entries in two flowfiles?I have one flowfile generated with SQL Processor with entries from my database. The other one contains new and "old" entries. So in order to only write the new entries in the database, I have to detect and remove the entries that already exist in the other flowfile.I already tried the HashContentProcessor but it hashes the content of the whole file. I would need a processor that hashes line for line (and then compares all hashes with each other).Thanks for your help!
We are trying to determine the correct latest version of Cloudera Runtime to use, CM is offering 2 possibilities for CDH 7.1.7-1: 7.1.7-1.cdh7.1.7.p74.210577657.1.7-1.cdh7.1.7.p1000.24102687 What is the best way to determine the latest version? Thanks in advance.
When using the XMLReader service, how do you retain the outermost parent element of an XML document so when I convert it to JSON I have the outermost element in the JSON version too? For example, given the following XML document how do I capture the "<a>" element rather than just the "<b>" element? <a> <b>45</b></a> What I want:{"a": {"b":45}} What I get:{"b":45}
Hi, I'm new to jolt and am trying to remove null values from a nested json using jolt. Example JSON:{"field1": {"subfield1": null},"field2": {"subfield21": "528010000072312","subfield22": null},"field3": {"field31": null,"field32": null,"field33": {"field331": null,"field332": null}},"field4": {"field41": null,"field42": "UNKNOWN","field43": "NONE","field44": {"field441": {"field4411": null,"field4412": null}}},"field5": {"field51": null,"field52": {"field521": null,"field522": null},"field53": {"field531": null,"field532": {"field5321": null},"field533": {"field5331": "0"}}} JOLT SPEC: [{"operation": "shift","spec": {"field1": {"*": {"*": {"@1": "&3.&2"}}},"field2": {"*": {"*": {"@1": "&3.&2"}}},"field3": {"*": {"*": {"@1": "&3.&2"},"field33": {"*": {"@1": "&3.&2"}}}},"field4": {"*": {"*": {"@1": "&3.&2"},"field44": {"field441": {"*": {"@1": "&3.&2"}}}}},"field5": {"*": {"*": {"@1": "&3.&2"},"field53":
Hello, I could use some help... I am using nifi in windows and I downloaded Nifi Registry in windows too. Although the official Nifi Registry web sais that Nifi Registry is not supported for windows I have created a Repository in github and it looks like the data flows were updated correctly into my repository. Nevertheless, If I do a git clone of the repository and try to use the new versios nifi is not able to import the flows. I wanted to ask why this error might be happening, and how to solve it. I have thought in two options: 1) Setting up a secure nifi and a secure nifi registry (I have not done that and probably is the cause of the error)2) Setting up nifi registry in docker (or a VM) and try to run nifi registry in linux. (If I have to use this option would you also recommend setting up nifi in docker (or the VM)?) Many thanks in advance for the help. PS: When I shut down Nifi and run it again the connection between nifi and nifi registry fails: "Failed to s
client :C++ client (librdkafka)Used below config securely connect to cloudera kafka broker . SASL_SSL/GSSAPI - truststore/keystore JKS files ssl.truststore.location=< JKS File >ssl.truststore.password ='xxxxxx'C++ client not accepting config as it is specific to java .Is there any alternate mechanism available securely connect to cloudera broker and config supported by c++ clients?
Hi, I want to connect connect two different MySQL database services(mysql 5.7 and Mysql 8.0 ). I am using two two CaptureChangeMySQL Processors for this. Mysql 5.7 is install in my local machine and CaptureChangeMySQL Processors connecting successfully. Mysql 8.0 is install on docker container and on host machine port 3310 like (sudo docker run --name mysql-8.0-container -p 127.0.0.1:3310:3306 -e MYSQL_ROOT_PASSWORD=password -d mysql:8.0). CaptureChangeMySQL connecting with mysql 8.0 is giving error. Can any suggest why i am getting this error and how to resolve it.Note: I am able to connect mysql 8.0 using mysql workbench. Error:2022-08-03 15:59:58,863 ERROR [Timer-Driven Process Thread-4] o.a.n.c.m.processors.CaptureChangeMySQL CaptureChangeMySQL[id=352daba9-0182-1000-a02a-faf160ab1329] Processing failedorg.apache.nifi.processor.exception.ProcessException: Could not connect binlog client to any of the spec
As per the accepted solution posted in this question, export_snapshot is not supported ( or may not work ) between major versions of Hbase. However, this CDP documentation mentions the supportability of export_snapshot from CDH 5 (Hbase 1.2.0) to CDP 7 (HBase 2.2.3). We are working on migrating Hbase tables from Hbase 1.2 to Hbase 2 and would like to know if "export_snapshot" work in this scenario.
Hi guys,I have a Data lake (Hive Managed tables base) and I would like to do an Incremental approach to the Warehouse (Hive Managed tables base) using spark v3.2, and I faced an issue connection to Hive Managed tables with spark3. And I would like to know,How to connect to Hive ACID Tables? Using JDBC if yes how? Or there are other ways? Thank you!
Hey guys, i've been a quite reader in this forums last months and you all provide me a lot of hints and solutions for multiple questions. Now i've search the documentation and this community, but i'm the first one with this problem i think. We store all our remote-server ips in a database, the login is the same for all. We want to fetch files (names changes possible) from remote servers and download them.The only (built in) sftp processor with input connection is "GetSFTP", but i have to know the filename to perform a download. i cannot guarantee that the filenames will never change, so this is no way to solve my problem ... I know, i can create an processor on my own with some development, but my first step is to get it work with built-in ones.Does anybody got a good idea how to solve it or have i to create a processor on my own? Greetings from Dortmund
Hi all, We have a NiFi 1.15 cluster with five nodes, after running for a while there is a node with below error, I couldn't find the root reason, please help me, thanks. 2021-12-08 19:58:35,803 ERROR [Load-Balance Server Thread-4385] o.a.n.c.q.c.s.ConnectionLoadBalanceServer Failed to communicate with Peer p-edi-nifi-003.gz.cvte.cn/10.21.46.32:55354org.apache.nifi.controller.queue.clustered.server.TransactionAbortedException: Attempted to receive FlowFiles from Peer p-edi-nifi-003 for Connection with ID bf3245df-e6d7-15a9-0000-00005fd59914 but no Connection exists with that ID at org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:205) at org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:134) at org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServer$CommunicateAction.run(ConnectionLoadB
Hi All, I have two cases :Case A : i have a kerberised production cluster i want to copy files from this linux box to gcp storage ( non kerberised).Currently performing manully : downloading linux files to local system using winscp and uploading to google cloud storageif can be done using Distcp provinding steps will be helpful Case B : from the same kerberised cluster want to copy data of HDFS to Google cloud storage( non kerberised) Thanks,Syed.
Hi, I have a background in traditional data warehousing (SQL, 3NF Inmon, Star schema Kimball). I used to design data models then create it in a SQL RDMS and then populate it with ETL data flows. Paying a particular attention to slowly changing dimensions. With Cloudera, how to build the equivalent of a data warehouse? and which feeding treatments? What are the components to use? Thanks.
How to check if one node is connected to another node?
When I tried to run simple commands os Zeppelin like this:%livy.pyspark sc.version I'm getting this error:Caused by: org.apache.zeppelin.livy.LivyException: org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://SERVER:8998/version": PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target; nested exception is javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target We changed the Livy Server instance to other hosts. How can I solve this?
Hi,Query 1: How to connect to two different source DB instants using captureChangeMySql processor?.Query 2: is it possible to configure two different source DB instants using captureChangeMySql processor to get data from both db instance and put it in destination DB?, If yes, Please suggest the configuration approach.Below is my current configuration screenshot to read the data from single instant connecting two source DB.
I have a table with an ID, key and 2 columns with each contains pipe separated values. How can i parse column into multiple rows with the respective values? Table that I have: IDkeyitemnovalue1A1|2|310|20|301B1|2|340|50|601C1|2|370|80|90 Expected output: IDkeyitemnovalue1A1101A2201A3301B1401B2501B3601C1701C2801C390 Really appreciate the help! Thanks
Hello Experts, We have HandleHttpRequests processor running with 16 concurrency.Container queue size configured is 1000.While running a load test suddenly HandleHttpRequests stopped emitting flowfiles recieved and started throwing error show in below image - So we stopped the load test and now even after 30 mins when we send requests it will just emit 2 or 3 flow files and again shows same error and does not release/emit any request flowfiles.Ideally when no request is getting hit to that port (which HttpRequestListener is configured) container should get cleared and flow files should get created from queue and emit right?What could be the reason this processor is just trhowing container full error and not emitting any flow files ?It started working only when we manually cleanup all repositories (content, provenance, ff) and restart the nifi !Any suggestion would be appreciated. Thanks,Mahendra
Hello,We would like to create a Hive table in the ussign pyspark dataframe cluster.We have the script below, which has run well several times in the past on the same cluster. After some configuration changes in the cluster, the same script is showing the error below.We were unable to identify what changes are made to the cluster to trigger this error in this script (we rearange some services at cluster, etc) The simple script is:# pyspark --master=yarndata = [("Java", "20000"), ("Python", "100000"), ("Scala", "3000")]rdd = spark.sparkContext.parallelize(data)dfFromRDD1 = rdd.toDF(columns)dfFromRDD1.printSchema()dfFromRDD1.show()from pyspark.sql import SQLContextfrom pyspark.sql import HiveContextsqlContext = HiveContext(sc)dfFromRDD1.registerTempTable("evento_temp")sqlContext.sql("use default").show() ERROR:Hive Session ID = bd9c459e-1ec8-483e-9543-c1527b33feec22/07/30 13:55:45 WARN metastore.PersistenceManagerProvider: datanucleus.autoStartMechanismMode is set to unsupported
Does anyone know the process for setting up URL aliases in NiFi registry when using Cloudera Manager to manage the instance? The property: nifi.registry.registry.alias.configuration.file is not available in the UI. And while there are options to ignore properties, there is nothing I can find to add arbitrary properties to the configuration. In addition to the above, you also need to add a registry-aliases.xml file to the server and I see no way to accomplish that. Ideally this would all be auto-configured as the registry connection URL information is required in this latter file. And the path to these configuration files is managed by Cloudera Manager. For reference, this is the feature of NiFi Registry I am attempting to make use of:https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#url-aliasing Thanks for any advice anyone can give.
Hello!I've read this and this and this but explanations are pretty scarce. I understand what "&" means. But I can't wrap my head around what [&] means =\Can someone please explain me what this [&] does applicable to this example? What exactly happening in lines? : "@1": "outer[&4].inner[&2].t", "@(3,eventTypeCode)": "outer[&4].inner[&2].etc" JSON:{ "uID": 1000358546, "events": [{ "eventTypeCode": "FEEDBACK", "transports": ["PUSH", "SMS"] }, { "eventTypeCode": "MARKETING", "transports": ["PUSH", "EMAIL"] }, { "eventTypeCode": "ORDER_STATUS", "transports": ["SOC_VK"] } ]} SPEC:[{ "operation": "shift", "spec": { "events": { "*": { "transports": { "*": { "*": { "@1": "outer[&4].inner[&2].t", "@(3,eventTypeCode)": "outer[&4].inne
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.