Q&A

Hi Team, We are currently using Cloudera 6.1.1 and now we are planning to change Log4j version from 2.8.2to 2.16.0 or above.Can you share the link of 2.16.0 or above log4j version so can i download it and apply patch. Thanks  

Hey together, we're running a CDP Private Cloud Base Cluster with actually 6 Nodes. All 6 Nodes running fully on SSD storage. I found out that HDFS thinks the SSDs are normal disk, because it is not configured as SSD storage like this --> "[SSD]/grid/n/dn", actually its "/grid/n/dn". Now we're planning to add four other servers from an older cluster with normal disks. My action plan is as follows: 1. configure existing storage/server to SSD -> "[SSD]/grid/n/dn"2. add the other four server an configure it -> "[DISK]/grid/n/dn"3. restart recommended services I've got the following questions: - Will HDFS distribute the blocks across all nodes (SSD and disk?) without any storage policy configured? So that we have a homogeneous storage. - How will services react to this changes in storage, for example HBase?- If we configure a homogeneous storage, as i described: does reconfiguring the storage paths of the Datanodes to SSDs add any

JsonTreeReader to CSV Writer produces MapRecord String for the inner Json.  Data"id" : "aaaa","billingaddress" : {"city" : null,"country" : "CHINA","geocodeAccuracy" : null,"latitude" : null,"longitude" : null,"postalCode" : null,"state" : null,"street" : null} Produces CSV out with MapRecord String attached. aaaa|MapRecord[{country=CHINA, city=null, street=null, latitude=null, postalCode=null, geocodeAccuracy=null, state=null, longitude=null}] Expected Output:aaaa|{country=CHINA, city=null, street=null, latitude=null, postalCode=null, geocodeAccuracy=null, state=null, longitude=null} I want to keep the inner JSON as-is in the CSV output. Any workarounds to this issue?

#1 How to find In which server the Kafka cluster and topic are created and  #2 Can we find any IP addresses anywhere in the configuration/properties files

Hi,I am still learning Apache NiFi but I need a quick solution for triggering a processor in a process group after the first process group finished processing.  I have two process groups and two of them extracting, little bit transforming and loading tables from source db to destination db. In both process group, the process starts with GenerateTableFetch processor. What I want to do is after first process group finished its work the second one start processing. I managed to configure process groups so that second one is wating the first one to finish all its process. But how can I manage to prevent the first one to transfer data to second processor? I don't need the data anymore after I put it to destination but I need to trigger second process as well.

Hi,All I want to do is replace multiple character in a field's value.The text in the field is:'I have different characters'If I try ${field.value:replace('a', 'i')} it works. The output is 'I hive different chiricters' Bu I want to replace one more character. How can I do this in same processor.I have tried someting like ${field.value:raplace('a', 'i'); field.value:replace('e', 'u')} I couln't find the right syntax. Please help.

we have hadoop cluster based on HDP from hortonworks version HDP 3.1.0.0-78cluster include 2 namenode services when one is the standby namenode and the second is the active namenode , all machines in the cluster are CentOs 7.9 version , and we dont'  see any problem on OS level also cluster include  87 data node machines ( 9 admin nodes various master services working on them ). All physical machines , around 7 PB data volume, 75% is full. the story begin with neither NN1 nor NN2 work in same time , I mean active , standby .  They were working for more that 2-3 years with out issue. Last 2-3 months they dont get running in the same time.  When I look at he NN logs , after one NN get active second one getting running and in some point 3 Journal Nodes get an exception and NN2 get down. /************************************************************have seen NN failover occur after Journal Node got Exception from 3 JN.see below logs. {Truncated unnecessary

Hi,We upgrade cloudera from 5.11.2 to 7.1.7 and we have a process of creating Env with that new version.It failed on :"Aborting command 1546335073 (CreateTmp) because timeout value 210 seconds exceeded. "How can I increase the default timeout from 210 to 400 for example?Thanks

Hello guys,Can someone please help me, how I can write a solr query where results to be grouped on one field (text type) and sum on another field. Here is my documents results:  {"responseHeader": {"zkConnected": true,"status": 0,"QTime": 30,"params": {"q": "*:*","doAs": "knoxui","fl": "access, agent, repo, resource, resType, event_count","fq": ["access:read","repo:cm_ozone","resType:key","action:read"],"_forwardedCount": "1","_": "1674633786010"}},"response": {"numFound": 8,"start": 0,"docs": [{"access": "read","agent": "ozone","repo": "cm_ozone","resource": "volume1/fso-bucket/dir1/dir2/dir3","resType": "key","event_count": 1},{"access": "read","agent": "ozone","repo": "cm_ozone","resource": "volume1/fso-bucket/dir1","resType": "key","event_count": 1},{"access": "read","agent": "ozone","repo": "cm_ozone","resource": "volume1/fso-bucket/dir1/test3.txt","resType": "key","event_count": 1},{"access": "read","agent": "ozone","repo": "cm_ozone","resource": "volume1/fso-bucket/dir

While writing data to Hbase, I am seeing the following logs at client end:AsyncRequestFutureImpl [ERROR] No ClientProtos.RegionLoadStats found for server=regionserver-1,16020,1662657306210, region=table1,,1662657579710.841db4e690a77a76e74b322f413b3c96.I am successfully able to write data, but I am seeing these logs for every write request. Looks like this is probably because Region Load Stats are disabled for regions in the region servers. Is there any way to enable them and fix this logging?Hbase Client version : 2.1.7Hbase Server version : 2.1.7

I am uncertain on how to pass the file name from the GetFile process to a bash script so that we can call clamAV on it.  I then need the result (success|failure) of the script passed back to nifi so we can route the flow. I can't seem to understand how to pass the arguments correctly or if I'm even using the correct process; ExecuteStreamCommand or ExecuteProcess.  We had an sys architect designing this with us but he left, leaving behind a very complicated non-working example. I thought I had configured an ExecuteProcess correctly but it doesn't allow me to connect GetFile to it.

Hi Team, cloudera embedded postgres db taking more memory. How to check and reduce embedded postgres db taking more memory?? Regards,Hanu

Hello Everyone  Can I Create custom processor in apache nifi to convert CSVfile to excel file ?? is that possible and if yes how ? Thank you  

Can someone explain what is different about these two spark execution engines.below? Environment: CDP private clusterSpark version 2 We have a full ACID hive managed table that we need to access from spark ETL. We used the documentation provided to connect to Hive WareHouse connector ->https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.1/integrating-hive/content/hive_hivewarehouseconnector_for_handling_apache_spark_data.html In addition to using hive warehouse connector to access the acid tables, what execution differences are there between two submissions. We don't see any DAG in spark history server and the query takes far too long (x3) than a similar query from SQLContext using a non-acid managed table. from pyspark_llap import HiveWarehouseSessionhive = HiveWarehouseSession.session(spark).build()df= hive.sql("select * from incidents LIMIT 100")...df.show(10)#additional spark transformation code..# NO DAG in spark history server, slower, takes higher mem

Hello cloudera community, we have the following problem: we are using powerbi with hortonworks odbc driver to connect to hive in cluster ambari 2.6.2.2, hdp 2.6.5 the connection is made successfully, but when making a query on a table that has 23 thousand rows, it returns the following error below: Erro do OLE DB ou do ODBC : [DataSource.Error] ERROR [HY000] [Hortonworks][Hardy] (35) Error from server: error code: '0' error message: 'Invalid OperationHandle: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=0345789f-6c9a-4990-adf5-f823232338]'..if we make a query putting a limiter of at most 10,000 thousand lines in the select, the result is okwhat could be causing this problem? PS: there are queries in powerbi with the same ODBC in other tables that have more than 200 thousand rows and the result is OK too 

 I am following the Apache nifi instructions:  nifi.apache.org/docs/html/administration-guide.html#basic-cluster-setup. And am receiving the following error dump (see below): One specific error that keeps repeating is: Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: The nifi.web.http.port must be specified if running in a cluster with nifi.cluster.protocol.is.secure set to false.at Can anyone provide any guidance/recommendations for resolving the errors I am encountering? Thank you! VR, Dave org.apache.nifi.processors.hadoop.FetchHDFSorg.apache.nifi:nifi-hadoop-nar:1.19.1 || ./work/nar/extensions/nifi-hadoop-nar-1.19.1.nar-unpackedorg.apache.nifi.processors.cassandra.PutCassandraRecordorg.apache.nifi:nifi-cassandra-nar:1.19.1 || ./work/nar/extensions/nifi-cassandra-nar-1.19.1.na

Hello I'm trying to implement an automation process to instanciate a nifi cluster and I have a problem with Context Parameters.Using the nifi REST API with curl commands (or python scripts), I have found how to create a context parameter, how to populate it with parameters, how to bound it to a process group, but I do not find how to unbind it from a processor group (like if I want to reset the "Process Group Parameter Context" of the processor group configuration window scrolling menu to "No parameter context") I meen it is possible (it is possible using the GUI)Does anyone have any idea how to do this.ThanksBest regards,Absolute Z

I have json file like this and want send it to hive using nifi, how can i change the timestamp type because when i send it to hive, in hive table the column type is string, and I read in hive DDL format for timestamp "yyyy-MM-dd HH:mm:ss.SSS" {"nama": "d", "alamat": "jkt", "_ab_cdc_updated_at": "2023-01-05T05:57:43.499Z", "_ab_cdc_lsn": 1640844128, "_ab_cdc_deleted_at": null}{"nama": "a", "alamat": null, "_ab_cdc_updated_at": "2023-01-05T05:57:43.510Z", "_ab_cdc_lsn": 1640844256, "_ab_cdc_deleted_at": "2023-01-05T05:57:43.510Z"}{"nama": "c", "alamat": "semarang", "_ab_cdc_updated_at": "2023-01-12T03:38:32.136Z", "_ab_cdc_lsn": 1684930552, "_ab_cdc_deleted_at": null}{"nama": "f", "alamat": null, "_ab_cdc_updated_at": "2023-01-12T04:11:53.329Z", "_ab_cdc_lsn": 1685073792, "_ab_cdc_deleted_at": "2023-01-12T04:11:53.329Z"}

Nifi - Is there a way where credentials for sftp NIFI can be read from DB and used

I need to pass headers in order make a WebSocket connection.How can I do this in NiFi? 

Hive is present on different cluster and spark(Stand Alone) is installed on different machine. In spark machine hadoop is not installed so, no direct access to hdfs.  In this scenario how can i connect to hive using spark. What are the available ways?

Hi Team,We have used the latest Solr version 8..11.1. In this same version, I tried to create a config set but got exceptions like "Can't create a config set with an unauthenticated request from a trusted baseConfigSet". Used below command for creating config set :http://localhost:8984/solr/admin/configs?action=CREATE&name=_client&baseConfigSet=_default&configSetProp.immutable=false&wt=xml&omitHeader=true Response with exception :Please let me know the steps for creating a config-set in New version 8.11.1.

Hi All, Need some guideline for "how to define/design Dynamic SPARK job" any parameter. Please suggest me, it would be more appreciate in advance. Regards,Pankaj

Hi,I wanted to create log files in nifi based on the processor group, could someone help me on the configuration that needs to be done in logback.xml file? Like currently i have 3 different processor groups and i wanted to create 3 different log files for each of them.