Q&A

Dear All，Yesterday I uploaded a parcel to CM and found some bugs within it. Therefore, I deleted the parcel and completely removed the files in the parcels and .flood directories.Now, when I click the 'Add Service' button, there is still a service type that I defined within the parcel I deleted. How can I remove this service type,  and where does CM record this information?

After using JOLT for many years now I still find myself fumbling my way into solutions, I am however stuck on the problem of how to selectively remove values from a JSON array.The scenario is that  I am trying to solve for is that I want to wholly remove any values in the array whereby there is an equals "=" or really any "special" character.Here is the example input: { "tags": [ "misp-pattern=\\\"Phishing - T1566\\\"", "circl:incident-classification=\\\"phishing\\\"", "IDS" ]} and the output should be; { "tags": [ "IDS" ]} For those interested I am trying to transpose tags from MISP (Threat Intel platform) and under certain conditions it inserts some horrible entries I want to wholly remove them.I understand I can do this in a few steps by extracting and "recasting" this back into JSON however I need to do it via JOLT

Hello All,We have 12 disk configured on each data node of 7.3 TB disk size of each, so that makes up the total space on each data node 87.6 TB. Disk size on each data node:/dev/sdb1 7.3T /data1/dev/sdc1 7.3T /data2/dev/sdd1 7.3T /data3/dev/sde1 7.3T /data4/dev/sdf1 7.3T /data5/dev/sdg1 7.3T /data6/dev/sdh1 7.3T /data7/dev/sdi1 7.3T /data8/dev/sdj1 7.3T /data9/dev/sdk1 7.3T /data10/dev/sdl1 7.3T /data11/dev/sdm1 7.3T /data12Due to growth in data we have already come close to the threshold value on each data node. Hence need suggestion/best practice to know is it better to add more disk of the same size or should we increase data nodes. considering we don't compromise with the load on the servers. So if adding nodes would be a better option then we will go for that.We are currently on CDP 7 and I want to ensure we follow the best practice so that should accommodate our space concerns in the long run.Kindly share if any one has to come to this situation and what would be the best app

I have a source table with 200 million of rows, which needs to fetch to another system.The other system offer a batch insert.  Is there an NiFi example I can follow that will read a source table and split the data into to 20 chucks and make 20 batch inserts, concurrently? 

Hello, im new in Hadoop and just want to know, can i using hdfs as local storage in my Spark driver?For example: im sending throught Livy a task wherekind":"pyspark" and "code" which contains some operations, that in result should be create some new file.When i do it in yarn cluster mode, i find that new file was created in a local storage of node with path like: /tmp/hadoop-username/nm-local-dir/usercache/root/appcache......Can i have any way for set path instead local in hdfs?I want save my spark results(new created file) in hdfsWhen i set spark.local.dir or yarn.nodemanager.local-dirs = hdfs:///temp Livy session just not startingMounting HDFS dfs-fuse not seems like the best way.Or i should use my own fileApp.jar that will be work on each node and each sessions? 

iam trying to publish data into multiple kafka topics with some conditions but instead of returning value of the expression , whole value passed for the property is coming.customerTopic should return me the value of the expression but its returning the expression itselfattaching the images of the flow.

I want to create a flow In NIFI where i want to consumer data from 3 Kafka topics and produce that data into 3 different Kafka topics . each topic should produce data into unique topic . for example - kafka topics 1  -->  produce to topic Akafka topics 2 --> produce to topic Bkafka topics 3 --> produce to topic Ci want to use only one Consumer processor and one producer processor . Right now i am using 3 produce Kafka processor .Can anyone suggest better approach to do so and in more optimized way. 

I have issue when i add service management and error log below2023-11-13 09:59:02,907 INFO scm-web-1336:com.cloudera.server.cmf.cluster.RulesCluster: Beginning automatic role assignment2023-11-13 09:59:02,909 ERROR scm-web-1336:com.cloudera.server.web.common.JsonResponse: JsonResponse created with throwable:java.lang.NullPointerExceptionat com.cloudera.server.cmf.cluster.RulesCluster.createRulesSession(RulesCluster.java:169)at com.cloudera.server.cmf.cluster.RulesCluster.doRoleAssignment(RulesCluster.java:328)at com.cloudera.server.cmf.cluster.RulesCluster.createAndConfigureServices(RulesCluster.java:600)at com.cloudera.server.web.cmf.wizard.service.AddServiceWizardController2.createCustomServiceAndRoles(AddServiceWizardController2.java:893)at com.cloudera.server.web.cmf.wizard.service.AddServiceWizardController2.buildCluster(AddServiceWizardController2.java:834)at com.cloudera.server.web.cmf.wizard.service.AddServiceWizardController2.buildTestDatabaseData(AddServiceWizardControll

How to save NIFI Error log to database table ? 

Hello Team,I am trying to run beeline commandline to export the hive table to a location. i am trying to set replication factor to 1 but it if failing with following error :Error: Error while processing statement: Cannot modify dfs.replication at runtime. It is not in list of params that are allowed to be modified at runtime (state=42000,code=1)Cluster details :3 nodes cluster and in all hvie-site.xml i have set the following property and then restarted hive and hiveserver2 <property><name>hive.security.authorization.sqlstd.confwhitelist.append</name><value>mapred.*|hive.*|mapreduce.*|spark.*|dfs.*</value></property>Beeline Command : sudo -u hive beeline -u "jdbc:hive2://machine1.dev.domain.com:2181/default;password=hive;principal=hive/_HOST@DEV.domain.COM;serviceDiscoveryMode=zooKeeper;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;trustStorePassword=****;user=hive;zooKeeperNamespace=hivese

Json input:{"schema": {"type": "struct","fields": [{"type": "struct","fields": [{"type": "string","optional": false,"field": "COUNTRY_ID"},{"type": "string","optional": true,"field": "COUNTRY_NAME"},{"type": "string","optional": true,"field": "REGION_ID"}],"optional": true,"name": "poc2.NIFITEST.COUNTRIES.Value","field": "before","version": null},{"type": "struct","fields": [{"type": "string","optional": false,"field": "COUNTRY_ID"},{"type": "string","optional": true,"field": "COUNTRY_NAME"},{"type": "string","optional": true,"field": "REGION_ID"}],"optional": true,"name": "poc2.NIFITEST.COUNTRIES.Value","field": "after","version": null},{"type": "struct","fields": [{"type": "string","optional": false,"field": "version","name": null,"version": null,"parameters": null,"default": null},{"type": "string","optional": false,"field": "connector","name": null,"version": null,"parameters": null,"default": null},{"type": "string","optional": false,"field": "name","name": null,"version": null,"p

Dear Experts,I want to transfer data from NiFi to Elasticsearch using the PutElasticsearchHTTP processor. However, I am encountering the error "Elasticsearch returend code 400 with message bad request" The setup, which I have depicted in the image below, was working previously. How should I resolve this problem?Regards 

I am running Hadoop version 3.2.4 in windows and want to perform a WordCount operation on the file located in hadoop/share/hadoop/mapreduce/share/mapreduce-examples-3.2.4.jar. However, it failed, and there is an error message as follows: Application application_1699601027477_0002 failed 2 times due to AM Container for appattempt_1699601027477_0002_000002 exited with exitCode: 1Failing this attempt.Diagnostics: [2023-11-10 14:44:16.479]Exception from container-launch.Container id: container_1699601027477_0002_02_000001Exit code: 1Exception message: '/tmp/hadoop-ryan' is not recognized as an internal or external command,operable program or batch file.[2023-11-10 14:44:16.479]Container exited with a non-zero exit code 1.[2023-11-10 14:44:16.479]Container exited with a non-zero exit code 1.For more detailed output, check the application tracking page: http://LAPTOP-IPDKGW12:8088/cluster/app/application_1699601027477_0002 Then click on links to logs of each attempt.. Failing the a

Hi community,I used multiget to get data but it always returns no data. I tried testing with a rowkey and it worked fineGET 'http://localhost:8000/Report.Stops/multiget?row=G38040-502793%2c1699493158'No rows found.Get by rowkeyGET 'http://localhost:8000/Report.Stops/G38040-502793%2c1699493158'<CellSet><Row key="RzM4MDQwLTUwMjc5MywxNjk5NDkzMTU4">...</Row></CellSet>Shellget 'Report.Stops','G38040-502793,1699493158'COLUMN CELL d:1 timestamp=1970-01-21T00:04:56.975, value=\x98\x94\x00\x00i'm using hbase: 2.4.14 

hi all   we have HDP version - 2.6.4    with active and standby namenode   is it possible to capture the active namenode IP by REST API?     we try by the following API , but API not return any output    curl -sH "X-Requested-By: ambari" -u admin:admin -H "X-Requested-By: ambari" -X GET http://master:8080/api/v1/clusters//services/HDFS    

I have an input fixed width file as belowJohn 32 M New York 100I want to convert it in to a Json like below{"Name": "John","Age": 32,"Gender": "M","City": "New York","Country": 100}

Does Cloudera still offers free HDP sandbox download??? in november 2023?Because, when I tried to download HDP sandbox, it is showing that the Access is Restricted...

I had imported a 100GB+ of parquet data into a manually derived schema table in Hue.  I had 14 of the 46 columns defined and imported the data successfully.  My next step was to define all columns.  When I imported the data, I got an error:java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: Cannot inspect org.apache.hadoop.io.ArrayWritableThe parquet files no longer appear in the hdfs filesystem.  Is there a way to find out what happened to the data? What went wrong? Can it be corrected?

Hi,I am looking to see if there is a way to conduct health check of Cloudera Services (Nifi, hive etc) on nodes which are running on CDP-PC Data Hub Clusters. I came across an old post, wondering if something like this would be doble in CDP-PC which would give me output of the status of services running on the nodes.https://community.cloudera.com/t5/Support-Questions/API-how-the-get-status-summary-for-all-services-in-ambari/td-p/178973Any response  in this regard would be of great assistance.

Hi Team,I would like to fetch values of tag"AltriDatiGestionali" into each line. whenever natura value is 3.5. I'm trying with the below logic but no luck can anyone help in thisINPUT:{"FatturaElettronicaBody": {"DatiBeniServizi": {"DettaglioLinee": [{"Natura": "N3.5","NumeroLinea": 1,"Quantita": "3.00000000","PrezzoUnitario": "209.04000000","AliquotaIVA": "0.00","ScontoMaggiorazione": [{"Tipo": "SC","Percentuale": "42.00","Importo": "263.40"}],"Descrizione": "A6510108918 COPERCHIO TESTATA CIL.","PrezzoTotale": "363.72000000"},{"Natura": "N3.5","NumeroLinea": 2,"Quantita": "7.00000000","PrezzoUnitario": "254.93000000","AliquotaIVA": "0.00","ScontoMaggiorazione": [{"Tipo": "SC","Percentuale": "42.00","Importo": "749.49"}],"Descrizione": "A4634230112 DISCO DEL FRENO","PrezzoTotale": "1035.02000000"},{"AltriDatiGestionali": {"RiferimentoTesto": "22122013260655367-000180 DEL 20/12/2022","RiferimentoData": "22122013260655367-000180 DEL 20/12/2022","TipoDato": "INTENTO"}}]}}}JOLT SPEC:[{"ope

Hi guys! In the QueryDatabaseTable processor configuration, the Maximum-value Columns parameter specifies a column from the database table. The database polling frequency is 5 seconds, but it happens that new data in the table does not have time to appear after 5 seconds and probably due to this the following error appears, but the processor is working and when data appears, it takes it, but this error annoying, can it be avoided somehow?  

Hello,I am just starting to learn NiFi and I really appreciate the power of this great tool.But I am struggling for several evenings to get what seems a simple flow to work correctly. I have reviewed several samples on how to make an API call and use pagination to retrieve difference pages. This seems to work but I get a unique FF every time. Usign MergeCOntent I was hoping I could create a combined file containing JSON array of all FF content.It seems so simple but I am missing something. Could anyone point me in the right direction?Logic:1) Create FF with starting values limit (500) and start (0)2) InvokeHTTP with URL and starting values3) Update filename with filename prefix (object_type) and current date YYYY-MM-DD4) Check for additional pages in Pagination group (pagination seems to work, getting multiple pages with different JSON data)5) Iterate through InvokeHTTP with new start offset for next page6) Using both next_page and Output output streams to ensure all pages are provided

Hello AllI need your HelpsI have a flowfile attribute with this expression attr1="${filename:find('OD_NCE.*.gz'):ifElse("/isod/brt_fin_fd_OD_NCE_pipe_sep","notfind")}"I want to have the value of expression not the stringHow I can execute this.For Information : I read this expression from file and after this I convert the content to attributes

Hi,I have a column where float values were stored as STRING type, e.g. 9989.64When I cast that column to obtain a FLOAT like so "SELECT a, cast(a AS FLOAT) ... " I obtain the following:acast(a AS FLOAT)5724.95 5724.9501953125  Why does Hive alter the decimal part? How can I get the exact number?BTW, casting the STRING to DOUBLE does render the expected behaviourBR

Hello, i installed CDP 7.4.4 (trial), already enable TLS and Kerberos. I want to add  Ozone in CM but got a this error which is very confusing: stdout:Using /var/run/cloudera-scm-agent/process/1546337746-ozone-STORAGE_CONTAINER_MANAGER as conf dirUsing scripts/control.sh as process scriptCONF_DIR=/var/run/cloudera-scm-agent/process/1546337746-ozone-STORAGE_CONTAINER_MANAGERCMF_CONF_DIR=Wed Aug 30 16:09:11 +07 2023: Running OZONE CSD control script...Wed Aug 30 16:09:11 +07 2023: Detected CDH_VERSION of [7]Wed Aug 30 16:09:11 +07 2023: KERBEROS_AUTH_ENABLED: trueWed Aug 30 16:09:11 +07 2023: OZONE_SSL_ENABLED: trueWed Aug 30 16:09:11 +07 2023: OZONE_SERVICE_ID: hydgWed Aug 30 16:09:11 +07 2023: CONF_DIR: /var/run/cloudera-scm-agent/process/1546337746-ozone-STORAGE_CONTAINER_MANAGERWed Aug 30 16:09:11 +07 2023: Found Ozone Manager(s) on demo-2Wed Aug 30 16:09:11 +07 2023: Found Ozone Storage Container Manager(s) on 'demo-1'Could not find or is not a file stderr:+