Find answers, ask questions, and share your expertise
Recently active
I am using a ForkEnrichment - JoinEnrichment combo to enrich some data. Within the JoinEnrichment processor, the "wrapper" and "sql" options for the "Join Strategy" both work fine. However, the "insert enrichment fields" option doesn't. I prefer this option because I want my enrichment data to be a nested dataset off one of the root fields. The field exists in the original data, and is properly referenced in the "Insertion Record Path" field. So... I'm scratching my head trying to figure out why this isn't working. Anyone experience this or have any ideas?
Hi guys,I'm testing an upgrade from CDH 6.3.4 to CDP 7.1.8 , I successfully performed the upgrade but I'm trying to test the rollback process, I'm using AUTO-TLS, but I'm having a one issue though, it says the below in the HDFS rollback Cloudera : (Clusters with TLS enabled only) Edit the /etc/hadoop/conf.rollback.namenode/ssl-server.xml file on all NameNode hosts (located in the temporary rollback directory) and update the keystore passwords with the actual cleartext passwords. So the question is : How can I get the cleartext password ?
I cannot figure out which local path is out of spaceI have a 3 node cluster. Each node has 1TB boot disk plus 1 TB X 2 attached 7200rpm disks. The directory structure on each of the two disks on each of the three nodes are /media/sanjay/hdd03/yarn/nm/media/sanjay/hdd03/dfs/dn/media/sanjay/hdd03/dfs/snn/media/sanjay/hdd03/dfs/snn/media/sanjay/hdd02/yarn/nm/media/sanjay/hdd02/dfs/dn/media/sanjay/hdd02/dfs/snndf -h /media/sanjay/hdd03Filesystem Size Used Avail Use% Mounted on/dev/sdc1 917G 228M 870G 1% /media/sanjay/hdd03df -h /media/sanjay/hdd02Filesystem Size Used Avail Use% Mounted on/dev/sdb1 917G 1.2G 869G 1% /media/sanjay/hdd02df -h /tmpFilesystem Size Used Avail Use% Mounted on/dev/mapper/ubuntu--vg-root 915G 41G 828G 5% / 2023-01-07 22:15:31,486 ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Local path for public localization is not found. May be disks failed.org.apache.hadoop.util.DiskChecker$DiskErrorException: No sp
I have a JSON dataset and need to create a unique key from it. No single field is unique, but the first two, together, are. What I want to convert from: [{"airport":"KDEN", "runway":"16R", "length":16000}, ...] and I want to concatenate the "airport" and "runway" values to create this: [{"key":"KDEN/16R", "airport":"KDEN", "runway":"16R", "length":16000}, ...] I've got it working with an ugly anti-pattern of SplitRecord, EvaluateJSONPath, UpdateAttribute, and AttributesToJSON which changes each flow file to several thousand, but am looking for a better way to do this.
I need help understanding the root cause of a job failure in Cloudera Data Science Workbench.What does it mean when a job is stuck in "Scheduling" phase and terminates as "Failed" after the timeout period? The log file, which usually displays the python code followed by the results from print statements and/or run time errors, states only a single line reading "Engine exited with status 34". Apparently, I was able to re-run the job successfully after a while. It could be a temporary glitch in the system but still trying to understand the root cause of this issue. Any insight on what happens during the "Job Scheduling" is much appreciated.
Hello everyone! Recently login ambari report 500 error box. I've been working on it for ages, but I can't solve it. The following are the error phenomena and related logs, hoping to get your help. Thanks!Error screenshot:500 status code received on GET method for API: /api/v1/clusters/bgybigdata/alerts?format=groupedSummary log:HTTP ERROR 500Problem accessing /api/v1/clusters/bgybigdata/alerts. Reason: Server Error Caused by:java.lang.NullPointerException at org.apache.ambari.server.controller.internal.AlertResourceProvider.toResource(AlertResourceProvider.java:260) at org.apache.ambari.server.controller.internal.AlertResourceProvider.getResources(AlertResourceProvider.java:240) at org.apache.ambari.server.controller.internal.AlertResourceProvider.queryForResources(AlertResourceProvider.java:166) at org.apache.ambari.server.controller.internal.ClusterControllerImpl$ExtendedResourceProviderWrapper.queryForResources(ClusterControllerImpl.java:966) at org.apache.ambari.s
Does anyone have experience with CDP Private Cloud on CentOS 9?Are there any plans to officially support it in the near future?
Hello,I'm brand new to Atlas and trying to see if this capability exists. I understand that you can use a Hive Hook and whenever a Hive table is modified, Kafka is able to communicate to Atlas and have those changes appear in Atlas. I'm trying to see if there is any reporting or auditing features that allows you to see what those changes were?For example: If a user modified a Hive table and added two columns and this gets replicated in Atlas, is there a way to see what change happened (columns added) and when it occurred? I'm not sure if this is something Atlas can do natively or if there might be 3rd party tools I could use to see this?Thanks in advance!
Hi Team, We are currently using Cloudera 6.1.1 and now we are planning to change Log4j version from 2.8.2to 2.16.0 or above.Can you share the link of 2.16.0 or above log4j version so can i download it and apply patch. Thanks
Hey together, we're running a CDP Private Cloud Base Cluster with actually 6 Nodes. All 6 Nodes running fully on SSD storage. I found out that HDFS thinks the SSDs are normal disk, because it is not configured as SSD storage like this --> "[SSD]/grid/n/dn", actually its "/grid/n/dn". Now we're planning to add four other servers from an older cluster with normal disks. My action plan is as follows: 1. configure existing storage/server to SSD -> "[SSD]/grid/n/dn"2. add the other four server an configure it -> "[DISK]/grid/n/dn"3. restart recommended services I've got the following questions: - Will HDFS distribute the blocks across all nodes (SSD and disk?) without any storage policy configured? So that we have a homogeneous storage. - How will services react to this changes in storage, for example HBase?- If we configure a homogeneous storage, as i described: does reconfiguring the storage paths of the Datanodes to SSDs add any
JsonTreeReader to CSV Writer produces MapRecord String for the inner Json. Data"id" : "aaaa","billingaddress" : {"city" : null,"country" : "CHINA","geocodeAccuracy" : null,"latitude" : null,"longitude" : null,"postalCode" : null,"state" : null,"street" : null} Produces CSV out with MapRecord String attached. aaaa|MapRecord[{country=CHINA, city=null, street=null, latitude=null, postalCode=null, geocodeAccuracy=null, state=null, longitude=null}] Expected Output:aaaa|{country=CHINA, city=null, street=null, latitude=null, postalCode=null, geocodeAccuracy=null, state=null, longitude=null} I want to keep the inner JSON as-is in the CSV output. Any workarounds to this issue?
#1 How to find In which server the Kafka cluster and topic are created and #2 Can we find any IP addresses anywhere in the configuration/properties files
Hi,I am still learning Apache NiFi but I need a quick solution for triggering a processor in a process group after the first process group finished processing. I have two process groups and two of them extracting, little bit transforming and loading tables from source db to destination db. In both process group, the process starts with GenerateTableFetch processor. What I want to do is after first process group finished its work the second one start processing. I managed to configure process groups so that second one is wating the first one to finish all its process. But how can I manage to prevent the first one to transfer data to second processor? I don't need the data anymore after I put it to destination but I need to trigger second process as well.
Hi,All I want to do is replace multiple character in a field's value.The text in the field is:'I have different characters'If I try ${field.value:replace('a', 'i')} it works. The output is 'I hive different chiricters' Bu I want to replace one more character. How can I do this in same processor.I have tried someting like ${field.value:raplace('a', 'i'); field.value:replace('e', 'u')} I couln't find the right syntax. Please help.
we have hadoop cluster based on HDP from hortonworks version HDP 3.1.0.0-78cluster include 2 namenode services when one is the standby namenode and the second is the active namenode , all machines in the cluster are CentOs 7.9 version , and we dont' see any problem on OS level also cluster include 87 data node machines ( 9 admin nodes various master services working on them ). All physical machines , around 7 PB data volume, 75% is full. the story begin with neither NN1 nor NN2 work in same time , I mean active , standby . They were working for more that 2-3 years with out issue. Last 2-3 months they dont get running in the same time. When I look at he NN logs , after one NN get active second one getting running and in some point 3 Journal Nodes get an exception and NN2 get down. /************************************************************have seen NN failover occur after Journal Node got Exception from 3 JN.see below logs. {Truncated unnecessary
Hi,We upgrade cloudera from 5.11.2 to 7.1.7 and we have a process of creating Env with that new version.It failed on :"Aborting command 1546335073 (CreateTmp) because timeout value 210 seconds exceeded. "How can I increase the default timeout from 210 to 400 for example?Thanks
Hello guys,Can someone please help me, how I can write a solr query where results to be grouped on one field (text type) and sum on another field. Here is my documents results: {"responseHeader": {"zkConnected": true,"status": 0,"QTime": 30,"params": {"q": "*:*","doAs": "knoxui","fl": "access, agent, repo, resource, resType, event_count","fq": ["access:read","repo:cm_ozone","resType:key","action:read"],"_forwardedCount": "1","_": "1674633786010"}},"response": {"numFound": 8,"start": 0,"docs": [{"access": "read","agent": "ozone","repo": "cm_ozone","resource": "volume1/fso-bucket/dir1/dir2/dir3","resType": "key","event_count": 1},{"access": "read","agent": "ozone","repo": "cm_ozone","resource": "volume1/fso-bucket/dir1","resType": "key","event_count": 1},{"access": "read","agent": "ozone","repo": "cm_ozone","resource": "volume1/fso-bucket/dir1/test3.txt","resType": "key","event_count": 1},{"access": "read","agent": "ozone","repo": "cm_ozone","resource": "volume1/fso-bucket/dir
While writing data to Hbase, I am seeing the following logs at client end:AsyncRequestFutureImpl [ERROR] No ClientProtos.RegionLoadStats found for server=regionserver-1,16020,1662657306210, region=table1,,1662657579710.841db4e690a77a76e74b322f413b3c96.I am successfully able to write data, but I am seeing these logs for every write request. Looks like this is probably because Region Load Stats are disabled for regions in the region servers. Is there any way to enable them and fix this logging?Hbase Client version : 2.1.7Hbase Server version : 2.1.7
I am uncertain on how to pass the file name from the GetFile process to a bash script so that we can call clamAV on it. I then need the result (success|failure) of the script passed back to nifi so we can route the flow. I can't seem to understand how to pass the arguments correctly or if I'm even using the correct process; ExecuteStreamCommand or ExecuteProcess. We had an sys architect designing this with us but he left, leaving behind a very complicated non-working example. I thought I had configured an ExecuteProcess correctly but it doesn't allow me to connect GetFile to it.
Hi Team, cloudera embedded postgres db taking more memory. How to check and reduce embedded postgres db taking more memory?? Regards,Hanu
Hello Everyone Can I Create custom processor in apache nifi to convert CSVfile to excel file ?? is that possible and if yes how ? Thank you
Can someone explain what is different about these two spark execution engines.below? Environment: CDP private clusterSpark version 2 We have a full ACID hive managed table that we need to access from spark ETL. We used the documentation provided to connect to Hive WareHouse connector ->https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.1/integrating-hive/content/hive_hivewarehouseconnector_for_handling_apache_spark_data.html In addition to using hive warehouse connector to access the acid tables, what execution differences are there between two submissions. We don't see any DAG in spark history server and the query takes far too long (x3) than a similar query from SQLContext using a non-acid managed table. from pyspark_llap import HiveWarehouseSessionhive = HiveWarehouseSession.session(spark).build()df= hive.sql("select * from incidents LIMIT 100")...df.show(10)#additional spark transformation code..# NO DAG in spark history server, slower, takes higher mem
Hello cloudera community, we have the following problem: we are using powerbi with hortonworks odbc driver to connect to hive in cluster ambari 2.6.2.2, hdp 2.6.5 the connection is made successfully, but when making a query on a table that has 23 thousand rows, it returns the following error below: Erro do OLE DB ou do ODBC : [DataSource.Error] ERROR [HY000] [Hortonworks][Hardy] (35) Error from server: error code: '0' error message: 'Invalid OperationHandle: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=0345789f-6c9a-4990-adf5-f823232338]'..if we make a query putting a limiter of at most 10,000 thousand lines in the select, the result is okwhat could be causing this problem? PS: there are queries in powerbi with the same ODBC in other tables that have more than 200 thousand rows and the result is OK too
I am following the Apache nifi instructions: nifi.apache.org/docs/html/administration-guide.html#basic-cluster-setup. And am receiving the following error dump (see below): One specific error that keeps repeating is: Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: The nifi.web.http.port must be specified if running in a cluster with nifi.cluster.protocol.is.secure set to false.at Can anyone provide any guidance/recommendations for resolving the errors I am encountering? Thank you! VR, Dave org.apache.nifi.processors.hadoop.FetchHDFSorg.apache.nifi:nifi-hadoop-nar:1.19.1 || ./work/nar/extensions/nifi-hadoop-nar-1.19.1.nar-unpackedorg.apache.nifi.processors.cassandra.PutCassandraRecordorg.apache.nifi:nifi-cassandra-nar:1.19.1 || ./work/nar/extensions/nifi-cassandra-nar-1.19.1.na
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.