Find answers, ask questions, and share your expertise
Recently active
Hi team, I'm facing the below-mentioned issue. executorLostFailure (executor 6 exited unrel+details ExecutorLostFailure (executor 6 exited unrelated to the running tasks) Reason: Container marked as failed: container_1677979357691_0001 on host: xyz Exit status: -100. Diagnostics: Container released on a *lost* node.Previously I'm using 18 r5.4xlarge machines, so now I want to move it to 5 r5.16xlarge machines. So I made a few config changesI'm using m5.2xlarge as a master in both the casesI didn't change the EBS volume size. Previously and now also it is 950G spark.executor.cores: "5"spark.driver.cores: "5"spark.executor.memory: "35G"spark.driver.memory: "30G"spark.executor.instances: "60"spark.dynamicAllocation.enabled: "false"spark.executor.memoryOverhead: "6G"spark.excludeOnFailure.enabled: "true"spark.excludeOnFailure.enabled: "true"spark.excludeOnFailure.killExcludedExecutors: "true"spark.excludeOnFailure.application.fetchFailure.enabled: "true"spark.excludeOnFailure.app
hi after running the create_parcel.sh file and copying the .parcel and .parcel.sha file to cloudera i get the following error error Error when distributing to serverxxx.local [Errno 20] Not a directory u'opt/cloudera/parcels/GCSCONNECTOR-3.0.0xxx.sha/meta/parcel.json' and error Error when distributing to serverxxx.local [Errno 20] Not a directory u'opt/cloudera/parcels/GCSCONNECTOR-3.0.0xxx.parcel/meta/parcel.json'
We have around 6k tables in HDFS. All these tables are stored as sub-dirctories under a parent directory. As of now we're passing the root directory and recursive subdirectories as true in LISTHdfs processor. And later using routeonattribute processor to filter out the relevant tables using the path name. But the problem here is LISTHDFS is taking larger time just to list out as it needs to go through 6K directories. Is there is anyway possible to pass only required directories in listhdfs processor or any workaround available for this?
Hello, Have recently deployed a CDP 7.1.7 SP1 cluster and unable to figure out issues with Solr service. When I open Solr Admin UI, it does opens the UI but with red strips and no information related to system, logs, audits, collections, etc. Ideally we should be able to see some system info on the home page and also various logs when in Logging section, and so on. I checked in logs but do not see any error. Kindly help / guide on how this can be investigated.Thankssnm1523
I want to access the reporting task id of a particular reporting task. For doing the same , I thought about fetching all reporting task id and then searching for a particular one. Is there any nifi-api that can give me all the reporting task ids or any other approach to solve this problem.
How can I use Apache NiFi to transform the current output of my Oracle database to the desired output format, as shown below? Desier out should be as mention below Avro formatCurrent output:{ "type":"record", "name":"nifiRecord", "namespace":"org.apache.nifi", "fields":[ {"name":"AGENT_CODE", "type":["null","string"]}, {"name":"CORR_ACC_NO", "type":["null","string"]}, {"name":"LOCAL_ACC_NO", "type":["null","string"]} ] }Desired output:{ "type": "record", "name": "Bank", "namespace": "com.xyz.model.vo", "fields":[ {"name": "instance", "type": "string"}, {"name": "agent_code", "type": "string"}, {"name": "corr_acc_no", "type": "long"}, {"name": "local_acc_no", "type": "string"} ] }What steps do I need to take in NiFi to achieve this transformation and output the results to the desired format? especially for changing { "type": "record", "name": "Bank", "namespace": "com.xyz.model.vo"
Hi All, I'm new to HBase. I have learnt it by some books and Udemy courses. Can someone help me to understand, how to automate it? For example, if one wants to create tables or wanted to perform some administrative tasks, has to create script and run it periodically. in this case, we can wrte script and schedule it in scheduling tool. Which scripting is being used here? Shell Script or HBase has internal scripting mechanism? Any books, tutorial suggestion are welcome.
We have a large json file which is more than 100GB and we want to split this json file into multiple files. We used Split Text processor to split this json file into mutliple files by specifying Line Split Count. Is there any way we can pass attribute/variable in Line Split Count and then split the records based on the attribute/variable as currently Line Split Count does not support attributes/variables.Kindly suggest if there is another approach to split these json files based on attribute/variables Sample Json File {"name": "John","lastName": "Wick","phoneNumber": "123123123"} {"name": "Paul","lastName": "Jackson","phoneNumber": "123123123"}{"name": "Paul","lastName": "Jackson","phoneNumber": "123123123"}
On Cloudera Public Cloud the storage unit is GCS. Therefore, hive tables and any inserted data are stored on GCS rather than local disks ( like as on-prem ). But, this adds an overhead because when running a job, the node manager needs to get the data from gcs and cache it locally till the job is done. This eventually hits the performance on the env. Is there any way to move the data location from GCS to Local disks on Public Cloud clusters? As written on the docs, datahub hdfs/disks spaces are temporary places but I would take this risk in favor of performance. @steven-matison would really appreciate your help on this question. Thanks in advance.
Hello team, We are facing issue and expecting your help here regarding Problem Statement :We create groups using in Cloudera user management using APIs.We can see those groups in Cloudera user management but not could not find them as Ranger Groups. It seems groups are not being synced in Ranger.What are the possible solutions ? Any API is exist to sync users and groups?
Hi team, I have the error Faled running performance when i run the "Inspect network Performance" all the nodes of clouster are good except one. I see this error: [24/Feb/2023 09:19:19 -0600] 1547408 MainThread redactor INFO Started launcher: /opt/cloudera/cm-agent/service/perf/host_perf_diag.py input.json logs/result.json[24/Feb/2023 09:19:19 -0600] 1547408 MainThread redactor ERROR Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/1502-host-perf-inspector[24/Feb/2023 09:19:19 -0600] 1547408 MainThread redactor INFO Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 1547410Traceback (most recent call last):File "/opt/cloudera/cm-agent/bin/cm", line 8, in <module>sys.exit(main())File "/opt/cloudera/cm-agent/lib/python3.8/site-packages/cmf/main.py", line 399, in mainroot(obj=argparse.Namespace())File "/opt/cloudera/cm-agent/lib/python3.8/site-packages/click/core.py", line 1137, in __call
I have a pretty simple setup. We have two NiFi servers on different networks, and they need to transfer data between them. I have site to site set up and working using HTTPS, but for some reason it seems like every single file transfer takes a minimum of 15 seconds. A 1 byte file will take barely over 15000ms, at a transfer rate of a couple hundred bytes per second, or a much larger file will transfer in just under 30 seconds, at a transfer rate of about 85MB/s. I'm guessing the actual transfer of the file is happening quickly, but the transfer speed takes into account the 15 second handshake....or whetever the process is that is taking 15 seconds each time. I looked around and couldn't find anyone else experiencing quite the same thing. I have tried using RAW instead of HTTPS, but I haven't found a working setup for that yet--every time I try I end up with the "Magic Header not found" message. Here are some snippets of my config files, let me know
Greetings Cloudera Community!!!We have recently upgraded from Nifi 1.9.2 version to Nifi version 1.16.2 and we are getting the "Failed to index Provenance Events" Error, not all the time but it occurs occasionally. We have not faced such kind of error in nifi version 1.9.2. Can anyone please help us to figure out the possible cause of the occurrence of that error. Though it was not impacting the flow but still its occurrence is random. Please find the error below in nifi-app.log. Logs:2023-01-04 06:21:23,713 ERROR [Index Provenance Events-1] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Eventsorg.apache.lucene.store.AlreadyClosedException: this IndexWriter is closedat org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)at org.apache.lucene.index.IndexWriter.addDocuments(IndexWrite
Hello, I have a Nifi Flow working to ingest logs messages and send it up to ElasticSearch. Everything worked fine for one month but suddenly some issues appeared without changes about size and volumetry of incoming logs messages: - I have a lookupRecord processor to enrich data and the queue become full for it so i guess this processor is the problem but it never had this behavior before- Nifi tell me it's unable to write flowfile content to content repo due to archive file size constraint- The input processor of the flow is a listenUDP and warn me that the internal queue is full I tried to reboot everything, obsioulsy, and also to increase JVM memory settings to Xmx12g to be sure it's not a lack of memory.I saw that the archive cleanup issue could be a bug from Nifi but I'm actually on 1.17 and even tried to move to 1.20 and everything is same. Any idea of what could cause those problems ?Thanks
We have recentely upgraded our CDP environments from runtime 7.2.12 to 7.2.16 and we are getting a wierd vertex error that we are unable to find the root cause for. anyone have a suggestion where to start looking? ERROR:Vertex vertex_1677501096465_0141_1_61 [Reducer 21] killed/failed due to:OTHER_VERTEX_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1677588363293, firstTasksToStart=[ task_1677501096465_0141_1_61_000000 ], lastTaskFinishTime=1677591606664, lastTasksToFinish=[ task_1677501096465_0141_1_61_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTaskAttempts=0, numKilledTaskAttempts=2, numCompletedTasks=1, numSucceededTasks=0, numKilledTasks=1, numFailedTasks=0}, servicePluginInfo=ServicePluginInfo {containerLauncherName=TezYarn, taskSchedulerName=TezYarn, taskCommunicatorName=TezYarn, containerLauncherClassName=org.apache.tez.dag.ap
I'm using a tool in which I have to point out the master node (driver node) of the Cloudera Spark Cluster (spark :// <some-spark-master> : 7077). Also as I learned, Spark has "Master Node" (Driver Node) and "Worker Nodes".So I decided to go to the Cloudera Web Manager and checked the Configuration Tab of the Spark service, but all I found are "Gateway instance" and "History Server instance". Where are the "Driver instance" and "Worker instance"? I can't add these two instances in the "Add Role Instances" too My guess is that it's in Yarn service configuration, but I can't find anything related to "Master"/"Driver" or "Worker" either. So what is the link to "Spark Master" that ends with 7077? I can't find it anywhere in the Configuration tab
When I start restart Master is OK.but after 10s Master is shutdown.I don't see error in log. I only find CDH 6.3.2 enable kerberos
Hello i have the following problems when i use hbase hbck 10 inconsistencies detected.Status: INCONSISTENT2023-02-01 13:51:01,346 INFO [main] zookeeper.ZooKeeper: Session: 0x1860d1d2ec00091 closed2023-02-01 13:51:01,346 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService2023-02-01 13:51:01,346 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x1860d1d2ec000922023-02-01 13:51:01,346 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down2023-02-01 13:51:01,349 INFO [main] zookeeper.ZooKeeper: Session: 0x1860d1d2ec00092 closed2023-02-01 13:51:01,349 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down Also am unable to create a new table and this is the message that is being displayed when i use create 'users','userinfo'ERROR: java.io.IOException: Table Namespace Manager not ready yet, try again later
Hi all, I am new to Cloudera. Recently I am studying on CDSW and would like to have a try on it. May I ask is there any way that I can try CDSW for free? I don't have a paid account. I saw that there is 60 days trial on CDP Public/Private Cloud, is CDSW included? Hope someone can help me out. Thank you very much.
Hi, I have a workflow that is picking up an Excel file, that contains 3 sheets, and is attempting to run it through a ConvertExcelToCSVProcessor, but it is failing with the error below: Failed to process incoming Excel document. Tried to allocate an array of length 328,219,733, but the maximum length for this record type is 100,000,000. If the file is not corrupt or large, please open an issue on bugzilla to request increasing the maximum allowable size for this record type. As a temporary workaround, consider setting a higher override value with IOUtils.setByteArrayMaxOverride(): org.apache.poi.util.RecordFormatException: Tried to allocate an array of length 328,219,733, but the maximum length for this record type is 100,000,000. If the file is not corrupt or large, please open an issue on bugzilla to request increasing the maximum allowable size for this record type. As a temporary workaround, consider setting a higher override value with IOUtils.setByteArrayMaxOverri
Question;We have around 6k tables under Hdfs as directories and when we use hdfslist processor in nifi its listing everything and killing the cluster . How can we select limited number of tables may 100 or 200 instead of listing everything ?
I followed https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/securing-hive/topics/hive-enable-tls.html to configure Hive Kerberos with SSL. There was no issue, however , when I tried to obtain/display certs on client side to be used in connection, I obtained the following: openssl s_client -connect my_host:10000 -showcerts CONNECTED(00000005)read:errno=0---no peer certificate available---No client certificate CA names sent---SSL handshake has read 0 bytes and written 287 bytes---New, (NONE), Cipher is (NONE)Secure Renegotiation IS NOT supportedCompression: NONEExpansion: NONENo ALPN negotiatedSSL-Session: Protocol : TLSv1.3 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Start Time: 1675956752 Timeout : 7200 (sec) Verify return code: 0 (ok)
Hello, I'm facing issue while executing query (show databases;) on hive. Following is my setup: Cluster : CDP-7.1.7 Components Installed : HDFS, Yarn, Yarn Queue Manager, Oozie, Schema Registry, Hive, Ziplin, Spark, Tez and Zookeeper. I have accessed hive using beeline, but when I tried to execute 'show databases' it is giving following error : ERROR : FAILED: Hive Internal Error: org.apache.hadoop.hive.ql.metadata.HiveException(Error while invoking PostHook. hooks: java.lang.RuntimeException: java.lang.NullPointerExceptionat com.cloudera.navigator.audit.hive.HiveAuditPipeline.getInstance(HiveAuditPipeline.java:50)at com.cloudera.navigator.audit.hive.AbstractHiveExecHookContext.getPipeline(AbstractHiveExecHookContext.java:86)at com.cloudera.navigator.audit.hive.AbstractHiveExecHookContext.run(AbstractHiveExecHookContext.java:69)at org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296)at org.apache.hadoop.hive.ql.HookRunner.runPostE
Hello, I have a question about the performance of the local index in Apache Phoenix For example, I have a table TBL_X having 6 columns (A , B ,C ,D E, F).1. Primary Key is (A, B) 2. One of the query patterns is "Select * from TBL_X where A = 'val1' and C = 'val3' " For the better performance of the above query, I try to build the local index. Does [ Local index on (C, A) ] perform better than [ Local index on (C) ] ?? or Will there be no performance difference since A is already in the first part of the PK ??
I've installed and configured MIT Kerberos for Windows Hive ODBC connection.I could not create Kerberos ticket for more than one user, the new ticket created overwrites the existing ticket. Hence multiple user accessing that Windows machine could not obtain a valid ticket to connect to Hive ODBC connections.I've set krbs.conf from the Hive server and KRB5CCNAME is set to C:\temp\krb5cache.Please let me know how to configure this to have more valid tickets for each user.
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.