Find answers, ask questions, and share your expertise
Recently active
Hi,I am unable to start cloudera management services. The "Continue" button is enabled and all the nodes are added in cloudera manager as shown in ss below. I haven't created the cluster yet and haven't distributed the parcels aswell. I face the same issue when I try to create cluster it does not move forward even the continue button is enabled as seen in screenshot below: Kindly help me resolve the issue. Regards
Hi,There are two HikariCP jars in the class path:/data/perftools/spark-prod-reports-3.0.0/jars/HikariCP-2.5.1.jar/data/perftools/collector-spark-sandbox/lib/HikariCP-3.2.0.jarThe 1st is pre-packaged with spark, the 2nd is included by me. Since /data/perftools/spark-prod-reports-3.0.0/bin/spark-submit's list of jars is checked first before checking my jar list, the classloader is loading HikariCP-2.5.1.jar instead of the desired one, HikariCP-3.2.0.jar that is provided via the flag --jars . The problem is shown below. Notice the 4th and 5th lines are loading Hikari from the default spark package jar folder, not from my jar list located at /data/perftools/collector-spark-sandbox/lib/.[Loaded slick.util.ConfigExtensionMethods$ from file:/data/perftools/collector-spark-sandbox/lib/slick_2.12-3.3.3.jar][Loaded slick.jdbc.JdbcDataSource$$$Lambda$1478/767152605 from slick.jdbc.JdbcDataSource$][Loaded slick.jdbc.hikaricp.HikariCPJd
Hi, I'm trying to setup the data engineering experiences (ECS) on top of a CDP base installation.These are licensed installations, so we first installed and configured a base cluster and we have been working on it.We want to setup the data experiences so we can move some work between the cdp public cloud and the cdp private cloud experiences since the control planes are similar. The goal is to work from the CDP private ECS enabled cluster and not use the base cluster unless any components are needed for the ECS/experiences cluster from the base. Trying to comb through the documentation to find the relationship between CDP base and CDP ECS enabled experiences cluster. What roles/manager/services are being used from base cluster in the experiences cluster? Understand a base "core" (manager/runtime etc.) is needed but not sure if I understand the dependencies between the two. The documentation says start with a base cluster and install ECS - well there a many flavors for base te
Hi, I am unfamiliar with Cloudera and tasked with enforcing a log retention policy on all services, currently working in NIFI. Our NIFIis part of CFM-2.2.3.2.I have changes to make to logback.xml, but they are overwritten when a new process directory is created. What is the source of the config.zip that is downloaded to create the process directory and how can I rebuild config.zip to include a copy of logback.xml with my changes? I couldn't find this in the docs I had access to, nor clearly in the cloudera logs I examined. Many thanks in advance!
We have a NIFI cluster which have 3 nodes. All the nodes were working fine earlier. But now if all 3 nodes are connected to cluster and we right click on NIFI UI canvas and select "Configure" then we are getting error - "An unexpected error has occurred. Please check the logs for additional details." Below are more details fetched from network tabs on inspection on chrome. The same error can be found if we access NIFI UI using any node's hostname - Request URL:https://<node1>:<port>/nifi-api/flow/process-groups/{process_group_id}/controller-servicesRequest Method:GETStatus Code:500 Internal Server ErrorIf we disconnect node 3 from cluster then we are able to click on "Configure" and list all controller services on NIFI UI. Issue seems to be with node 3. We have synced flow.xml.gz file from node2 to other node ie.node1, node3 and also restarted the NIFI. But still the issue is there.We have got this error in nifi-user.log -"o.a.nifi.web.api.config.
Hi community, I am a newbie to Nifi and unfortunately, We don't have a system admin with the knowledge of Nifi installation for production and I have to build a 3 node secured cluster without Zookeeper. I understood that with the NiFi 0.x baseline, the cluster has a cluster manager which serves a similar role and did not use a zookeeper at all. But I am not sure how to use NiFi 0.x baseline, If someone can help me with steps/guidance to configure using NiFi 0.x baseline to set up a three-node cluster that will be great. Best,Shanoj
Hi! So I am very confused about how regular expressions and groups work in nifi.I read documentation and I saw that ExtractText processors always exctracts more attributes than needed somehow.So I have this file with the line like this9999, text And I wrote regular expression to extract value 9999 for attribute call number. (\d{4})But instead of one attribute number I am getting number0, number and number1 attributes. Can someone please explain me why is this happening, because documentation explanation is quite complex really. Thank you beforehand!
Hello! Sorry for my english. I am completely new to nifi and I am learning SplitText processor. So here's the case. I have the comma separated txt file, something like this:KeyWord, SomeInformation <---1st line is schema.KeyWord1, "information"KeyWord2, "information"KeyWord1, "another information"KeyWord2, "another information" and so on. So the question is how can I split this file on a few files based on KeyWord? So the every line with KeyWord1 go to one file, every line with KeyWord2 go to another file and so on? Thank you beforehand!
Hi All, We did the disk partition activity on one of our nodes on which NN services were running. After rebooting Active NN goes down. Need urgent solution on this. Please help ---Version---Ambari version -> 2.6OS -> Linux (centos 7.9.2009)HDP-2.6.5.1100Attaching the required NN log 2022-04-30 06:48:30,683 WARN namenode.NameNode (NameNodeRpcServer.java:errorReport(1548)) - Fatal disk error on DatanodeRegistration(10.0.3.26:1019, datanodeUuid=a500e7d8-3052-4aa8-a18d-e768a15d22da, infoPort=1022, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-56;cid=CID-dc49d075-7a70-4ddf-b66b-b39b4b445af0;nsid=897785728;c=0): DataNode failed volumes:/grid1/hadoop/hdfs/data/current; 2022-04-30 06:48:34,123 ERROR namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(410)) - Error: flush failed for (journal JournalAndStream(mgr=FileJournalManager(root=/grid1/hadoop/hdfs/namenode), stream=EditLogFileOutputStream(/grid1/hadoop/hdfs/namenode/current/edits_inprogres
Hi all, i saw this guide: https://community.cloudera.com/t5/Support-Questions/how-to-change-password-for-ambari-db-post-install/td-p/126685 but it was for Ambari 2.2.0 and i was wondering if do you think is the same for Ambari 2.7.4.I have in ambari.properties: - server.jdbc.user.passwd=/etc/ambari-server/conf/password.datDoing "cat /etc/ambari-server/conf/password.dat " i see the default password (bigdata) in plain, so NO PASSWORD ENCRYPTION ENABLED.I'm using Postgres 9.3Thanks!!
Hello Community, I am going through this tutorial https://www.cloudera.com/tutorials/setting-up-a-spark-development-environment-with-java.html and I am trying to submit it through Oozie work flow. In Hue I am going to Query > Scheduler > Workflow then dropping Java program to actions and then uploading jar file then adding the main class which is Hortonworks.SparkTutorial.Main. When I click to run the Oozie work flow job I keep getting the error of: "Caused by: java.lang.ClassNotFoundException: Class Hortonwork.SparkTutorial.Main not found". I'm using intelliJ to do this project so I hold down ctrl and hover over Main and it takes me to MANIFEST.MF file and it says Main-Class: Hortonworks.SparkTutorial.Main so I'm getting the main class definition right I feel like. I cannot figure out why it is saying it can't find my class.
Hello Team, In a customer cluster facing the issue with too many open file descriptors. The CDP version is 7.1.5 Bad : Open file descriptors: 31,364. File descriptor limit: 32,768. Percentage in use: 95.72%. Critical threshold: 70.00%. Can you please share how to mitigate this issue? Is it okay to increase the Maximum Process file descriptors and what would be the recommended value? Best Regards
we need to block the DDL (create, drop, insert, update) for whoever is using IMPALA, is it possible to do this type of blocking? ps: user should only be able to SELECT with IMPALA and nothing else. DDL should only work via HIVE. Is it possible to do this type of blocking configuration?
HiFrom an API I receive a parquet as an Array[Byte] (stored here as parquetPayload), and want to convert that into a DataFrame. My current function is below, and includes a write to disk val tempFilePath = new Path("/tmp/", java.util.UUID.randomUUID().toString + ".tmp")val fs = FileSystem.get(spark.sparkContext.hadoopConfiguration)val stream = fs.create(tempFilePath, true)try { stream.write(parquetPayload, 0, parquetPayload.length)} finally { stream.close()}val df = spark.read.parquet(tempFilePath.toString) This works, but I would like to avoid the write to disk, and keep this all in memory. Is this possible?
Hi Team, I am ex-employee of CLoudera/Hortonworks and completed HDPCA certification in Dec'19. Registered mail ID was smahendrada@cloudera.com. I missed to save my certificate and I dont have the certificate copy. Could you please help me. Thanks,Sruthi
Hi Guyz, please help me to build the python script for cleaning HDFS dir which are older than 3 days.give me the suggestions
Hello !I have a ListenFTP processor opened on a port and when i am trying to connect to it via FileZila i have an error "Failed to retrieve directory listing". The connection seems to be establish first but then this error occurs. Nifi is hosted on an ubuntu server running in a docker image ListenFTP processor is opened on port 2221 I tried to change some configuration in FileZila but nothing worked. The connection works well on localhost. Somone has already encoutered that kind of issue ?
[root]# hiveSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]WARNING: Use "yarn jar" to launch YARN applications.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://
hello all,I am currently running a python version of 3.6.8, Can I still install spark 3 on my cdp cluster?
Hi,I am not able to permanently delete tombstone in Hbase.Tombstone markers are flushed for non-MOB column family but it is not flushed for the MOB column family. Any solution to delete the same so that we save some hdfs space. Regards,Satya
Hi, I am trying to export large hbase table into sequence files. Our requirement is to split the sequence files with specific size limit. We have tried multiple options but output part files are getting generated with large size like 6-7 GB. We used the below sample export command which trying to generate sequence file in a range of 100mb - 135mb, but didn't work as expected. hbase org.apache.hadoop.hbase.mapreduce.Export -D dfs.blocksize=1048576 -D mapred.min.split.size=102400 -D mapred.max.split.size=1048576 \table_name /output/hbase/ Could you please help me here to export the table into small chunks ?
Hi,every one: I have a question.I found there are many "cmd=get_database" log in hadoop-cmf-hive-HIVEMETASTORE.out file.But I haven't done this much.Is it because there are so many other actions that trigger it?
If I took an export of the main flow (NiFi Flow) from a server via the Export to JSON option on the right-click menu, how can I import that into a new server as the top-level flow? I know I can create new process groups with the export. But I want to restore the controller services and parameter contexts. I know I could import into a process group, and then move things out again. But, that would also require re-creating all the controller services back at the top level...which would create new UUIDs...and require updating 400-500 processors manually to map them to the new controller services. I've seen various posts about using NiFi registry and the toolkit. But even when I look at that, I don't see how that works with the top-level group. If there is some way to handle this that will require a minimum of reconfiguration on services, processors or process groups after the import, it would be appreciated.
2021-05-18 16:50:45,740 INFO scm-web-112:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/testConnection2021-05-18 16:50:45,744 INFO scm-web-112:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command HueTestDatabaseConnection with sensitive arguments. Host: DbHost{id=2, hostId=4f19de0c-8fc0-4c39-9ed8-fb65346048ba, hostName=cn2.hadoop.com}2021-05-18 16:50:45,854 INFO scm-web-112:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/testConnection, Status:2002021-05-18 16:50:48,768 INFO scm-web-113:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult2021-05-18 16:50:48,779 INFO scm-web-113:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:2002021-05-18 16:50:51,770 INFO scm-web-105:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operat
Hello, I am trying to install Cloudera Manager 7.4.4 trial from CDP Private Cloud Base Free Trial | Cloudera. The install script worked as expected and I am able to access the CM portal through port 7180. My trial licence is valid for remaining 46 days. But when trying to set up a cluster it constantly fails with the following error /tmp/scm_prepare_node.e4VFgyneFailed to download package. Status code: 404 I am new to Cloudera and already tried to contact sales but no feedback so far. :-(((
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.