Solved

Tried to add ambari metrics to a cluster and the metrics collector keeps failing on a node due to some issuehere are the logs:stderr:Traceback (most recent call last):File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_collector.py", line 90, in <module>AmsCollector().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in executemethod(env)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_collector.py", line 51, in startams_service('collector', action = 'start')File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunkreturn fn(*args, **kwargs)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/ams_service.py", line 84, in ams_serviceuser=params.ams_userFile "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__self.env.run()File "/usr/lib/ambari-agent

I have a 3 node NiFi cluster set up and running in our environment.  I used a user called "ec2-user" to perform the NiFi install. For this initial install (in a development environment) I used the NiFi Toolkit to create the certificates for the 3 nodes as well as for the ec2-user   I would now like to use the NiFi registry tool to create additional NiFi users. I have been following this post by alim: Setting Up a Secure Apache NiFi Registry - Cloudera Community - 247753.   I started with "Registry Configuration" in this post since I had already used the Toolkit to generate the keystore, truststore and a client certificate for the ec2-user. I copied the keystore and truststore to the conf directory of the Registry install. I then copied the values rom the keystore and truststore properties from the nifi.properties file into the corresponding values in the nifi-registry.properties file. I also modified the HTTP and HTTPS web properties as indicated in the post. I then

Hello,Does nipyapi package works with windows 10 (64bit)?I have tried to install nipyapi on two machines, one with winpython (manual installation of downloaded packages) another with anaconda (installation with pip).Both failed, obviously due a number of missing obscure dependencies like libxml2. I have only a little knowledge, but the installation looked like a tough stuff under windows 10. Is nipyapi recommended at all or are there better alternatives? Best regardsJaro

Has anyone already configured the processor for twitter?GetTwitter return 404ConsumeTwitter return 403  

First attempt:Command Arguments = ${absolute.path:append('*.log')};${absolute.path:append('logfiles')}Command Path = /bin/mvNo working directory setExecution error = /bin/mv: cannot stat â/path/input/*.logâ: No such file or directory So I ran "/bin/mv /path/input/*.log /path/input/logfiles" from putty — no errors, files moved. Second attempt:Based on some online troubleshooting, e.g., https://unix.stackexchange.com/questions/77007/mv-cannot-stat-no-such-file-or-directory-in-shell-scriptSame as first, but first argument changed to "${absolute.path}"*.logExecution error = same as above, except double quotes around first path Third attempt:Command Arguments = *.log;logfilesCommand Path = /bin/mvWorking Directory = ${absolute.path}Execution error = /bin/mv: cannot stat â*.logâ: No such file or directory So in putty, I cd into /path/input and type "/bin/mv *.log logfiles" — no errors, files moved. Many folks online having the same/similar problem say "my bad, I

Hi Team,  Prod master node is not coming up. Getting below error, could you pls tell me how to resolve the issue as the data is very important.  2023-01-23 09:40:50,748 ERROR [master/ctrlsu-hbaseRS1:16000:becomeActiveMaster] master.HMaster: Failed to become active masterjava.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILEDat org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379)at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:319)at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1324)at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1055)at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2184)at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:519)at

I am getting following error message while trying to set up ODBC. I am using SSL connection to connect. [Cloudera] [ImpalaODBC] (100) Error from the Impala Thrift API: authorize: cannot authorize peer

 

Hello All,Hello Everyone,I installed nifi on a linux server under /opt/nifi_server/nifi. I can connect to nifi using the provided passwords, but when I try to access it remotely, the error "The supplied username and password are not valid" appears. Earlier, it was working fine. I started facing issues when I removed the NIFI folder, re-extracted it to the same location, and started the services.Any assistance would be greatly appreciated. Thanks,Shakib   

We are upgrading our cluster from HDP 2.6.5 to HDP-7.1.70 :During the upgrading process ( about 65 %), it fails to restart the mapreduce/historyserver with the below error.Unable to find a solutionAny idea or help ? raceback (most recent call last):File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/historyserver.py", line 126, in <module>HistoryServer().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in executemethod(env)File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 987, in restartself.stop(env, upgrade_type=upgrade_type)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/historyserver.py", line 47, in stopimport paramsFile "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/params.py", line 29, in <module>from params_linux import *File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/s

Good morning, afternoon, evening..I am a bit disappointed because I start recently a experience like agent for Cloudera.. well, I start slowly getting my skills done for moving forward.But unfortunately, since few days, I get trouble with the missions. I need to do all the time an deposit for clear up the overdrawn. in the WhatsApp group, some try to reassure me and some don’t have any empathy for my reaction. why all the time, I get this high package?in silver membership, I was wondering if I could be in this situation so often. well, now here we go again.. I am stock and I put all my money in for finish this mission, 39/40. Again!!! Why ????that’s horrible this feeling to lose everything like this. I suppose to get some profit no troubles.  I don’t have any funds and I don’t know what I can do. 

Where are the Spark application logs located, and what is the default path of that directory?

sqoop import --connect 'jdbc:sqlserver://X.X.X.X:1433;database=TOTO' --table TOTO --username=toto --password=toto SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]23/03/25 10:33:22 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7.7.1.8.0-80123/03/25 10:33:22 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.23/03/25 10:33:22 INFO manager.SqlManager: Using default fetchSize of 100023/03/25 10:33:22 INFO tool.CodeGenTool: Beginning code generation23/03/25 10:33:23 E

Hello,I have built a custom Type with no Entities.  However when I try to delete it I get an error, "Given type InformationAssets has references".  When I pull back that Type using the API I can see it has some built-in relationship:"relationshipAttributeDefs": [{"name": "meanings","typeName": "array<AtlasGlossaryTerm>","isOptional": true,"cardinality": "SET","valuesMinCount": -1,"valuesMaxCount": -1,"isUnique": false,"isIndexable": false,"includeInNotification": false,"searchWeight": -1,"relationshipTypeName": "AtlasGlossarySemanticAssignment","isLegacyAttribute": false}]There are no entities under this Type, so there are no Glossary Terms assigned to it.  I'm not sure how this relationship was created in the first place.  I've deleted relationshipDefs before, but this one "AtlasGlossarySemanticAssignment" appears to be some built-in relationship.  Does anyone know how I can remove this in order to delete the Type?Thanks in advance!

Current version: 1.6.0Upgrade version: 1.8.0Applying the cluster, accessing the ui results in an error.(x.x.x.x is hostname, my server ip..)The key was not created as a wildcard.---UI Error---javax.net.ssl.SSLPeerUnverifiedException: Hostname x.x.x.x not verified: certificate: sha256/cj+QcD4F2YafhbXSeRmuLznbS24EWVhAvCbU71aVGWo= DN: CN=x,x,x,x, OU=NIFI subjectAltNames: [x.x.x.x]---api access error---https://x.x.x.x:1234/nifi-api/flow/current-userjavax.net.ssl.SSLPeerUnverifiedException: Hostname x.x.x.x not verified: certificate: sha256/!!!#$@!#qtA3h/Nhqm7wcLzM= DN: CN=x.x.x.x, OU=NIFI subjectAltNames: [x.x.x.x]---log Error---2019-04-08 08:26:28,522 WARN [Replicate Request Thread-5] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/current-user to x.x.x.x:9443 due to javax.net.ssl.SSLPeerUnverifiedException: Hostname x.x.x.x not verified: certificate: sha256/IiBnANAzEOL~~~qm7wcLzM= DN: CN=x.x.x.x, OU=NIFI subjectAltNames: [x

Hi all, I am practicing the feature in CDSW. When trying to do an experiment,  I got an error in the session as below: -------error-------import pickleimport cdswmodel = pickle.load(open('model.pkl', 'rb'))ModuleNotFoundError: No module named 'sklearn'ModuleNotFoundError Traceback (most recent call last)/tmp/ipykernel_115/2912545845.py in <module>----> 1 model = pickle.load(open('model.pkl', 'rb'))ModuleNotFoundError: No module named 'sklearn'Engine exited with status 1.-------error------- However, in the Build session in Experiment, I see Sklearn is installed in the docker container. -----build-log-----Step 1/5 : FROM docker.repository.cloudera.com/cloudera/cdsw/ml-runtime-workbench-python3.8-standard:2022.11.2-b2---> 7ffae291c607Step 2/5 : WORKDIR /home/cdsw---> 22e8b9772338Removing intermediate container 404b7da93746Step 3/5 : COPY sources /home/cdsw---> 90d2c92ea316Removing intermediate container 959005050d3cStep 4/5 : RUN su cdsw -c "mkdi

Hi all, I am exploring the features in my CDP cluster. I added Spark service to the cluster, when I try to study Spark and run pyspark in terminal, I got the following error: Type "help", "copyright", "credits" or "license" for more information.Warning: Ignoring non-Spark config property: hdfsWarning: Ignoring non-Spark config property: ExitCodeExceptionWarning: Ignoring non-Spark config property: atSetting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).23/03/29 02:47:40 WARN conf.HiveConf: HiveConf of name hive.masking.algo does not exist23/03/29 02:47:43 WARN conf.HiveConf: HiveConf of name hive.masking.algo does not exist23/03/29 02:47:49 ERROR spark.SparkContext: Error initializing SparkContext.java.io.FileNotFoundException: File file:/home/asl/2023-03-28 23:17:30,775 WARN [TGT Renewer for asl@MY.CLOUDERA.LAB] security.UserGroupInformation (UserGroupInformation.java:run(1026)) - Exception encountered

I download Nifi flow definition on a Process Group. Then I search the real id of a processor and get nothing. Why the definition does not keep everything as same as those in a Nifi flow? I need to compare two Nifi flows in two environment, DEV and UAT. But the definition does not keep all processors' identifier. It is so difficult for me to find differences.

Does Apache NiFi 1.19.1 or 1.20.0 Have Kotlin in the Software Stack?  And, how can I confirm this? Thank you

Hello, I'm trying to start/stop a processor using the NiFi API in a kerberized NiFi cluster (Cloudera Data Flow) but the PUT request to do it fails in identifying the authenticated user. Other requests like get processor information, get current user information, etc. works fine. I'm running the process in a bash script in the same machine as the NiFi node. I have just one node in this environment with load balancer. Here is the process I follow: 1. Create kerberos token in the system (kinit my.username) 2. Create NiFi token calling the /access/kerberos endpoint curl -k -X POST --negotiate -u : "https://node-ip:8443/nifi-api/access/kerberos"​  3.Store the token in a variable and check I'm authenticated curl -k --header "Authorization: Bearer $token" "https://node-ip:8443/nifi-api/access" {"accessStatus":{"identity":"my.username","status":"ACTIVE","message":"Access Granted: Token authenticated."}}​ 4.Get processor

Hi Team, It would be appreciated if someone please guide me how to set spark memory for spark job, where cluster utilization should take 1%-2% memory only for each spark job. Please share math's logic how to calculate on below cluster node details as - #1 How many working Nodes Cluster we have currently? |>Nodemanagers:166 >Datanodes:159  #2 How many Cores per Node we have currently ? |>64 Cores  #3 How much GB RAM per node. we have currently ? |>503 GB   ==== Wanted to calcuate for spark job ===#1driver-memory#2 executor-memory#3 driver-cores#4 executor-cores#5 num-executor========================Please suggest if any additional parameter help to tune the spark job (execution time and cluster utilization) wise.

Team, My current Apache spark jobs are executing very slow, could you please help me to make it Run Faster. Looking for thumb rule to make spark job it RUN FASTER. Thanks in advance for your kind support always. 

I'm using a tool in which I have to point out the master node (driver node) of the Cloudera Spark Cluster (spark :// <some-spark-master> : 7077). Also as I learned, Spark has "Master Node", "Driver Node" and "Worker Nodes".So I decided to go to the Cloudera Web Manager and checked the Configuration Tab of the Spark service, but all I found are "Gateway instance" and "History Server instance". Where are the "Driver instance" and "Worker instance"? I can't add these two instances in the "Add Role Instances" too  My guess is that it's in Yarn service configuration, but I can't find anything related to "Master", "Driver" or "Worker" either.  So what is the link to "Spark Master" that ends with 7077 (what is the Node)? I can't find it anywhere in the Configuration tab

I'm facing weird issue, not sure why Spark is behaving like this.samplefile.txt:COL1|COL2|COL3|COL4 "1st Data"|"2nd ""\\\\P"" data"|"3rd data"|"4th data" This is my spark code to read data:val df = spark.read.format("csv").option("header","true").option("inferSchema","true").option("delimiter","|").load("\samplefile.xtx")df.show(false)Some how it is combining 2 columns data into one. Spark Scala : 2.4 VersionAny idea why spark is behaving like this.

This error has been reported earlier https://community.cloudera.com/t5/Support-Questions/org-apache-hadoop-util-DiskChecker-DiskErrorException-No/m-p/361337#M238564  I have a 3 node hadoop cluster for doing my research on Medical Side EffectsEach node has Ubuntu 18.04.5 LTS.CDH 6.3.4-1.cdh6.3.4.p0.6751098 Basically I have not been able to run any MR jobs or Hive queries/jobsHive select * works of course because select * does not launch an MR job...However querying a Hive table with just 2 cols and 2 rows fails  hive -e "show create table beatles"CREATE EXTERNAL TABLE `beatles`(`id` int,`name` string)COMMENT 'Beatles Group'ROW FORMAT SERDE'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'WITH SERDEPROPERTIES ('field.delim'=',','line.delim'='\n','serialization.format'=',')STORED AS INPUTFORMAT'org.apache.hadoop.mapred.TextInputFormat'OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'LOCATION'hdfs-//hp8300one:8020/data/demo'TBLPROPERTI