Find answers, ask questions, and share your expertise
Recently active
I have a 3 node NiFi cluster set up and running in our environment. I used a user called "ec2-user" to perform the NiFi install. For this initial install (in a development environment) I used the NiFi Toolkit to create the certificates for the 3 nodes as well as for the ec2-user I would now like to use the NiFi registry tool to create additional NiFi users. I have been following this post by alim: Setting Up a Secure Apache NiFi Registry - Cloudera Community - 247753. I started with "Registry Configuration" in this post since I had already used the Toolkit to generate the keystore, truststore and a client certificate for the ec2-user. I copied the keystore and truststore to the conf directory of the Registry install. I then copied the values rom the keystore and truststore properties from the nifi.properties file into the corresponding values in the nifi-registry.properties file. I also modified the HTTP and HTTPS web properties as indicated in the post. I then
Hello,Does nipyapi package works with windows 10 (64bit)?I have tried to install nipyapi on two machines, one with winpython (manual installation of downloaded packages) another with anaconda (installation with pip).Both failed, obviously due a number of missing obscure dependencies like libxml2. I have only a little knowledge, but the installation looked like a tough stuff under windows 10. Is nipyapi recommended at all or are there better alternatives? Best regardsJaro
Has anyone already configured the processor for twitter?GetTwitter return 404ConsumeTwitter return 403
Hello,I built a flow to pull data from the source (Oracle) and push it into the data warehouse. (Oracle). The ETL procedure log must be created in a database, for example. FLOW_ID, FLOW_NAME, START_TIME, END_TIME, STATUS, and ERROR_MESSAGE How do I go about doing this? Flow Example: Log message into database like below
First attempt:Command Arguments = ${absolute.path:append('*.log')};${absolute.path:append('logfiles')}Command Path = /bin/mvNo working directory setExecution error = /bin/mv: cannot stat â/path/input/*.logâ: No such file or directory So I ran "/bin/mv /path/input/*.log /path/input/logfiles" from putty — no errors, files moved. Second attempt:Based on some online troubleshooting, e.g., https://unix.stackexchange.com/questions/77007/mv-cannot-stat-no-such-file-or-directory-in-shell-scriptSame as first, but first argument changed to "${absolute.path}"*.logExecution error = same as above, except double quotes around first path Third attempt:Command Arguments = *.log;logfilesCommand Path = /bin/mvWorking Directory = ${absolute.path}Execution error = /bin/mv: cannot stat â*.logâ: No such file or directory So in putty, I cd into /path/input and type "/bin/mv *.log logfiles" — no errors, files moved. Many folks online having the same/similar problem say "my bad, I
Hi Team, Prod master node is not coming up. Getting below error, could you pls tell me how to resolve the issue as the data is very important. 2023-01-23 09:40:50,748 ERROR [master/ctrlsu-hbaseRS1:16000:becomeActiveMaster] master.HMaster: Failed to become active masterjava.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILEDat org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379)at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:319)at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1324)at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1055)at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2184)at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:519)at
I am getting following error message while trying to set up ODBC. I am using SSL connection to connect. [Cloudera] [ImpalaODBC] (100) Error from the Impala Thrift API: authorize: cannot authorize peer
Hello All,Hello Everyone,I installed nifi on a linux server under /opt/nifi_server/nifi. I can connect to nifi using the provided passwords, but when I try to access it remotely, the error "The supplied username and password are not valid" appears. Earlier, it was working fine. I started facing issues when I removed the NIFI folder, re-extracted it to the same location, and started the services.Any assistance would be greatly appreciated. Thanks,Shakib
We are upgrading our cluster from HDP 2.6.5 to HDP-7.1.70 :During the upgrading process ( about 65 %), it fails to restart the mapreduce/historyserver with the below error.Unable to find a solutionAny idea or help ? raceback (most recent call last):File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/historyserver.py", line 126, in <module>HistoryServer().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in executemethod(env)File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 987, in restartself.stop(env, upgrade_type=upgrade_type)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/historyserver.py", line 47, in stopimport paramsFile "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/params.py", line 29, in <module>from params_linux import *File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/s
Good morning, afternoon, evening..I am a bit disappointed because I start recently a experience like agent for Cloudera.. well, I start slowly getting my skills done for moving forward.But unfortunately, since few days, I get trouble with the missions. I need to do all the time an deposit for clear up the overdrawn. in the WhatsApp group, some try to reassure me and some don’t have any empathy for my reaction. why all the time, I get this high package?in silver membership, I was wondering if I could be in this situation so often. well, now here we go again.. I am stock and I put all my money in for finish this mission, 39/40. Again!!! Why ????that’s horrible this feeling to lose everything like this. I suppose to get some profit no troubles. I don’t have any funds and I don’t know what I can do.
Hi everyone!Can somebody assist me, please?I have a task to consume data from different WebSocket servers (hosted on different IPs). Each WebSocket server has a lot of sensors producing messages, and I need to subscribe and listen to each of the sensors. I need connections to stay alive.This is a simplified communication diagram: As I understood, I can use a ConnectWebSocket processor with JettyWebSocketClient under the hood to create a connection to a particular WebSocket server. The problem I faced with is that I can not dynamically create ClientId in Connect WebSocket processor.That leads to a problem. Let's say, there are 50 sensors producing messages on each of the 3 WebSocket servers I have. I will receive the WebSkocet server's IP address and specific sensor id from upstream processors. I can dynamically change the IP address in JettyWebSocketClient so that I can connect to 3 different servers, but after that, I can not distinguish sensors within ea
Where are the Spark application logs located, and what is the default path of that directory?
Hi everyone,Please find root cause and why yarn resourcemanager not autofailover on my cluster ?I get message ERROR on fatal event from my yarn-resource.log on yarn rm node (Active) below:2023-02-25 08:04:03,805 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1670355762504_0824_01_000007 Container Transitioned from ACQUIRED to RELEASED2023-02-25 08:29:08,810 WARN org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 6669ms for sessionid 0x1000048d78800012023-02-25 08:29:08,810 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 6669ms for sessionid 0x1000048d7880001, closing socket connection and attempting reconnect2023-02-25 08:29:08,911 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session disconnected. Entering neutral mode...2023-02-25 08:29:08,911 WARN org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService: Lost contact with
sqoop import --connect 'jdbc:sqlserver://X.X.X.X:1433;database=TOTO' --table TOTO --username=toto --password=toto SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]23/03/25 10:33:22 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7.7.1.8.0-80123/03/25 10:33:22 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.23/03/25 10:33:22 INFO manager.SqlManager: Using default fetchSize of 100023/03/25 10:33:22 INFO tool.CodeGenTool: Beginning code generation23/03/25 10:33:23 E
Hello,I have built a custom Type with no Entities. However when I try to delete it I get an error, "Given type InformationAssets has references". When I pull back that Type using the API I can see it has some built-in relationship:"relationshipAttributeDefs": [{"name": "meanings","typeName": "array<AtlasGlossaryTerm>","isOptional": true,"cardinality": "SET","valuesMinCount": -1,"valuesMaxCount": -1,"isUnique": false,"isIndexable": false,"includeInNotification": false,"searchWeight": -1,"relationshipTypeName": "AtlasGlossarySemanticAssignment","isLegacyAttribute": false}]There are no entities under this Type, so there are no Glossary Terms assigned to it. I'm not sure how this relationship was created in the first place. I've deleted relationshipDefs before, but this one "AtlasGlossarySemanticAssignment" appears to be some built-in relationship. Does anyone know how I can remove this in order to delete the Type?Thanks in advance!
Current version: 1.6.0Upgrade version: 1.8.0Applying the cluster, accessing the ui results in an error.(x.x.x.x is hostname, my server ip..)The key was not created as a wildcard.---UI Error---javax.net.ssl.SSLPeerUnverifiedException: Hostname x.x.x.x not verified: certificate: sha256/cj+QcD4F2YafhbXSeRmuLznbS24EWVhAvCbU71aVGWo= DN: CN=x,x,x,x, OU=NIFI subjectAltNames: [x.x.x.x]---api access error---https://x.x.x.x:1234/nifi-api/flow/current-userjavax.net.ssl.SSLPeerUnverifiedException: Hostname x.x.x.x not verified: certificate: sha256/!!!#$@!#qtA3h/Nhqm7wcLzM= DN: CN=x.x.x.x, OU=NIFI subjectAltNames: [x.x.x.x]---log Error---2019-04-08 08:26:28,522 WARN [Replicate Request Thread-5] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/current-user to x.x.x.x:9443 due to javax.net.ssl.SSLPeerUnverifiedException: Hostname x.x.x.x not verified: certificate: sha256/IiBnANAzEOL~~~qm7wcLzM= DN: CN=x.x.x.x, OU=NIFI subjectAltNames: [x
So guys,This time I could really use your help with something because I cannot figure this on my own and neither do I know where to look in the source code exactly.My flow is as follows:ConsumeKafka ----> MergeContent (as I have plenty of small files I prefer to merge them in bigger files for further processing) ----> ReplaceText (I have some empty spaces and I want them removed) ---> PartitionRecord. The problems comes here, in PartitionRecord. I have defined two Controller Services, one Record Reader (CSVReader, with a pre-defined working schema) and and Record Writer (ParquetRecordSetWriter, with the same exact schema as in the CSV reader). I have no strange data types, only a couple of FLOATs and around 100 STRINGS. I defined a property called time, which extracts the value from a field in our File. Unfortunately, when executing the flow, I keep on getting the following error message:" PartitionRecord[id=3be1c42e-5fa9-3144-3365-f568bb616028] Processing halted: yi
Hi all, I am practicing the feature in CDSW. When trying to do an experiment, I got an error in the session as below: -------error-------import pickleimport cdswmodel = pickle.load(open('model.pkl', 'rb'))ModuleNotFoundError: No module named 'sklearn'ModuleNotFoundError Traceback (most recent call last)/tmp/ipykernel_115/2912545845.py in <module>----> 1 model = pickle.load(open('model.pkl', 'rb'))ModuleNotFoundError: No module named 'sklearn'Engine exited with status 1.-------error------- However, in the Build session in Experiment, I see Sklearn is installed in the docker container. -----build-log-----Step 1/5 : FROM docker.repository.cloudera.com/cloudera/cdsw/ml-runtime-workbench-python3.8-standard:2022.11.2-b2---> 7ffae291c607Step 2/5 : WORKDIR /home/cdsw---> 22e8b9772338Removing intermediate container 404b7da93746Step 3/5 : COPY sources /home/cdsw---> 90d2c92ea316Removing intermediate container 959005050d3cStep 4/5 : RUN su cdsw -c "mkdi
Hi all, I am exploring the features in my CDP cluster. I added Spark service to the cluster, when I try to study Spark and run pyspark in terminal, I got the following error: Type "help", "copyright", "credits" or "license" for more information.Warning: Ignoring non-Spark config property: hdfsWarning: Ignoring non-Spark config property: ExitCodeExceptionWarning: Ignoring non-Spark config property: atSetting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).23/03/29 02:47:40 WARN conf.HiveConf: HiveConf of name hive.masking.algo does not exist23/03/29 02:47:43 WARN conf.HiveConf: HiveConf of name hive.masking.algo does not exist23/03/29 02:47:49 ERROR spark.SparkContext: Error initializing SparkContext.java.io.FileNotFoundException: File file:/home/asl/2023-03-28 23:17:30,775 WARN [TGT Renewer for asl@MY.CLOUDERA.LAB] security.UserGroupInformation (UserGroupInformation.java:run(1026)) - Exception encountered
I download Nifi flow definition on a Process Group. Then I search the real id of a processor and get nothing. Why the definition does not keep everything as same as those in a Nifi flow? I need to compare two Nifi flows in two environment, DEV and UAT. But the definition does not keep all processors' identifier. It is so difficult for me to find differences.
I want to remove a node from cluster and inside the cluster all node's need to updated with removed node ,example : i have a cluster with 4 nodes node -1 , 2 [ Primary ], 3 , 4 i want to remove node -3 from cluster , and it should be updated automatically in [ authorizers.xml ] to remaining working nodes. please help its bit urgent
We are running a spark streaming job which reads data from kafka and writes to RDBMS. We dont want this job to fail easily due to minor fluctuations in cluster health/issue. Write now spark job has configuration to retry 5 attempts before the whole job fails. But all these retries are happening in quick succession , one after another. Is there a way we can put some delay/sleep time between retry attempts for this job?
Does Apache NiFi 1.19.1 or 1.20.0 Have Kotlin in the Software Stack? And, how can I confirm this? Thank you
Hello, I'm trying to start/stop a processor using the NiFi API in a kerberized NiFi cluster (Cloudera Data Flow) but the PUT request to do it fails in identifying the authenticated user. Other requests like get processor information, get current user information, etc. works fine. I'm running the process in a bash script in the same machine as the NiFi node. I have just one node in this environment with load balancer. Here is the process I follow: 1. Create kerberos token in the system (kinit my.username) 2. Create NiFi token calling the /access/kerberos endpoint curl -k -X POST --negotiate -u : "https://node-ip:8443/nifi-api/access/kerberos" 3.Store the token in a variable and check I'm authenticated curl -k --header "Authorization: Bearer $token" "https://node-ip:8443/nifi-api/access" {"accessStatus":{"identity":"my.username","status":"ACTIVE","message":"Access Granted: Token authenticated."}} 4.Get processor
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.