Find answers, ask questions, and share your expertise
Recently active
I am trying to add a node in my cluster. However, after trying to register the new node to the cluster. I am getting the error below while registering it.Ambari version. 2.3.4.7OS : Ubuntu 14.04ERROR 2020-04-15 14:10:08,335 NetUtil.py:84 - [Errno 8] _ssl.c:510: EOF occurred in violation of protocolERROR 2020-04-15 14:10:08,336 NetUtil.py:85 - SSLError: Failed to connect. Please check openssl library versions.Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.WARNING 2020-04-15 14:10:08,336 NetUtil.py:112 - Server at https://xxxxxxx:8440 is not reachable, sleeping for 10 seconds...', None)Troubleshooting:- Python version is the same new node and the cluster. I am using python 2.6.- Added also in ambari-client.ini the security protocol TLSv1.- Java version is also the same on all node like the new node. I am using java version "1.8.0_92".- DNS is also properly set. Got positive result when digging fqdn is resolving in the DNS server.- Host file is also pr
Hi. I am trying to write data to hadoop cluster running in HDP 2.6.5 from a local nifi instance. But I receive the following error on writeorg.apache.hadoop.ipc.RemoteException(java.io.IOException): File /home/maria_dev/.2d0282a9-8c6a-4bde-beb1-82eb1b061941 could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.In the HDFS data node logs, I see the below error logs2023-04-18 12:05:13,386 WARN blockmanagement.BlockPlacementPolicy (BlockPlacementPolicyDefault.java:chooseTarget(385)) - Failed to place enough replicas, still in need of 1 to reach 1 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy and org.apache.hadoop.net.NetworkTopology Have validated follow things
Hi, I have flow file with an Avro record. Is there any chance to extract one field value from this record to attribute without converting record to Json format and without using the groovy script. I need to extract field value from content to attribute for using it as a part of cache identifier in the further processor. Thank you.
Hello all,I'm setting up a process in NIFI to download some files from S3 bucket. The process contain a ListS3, UpdateAttribute, FetchS3Object and PutHDFS on a 3 nodes cluster environment. The ListS3 retrieve the files but the files get stuck in the queue before the FetchS3Object and will not move past the FetchS3Object. I don't received any error message and the files would just sit there. Any reason why the files are not moving to the FetchS3Object processor? I have all default settings set on both ListS3 and FetchS3Object.Thanks!
I am trying to insert a flow file into MongoDb which has a createdAt date key record as an attribute.When I inserted that flow file into MongoDb using PutMongo processor, it saves the "createdAt" attribute as a string. I want this to be saved as an ISO date object in mongoDB.When I inserted flow file by sending "2020-08-26T04:00:00.000Z" PutMongo Processor inserts as a string.When I sent like this “ISODate("2020-08-26T04:00:00.000Z”)” it also inserts same string as it is into mongo.When I tried like this ISODate("2020-08-26T04:00:00.000Z”) without double quotation it throws error invalid object. I need output as : {“createdAt”: ISODate("2020-08-26T04:00:00.000Z”)} Kindly help if there is any way to do so.
Hi All,I am getting error when I restart the NiFi server and run the CaptureChangeMySql processor or re-configure any processor and start the processor. It works fine when I creates CaptureChangeMySql processor and run it. When I restart the server this I start getting below error. It works fine again when processor is recreated freshly. Any Idea how I can fix this issue. 2022-07-07 15:10:58,307 ERROR [Timer-Driven Process Thread-8] o.a.n.c.m.processors.CaptureChangeMySQL CaptureChangeMySQL[id=d7f0536f-0181-1000-db94-5950dddc0439] Processing failedorg.apache.nifi.processor.exception.ProcessException: java.io.IOException: COMMIT event received while not processing a transaction (i.e. no corresponding BEGIN event). This could indicate that your binlog position is invalid.at
How to know when NiFi has done processing with all the flowfiles in the processor group? Also is there any way to notify external API?
I am assigning a value to "reasonId" and later using it as a dynamic property in other processor(ExractText). Now, I want to assign this as an dynamic property so that I can store some API response into this. Means reasonId value and Api response both should change evertytime. Here I want this [${reasonId}=(.*)] to be like this:-6099 = Api response (Everytime value of 6099 should change as it is a dynamic variable)But,I am getting it like this${reasonId}=Api response Why there is no 6099 instead of reasonId. Does it means Dynamic property assignment is not working properly? or I can use some jolt spec and all for this? Basically I want reasonId to hold its value 6099 and Api response should be store in 6099 instead of ${reasonId}.
How can I iterate on this array in NIfi, I am extracting this array from an API response in nifi but unable to extract every index value one by one for completing my whole flow.Example: Need 6100 in any variable so that I can use it for further process and in next iteration need 6101 in that variable and so on.Edit: This array is extracting like this only, it is not assigned to any named array such as arr[] or anything. [ "6100", "6101", "6102", "6103", "6104", "6105", "6106", "6107", "6108", "6423", "23382", "354548" ]
Hello! I am new to NiFi. I hope someone here can advise me about my problem with time zone. I have these processors:ListDatabaseTables -> GenerateTableFetch -> ExecuteSQLRecord (writing to csv file by CSVRecordWriter) -> ... ... PutSQL (loading csv file to MySQL using Load Data command) The source DB is Oracle. CSVRecordWriter has the following properties:Schema Write Strategy -> Do Not Write SchemaSchema Access Strategy -> Inherit Record SchemaSchema Name -> ${schema.name}Schema Text -> ${avro.schema}Date Format -> yyyy-MM-ddTime Format -> HH:mm:ssTimestamp Format -> yyyy-MM-dd HH:mm:ss My source DB and the target DB are both in US east time zone. However, I noticed that the output of ExecuteSQLRecord having time values converted to UTC (added to 5 hours). That results in the wrong time values in the target DB. There may be some ways to convert each date/time column individually, but that will require a huge amount of development effort.&nb
Dears, My usecase is to extract user name entered and store it in an attribute and load it into a column of a database. How can I extract user name entered by user while logging in. Thankyou!
HelloI have created a JSON body but I need to reconstruct it so I can send it in a REST API.Input : { "systemId" : "4", "systemName" : "Main", "items.nid" : "123", "items.birth_date" : "1978-01-01", "items.last_name" : "Willson", "items.nationality" : "GN", "formId": 1} Output : { "formId": 1, "systemId" : "4", "systemName" : "Main", "items" { "nid" : "123", "birth_date" : "1978-01-01", "last_name" : "Willson", "nationality" : "GN" }} I used JoltTransformJSON but with no luck always get null as a result, I can't figure how to regroup them based on the name "items.*" . how can I regroup my items or is there a better way to solve this point ?
Greetings Cloudera Community!! Text4shell vulnerability is impacting the apache application which is using commons-text version 1.5 to 1.9 and our application Nifi version 1.16.2 hosted on linux server (Red Hat Enterprise Linux Server 7.9) is using commons-text version 1.8 jar file in lib folder. Can anyone please help to figure out the best solution to handle this vulnerability in our production servers: We have few queries for the vulnerability: 1:Is the nifi version 1.16.2 application is affected by this vulnerability? 2: In Nifi configuration files, we are not using any calls related to StringSubstitutor API. Are we still vulnerable to test4Shell? 3: If nifi version 1.16.2 vulnerable then can we just replace the commons-text jar file from 1.8 to 1.10 in nifi 1.16.2. Is there any impact of this in our prod servers? Please do let us know on this vulnerability for nifi 1.16.2. If it is impacting nifi 1.16.2 version then what would be th
Hi,I'm running a reconciliation application using Spark. This application needs to connect a table from 2 different clusters Hbase table and compare the data. My application is able to connect and get the data from where the application is running on a cluster. but for remote cluster I'm getting below error failed on local exception:failed on local exception:exception is Failed after attempts=4, exceptions: 2023-04-19T17:11:13.491Z, RpcRetryingCaller{globalStartTime=1681924273451, pause=1000, maxAttempts=4}, java.io.IOException: Call to Host:16020 failed on local exception: java.io.IOException: java.lang.RuntimeException: Found no valid authentication method from options 2023-04-19T17:11:14.505Z, RpcRetryingCaller{globalStartTime=1681924273451, pause=1000, maxAttempts=4}, java.io.IOException: Call to host:16020 failed on local exception: java.io.IOException: java.lang.RuntimeException: Found no valid authentication method from options 2023-04-19T17:11:16.522Z, RpcRetryingCaller{gl
Hello everybody Trying to connect NIFI Server to Kafka brokerUse kerberous and SSL_SASLGet error from java : Failed to process session due to Failed to construct kafka producer; Processor Administratively Yielded for 1 sec: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user↳ causes: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user↳ causes: org.apache.kafka.common.KafkaException: Failed to construct kafka producer Already checked some number of Topics here on cloudera but cant find anything that work for me Someone can help? log error
User trying to connect via beeline but connection itself taking long time. Please suugest what is the issue.
Using Sqoop export on CDP cluster (hdfs), I'm trying to export data geometry to Postgresql table but I'm facing a Sqoop ERROR orm.ClassWriter: No Java type for SQL type 1111 for column 'geom'.The 'geom' column definition on PostGis is : geometry (Point,3945).Here is my Sqoop export query :```sqoop export --connect "jdbc:postgresql://$database_server/$database_source_name" \--username $username --password $password_alias \--table $database_table_name \--export-dir "${DIR_OUT_HDFS}" \--fields-terminated-by '|' --lines-terminated-by "\n" \-- --schema $database_table_name 1>>${LOG_FILE} 2>>${LOG_FILE}```Here is Sqoop output error :```ERROR orm.ClassWriter: No Java type for SQL type 1111 for column geomERROR orm.ClassWriter: No Java type for SQL type 1111 for column geomERROR orm.ClassWriter: No Java type for SQL type 1111 for column geomERROR orm.ClassWriter: No Java type for SQL type 1111 for column geomERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerExcep
Dears, In Apache nifi, the flow is stored in the backend as an xml file . That file contains all the configuration and connection information for the nifi flow, including processors, input and output ports,.But we are unable to find that XML file in the backend directory .Could you please suggest the directory path in the nifi server that where exactly we find the XML files? Thank you!
HI ,Hi,I was using NIFI 1.12.0 version and now shiting a job into NIFI 1.18.0 version because of some security concerns. I am facing challenges in that the NIFI queue is not getting empty after successfully data transfer to EventHub. Below is the NIFI
we are facing space issue on following directories. it is safe to remove or move these directories to another directory or make a tar of it. i need space . PLease help[root@mprl509 lib]#pwd/var/lib/[root@mprl509 lib]# du -sh cloudera-*1.3G cloudera-host-monitor300K cloudera-scm-agent20M cloudera-scm-eventserver4.6M cloudera-scm-headlamp3.0G cloudera-scm-server272M cloudera-scm-server-db5.5G cloudera-service-monitor
Is it possible...does it make sense to place an AWS application load balancer in front of a NiFi cluster?
Hi Community Team, I have a Private Cloud Base Platform 7.1.7.SP1 with hive 3.1. I need to connect my SAP server (Windows server) to hive. I installed Cloudera ODBC driver 2.6.13 and when I configured the DSN, shows me an error "ERROR 1100 SSL certificate verification failed because the certificate is missing or incorrect. - I have the AUTO TLS/SSL installed in the cluster. Hive has enabled TLS/SSL.My Questions: - How I generate a certificate .jks to this server that is not part of the cluster CDP.- What other steps I do to resolve this. Thank for your support.
I have a NiFi flow that is exporting Ranger Policies from one Cloudera Cluster and another NiFi flow that is importing them into another NiFi cluster. The API call to export them is ${ranger.local.baseurl}/service/plugins/policies/exportJson? and the API call to import them is ${ranger.local.baseurl}/service/plugins/policies/importPoliciesFromFile?Everything works great but once in a blue moon when trying to import the Policies I get - {"statusCode":1,"msgDesc":"XXPolicy not found","messageList":[{"name":"DATA_NOT_FOUND","rbKey":"xa.error.data_not_found","message":"Data not found","objectId":295}]}The best solution I have right now is to restart Ranger Admin and after it works. Does anyone know why this happens or how it gets into this state?
Hello, I have a 3 node NiFi cluster. I would like to add new users to the cluster that would be administrators i.e., a new user with all of the capabilities of the initial admin identity user (in my case, the initial admin identify is user nifi). I am using PKI certs as the means of identifty/authenticating users and this is working well. So, my question is, can I create a handful of users that are administrators that have a policy set equivalent to the initial admin identity? Is this possible?
hello guys, I need to copy hbase data from replication set cluster to ec cluster, i have tried hbase snapshot export and hadoop distcp ,they look like throw the same error: Not replicated yet, i think this was caused by when they transporting ,they will check if destination cluster replication num was same as local, but ignore if destination cluster used replication set. how can i do it ,thx. There is all the errors.snapshot export:2023-04-19 19:44:10,199 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: BLOCK* blk_-9223372036854751104_9943 is COMMITTED but not COMPLETE(numNodes= 5 >= minimum = 3) in file /hbase/archive/data/default/test/09748ed90d0d58c0fe7ac4b3c08f3cd4/cf/e35fadd88d244766800728318ccea5082023-04-19 19:44:10,200 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 8020, call Call#31 Retry#0 org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 172.31.0.146:52210org.apache.hadoop.hdfs.server.namenode.NotRepl
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.