Find answers, ask questions, and share your expertise
Recently active
Hi, I want to ask a few questions about having multiple Hive Mestastore instances and how to correctly configure the compaction initiator and cleaner threads on these. The point of having multiple Hive Metastore instances is because we have many concurrent connections and, as we have seen on the official documentation (https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/release-guide/topics/cdpdc-hive.html), it is not recommended to have more than 16-24GB of Java Heap due to the impact of Java garbage collection on active processing by the service. 16-24GB of Java Heap is the recommended amount of memory to manage 41-80 concurrent connections (we have around 50 on each Metastore). So, what i want to ask is what is the correct configuration for the compactor initiator and cleaner threads (hive.compactor.initiator.on) if we have multiple Hive Metastore instances. Can it be enabled on every instance? Should it be only enabled on 1 instance? Our environment is ru
Hi folks,I have a problem on my CDP Datalake causing failure on master node.Apparently some services fail due to lack of free memory space. Everything seems to work correctly but I don't know what can happen if the free space reduces further.I have tried these solutions, however, with no effect if not momentary:- lowering some alarm thresholds (from 10% to 7% of free space)- removing some log files from the machine in /var/log and /tmp folders @shehbazk Can you give me some advice?And I would also like to understand if it is possible and how to increase the capacity higher than the current 236 GB. @pandav
Hello all, I am a student of Purdue Global University. As part of several classes, we have been instructed to utilize Oracle VM VirtualBox Manager, alongside an installation of Cloudera-Quickstart-VM-5.13.0-0-Virtualbox. Last semester this installation worked throughout my class just fine. Then, this semester a class requires it as well, but when I went to boot the original installation I was surprised to find it won't boot. I am running into what I think is a Kernel Panic error on boot. I have tried removing the Cloudera instance and reinstalling it using the PDF guide provided to us by the school. However, on reinstall, the Kernel boot error remained. I have also tried shutting down and restarting my computer, as well as changing the chipset setting from PIIX3 to ICH9, as suggested in another thread. I have a video showing a screen capture of what I receive on boot, as well as the logs, and the PDF we use for installation. <L
Hi All, I need to download my old certofocation from cloudera that was completed in 2020 but I cant find any resource on how to. Can someone please help with this
Hi, I am trying to secure NiFi using LDAP configuration. I followed blog by mintops and pvillard articles for reference. I am running NiFi on windows (not in cluster configuration). I am able to get to the login screen, but then I am getting an error on logging in ( The supplied username and password are invalid) I followed most instructions as mentioned in the blog by mintops. Can someone help me in the direction, on what might be the problem?PS: NiFi version and toolkit versions -1.8.0.I haven't created certificates, just the configs needed to update in Keystore and truststore passwords by the toolkit.The LDAP configurations were already being used in another program, so that is also not the issue. login-identity-provider.xmlauthorizers.xml: nifi.properties &nb
I am working on a proof of concept situation with a Nifi Cluster on Openshift. As part of the PoC I want to enable authentication (currently have a working cluster with no auth), so I am wondering if it possible to do it with the Single User method as it is built in and should be the easiest to setup. I have been unable to find any documentation outlining this process or possibility, so if there is any outlining this setup, assuming it is possible, please direct me towards it.
I have a Handle HTTP Request Processor to receive incoming requests , which have various workflows connected to it, I want to apply authentication on this processor for incoming requests, I have figured out that I need to set "Need Authentication" and configure a secure socket layer Context service, But I have no idea how secure socket layer works,what are key stores and trust stores ? how to generate them and how to use them, Do the client needs to send certificates ? how do I verify it ? I intend to test it by sending request to this processor using Postman.Any help is appreciated, Thank You.
"message":"Could not login: the Client is being asked for a password, but the ZooKeeper Client code does not currently support obtaining a password from the user. Make sure that the Client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the Client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookeeper Client using the command 'kinit <princ>' (where <princ> is the name of the Client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this Client. If you continue to see this message after manually ref
Hi everyone.We have an NiFi cluster consisting of 3 nodes. After the failure of the disk subsystem on one of the nodes, it was in the ReadOnly state for a long time. After resolving the issue and after restarting cluster we are getting the following error on problematic node:2022-09-16 15:31:07,255 ERROR [Load-Balanced Client Thread-4] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Failed to communicate with Peer xxxxxx:9443java.io.EOFException: Expected StandardFlowFileRecord[uuid=6ce9e262-b20b-4372-a3b9-43c2c00e8caa,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1663256223724-231072817, container=default, section=49], offset=387990, length=2203],offset=0,name=04190e1f-fdca-4352-a796-6b6c9ce41baa,size=2203] to contain 2203 bytes but the content repository only had 1130 bytes for it at org.apache.nifi.controller.queue.clustered.ContentRepositoryFlowFileAccess$1.ensureNotTruncated(ContentRepositoryFlowFileAccess.java:83) at org.apache.nifi.controller.queue.clustered.Con
I have about 25-30 XML message types and each message type has its own XSD. I need to validate each message against their respective XSD. When using the ValidateXML processor is there any way to dynamically assign the appropriate XSD to a flow file based on on attribute value? I don't see the purpose/benefit of using so-called variables when said variables aren't even variable--they are STATIC! Why does this processor ONLY use variable_registry variables and not attribute values like every other processor in NiFi?
Hi all,I want to add 15 minutes to a date-time value using the transform json processor, but with my configuration I get the same value as before. I expect for the end field to have 15 minutes more than the start field. Does anyone have any idea? [ { "store_id" : 1001, "date" : 20220918, "start" : "2022-09-18 00:15:00", "end" : "2022-09-18 00:15:00", "out" : 0}, { "store_id" : 1001, "date" : 20220918, "start" : "2022-09-18 00:30:00", "end" : "2022-09-18 00:30:00", "out" : 0}] My jolt Specification [ { "operation": "modify-overwrite-beta", "spec": { "*":{ "end": "=${end:toDate('yyyy-MM-dd HH:mm:ss'):toNumber():plus(900000):format('yyyy-MM-dd HH:mm:ss')}" } } }] The result is the same [ { "store_id" : 1001, "date" : 20220918, "start" : "2022-09-18 00:15:00", "end" : "2022-09-18 00:15:00", "out" : 0}, { "store_id" : 1001, "date" : 20220918, "start" : "2022-09-18 00:30:00", "end" : "2022-09-18 00:30
Hi. Could somebody help me please with composite group provider settings in authorizers.I'm not sure what should I put in "Initial admin identity" and "Initial user identity 1".1) When I write just username (like nifi-admin) I cannot login to NiFi since I don't undertand where I can get a password.2) When I write a full LDAP name (like uid=freeipa,cn=users,cn=accounts,dc=foo,dc=bar) I successfully log in but I cannot access a NiFi interface.3) When I use name of user who already exists in LDAP, I get an error, that user presents in both file and LDAP provider settings. This case is configured below in my authorizers.xml file:<authorizers><userGroupProvider><identifier>file-user-group-provider</identifier><class>org.apache.nifi.authorization.FileUserGroupProvider</class><property name="Users File">./conf/users.xml</property><property name="Legacy Authorized Users File"></property><property name="Initial User Identity 1"
Hi all.We have a cloud HDFS cluster.We have DataNode with different size '/data' (from 1TB to 3TB). Node #1, which has 3TB, is 500Mb full, node #2, which has 1TB, is 900Mb full. Can you please tell me why this is happening and how to fix it?The value of rebalancing threshold = 5.
Hey Everyone,i would like insert a string into another string with the query Record Processor with the sql function STUFF but i run into an error. Anyone have an Idea how i can adjust the sql statemant or how i can this string manipulation with the nifi expression language I want to insert a colon in a column with the time of dayfrom thatto the001500:15 This is my input Data from an json file [ { "store_id" : 1011, "date" : 20220915, "start_time" : "0015", "start_end" : "0015", "out" : 0}, { "store_id" : 1011, "date" : 20220915, "start_time" : "0030", "start_end" : "0030", "out" : 0}] This is my Sql expression SELECT "store_id", "date", STUFF("start_time",2,0,':') as "start", "start_end", "out"FROM FLOWFILE This is the error message i got QueryRecord[id=46658fcd-0183-1000-0000-000015e8db65] Unable to query StandardFlowFileRecord[uuid=641e1c71-ff9b-45dc-a948-e527eff450f6,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id
Hi Guys ,i have a problem with Nifi login please can´t help me ?this problem start when open the web interface of Nifi later config ldap login .Present the next message has been attached imagen . for fix the problem -> select log out is there any other solution ?Regards!
Hey everyone, i get a csv file as record from an api call and don't need all columns and want to convert the datetime columns to a time format. For example from 15.09.2022 00:15:00 to 00:15 This is my csv recordInterval startInterval stopCamera serial numberCounter namePedestrians coming inPedestrians going out15.09.2022 00:00:0015.09.2022 00:15:00000000111123400 This is my setting in the Query record ProcessorQuery SELECT #{TodayAxisV2Param} as "date", "Interval start" as "start time", -- Expected FORMAT(HH:mm) "Interval stop" as "end time", -- Expected FORMAT(HH:mm) "Pedestrians going out" as "out"FROM FLOWFILE I expect this result for the time fields date start timeend timeout2022091500:0000:150 Does anyone have an idea how I can do this with the Query Record Processor?
hello cloudera community, we are trying to create a keytab with the main one: "HTTP/hostname@DOMAIN.LOCAL" with the command: ktpass -princ HTTP/hostname@DOMAIN.LOCAL -mapuser livy-http -crypto ALL -ptype KRB5_NT_PRINCIPAL -pass password2022 -target domain.local -out c:\temp\livy-http.keytab but I try to validate the ticket with this keytab returns the error: Exception: krb_error 24 Pre-authentication information was invalid (24) Pre-authentication information was invalid KrbException: Pre-authentication information was invalid (24)at sun.security.krb5.KrbAsRep.<init>(Unknown Source)at sun.security.krb5.KrbAsReqBuilder.send(Unknown Source)at sun.security.krb5.KrbAsReqBuilder.action(Unknown Source)at sun.security.krb5.internal.tools.Kinit.<init>(Unknown Source)at sun.security.krb5.internal.tools.Kinit.main(Unknown Source)Caused by: KrbException: Identifier doesn't match expected value (906)at sun.security.krb5.internal.KDCRep.init(Unknown So
Hello everyone. I am trying to train a ML model on a cluster by using a docker image and spark-submit with yarn.I already tried to follow this process before on a training cluster I made and I succeeded. But when I run it this time, yarn prompts that one of the mounts is invalid.Of course I tried with Kerborise and without, both didn't work and both runs didn't imply any problems related to connection so we are good from this side. This is what I tried:spark-submit \--master yarn \--deploy-mode cluster \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=image:v1 \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro,/opt/cloudera/parcels/:/opt/cloudera/parcels/:ro,/data01/yarn/nm/:/data01/yarn/nm/:ro,/data02/yarn/nm/:/data02/yarn/nm/:ro,/data03/yarn/nm/:/data03/yarn/nm/:ro,/etc/krb5.conf:/etc/krb5.conf:ro" \--conf s
hello to all,I have many connections in time_wait on the ipc port 1019 of the datanode:More than 600 time_wait and about 250 established.Is that normal?I’m afraid that’s why index write closed on solr errors (the index is on hdfs).The servers are downloaded and the datanode does not saturate the jvm heap.I couldn’t find any max connection configuration for port 1019Any ideas? Environment:HDP 3.1.5.0-152 with HDFS 3.1.1 Thanks in advance
Hi, I'd like to catch any key that match a certain reg ex in jolt. The purpose is to clean and order keys into the proper name. An example worth thoushand words: possible inputs:{"my_id":234535,"my company sa": "whatever value","my comp RH": "whatever value",...}{"my_id":234536,"my_comp_sa" : "whatever value","DRH" : "whatever value"...} output:{"my_id":234535,"my_company_sa": "whatever value","my_company_rh" : "whatever value"}
Hi community,hope someone can help me on below. My Flow File content is something like this: [ {"version" : "5","shipment_number" : "2022065597","situation_code" : "MLV","justification_code" : "CFM","situation_date" : "202206071253","new_appointment_date" : "202301081253",}, {"version" : "5","shipment_number" : "2022065597","situation_code" : "MLV","justification_code" : "CFM","situation_date" : "202206071252","new_appointment_date" : "",} ] I need to add a Query Record processor in order to grab only one JSON record for each Flow File.The condition is: Get a single result with highest value of situation_date or new_appointment_date. Just to explain better, the logic behind should be the following:Grab the max of situation_date, the max of new_appointment_date and compare each other, then take the highest of the comparison and get entire JSON record. In MYSQL or T-SQL it could be easier, but if i understand well Query Record is based on Apache Calci
Hi everyone,i had a .csv file converted to json with the convert csv to json processor and configured the csv reader. In my csv file i have names with german umlauts. I have use character set UTF 8 or ISO-8859-1 but the output is not the right for german umlauts. the csv file has the following character set from the source system This are my setting of the csv reader controller serviceDoes anyone have an idea for the correct setting.Thanks
Hello Nifi Community, We have integrated our Nifi 1.16.2 with LDAP AD server.We have created an Initial Local Admin (nifi_ldap) and used "composite-configurable-user-group-provider" as user group provider. We also restricted to one particular group of LDAP server (namely "EDH_ML"). But none of the users of this group ("EDH_ML") is able to access the Nifi and getting "Insufficient Permission Error". Could someone can help us to resolve this error? -- Sharing nifi screenshot and configuration settings/logs Nifi Users Nifi Login Error Nifi User Policies Authorizer.xml<userGroupProvider><identifier>file-user-group-provider</identifier><class>org.apache.nifi.authorization.FileUserGroupProvider</class><property name="Users File">./conf/users.xml</property><property name="Legacy Authorized Users File"></property><property name="Initial User Identity 1">cn=Service Account\, nifi_ldap,ou=Service Accounts,o
we are using ambari 2.6.2.2 with hdp 2.6.5 the permissions are being managed by the ranger, but checking the ranger settings, it's only managing the /apps/hive/warehouse directory on the hdfs the problem we are having is in the /area/data/pll directory on the hdfs, this directory has acl rwx permission for the user using beeline to create an external table is returning the error: Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [service_user] does not have [ALL] privilege on [hdfs://service1/area/data/pll] (state=42000,code=40000) but using hive cli does not return error, table is created successfully. What could be causing this problem?
Hello, I am running Flink 1.14.4 with 3 jobs running in it. JVM: openjdk version "1.8.0_312". All jobs are running without any issues (at least not obviously noticed) however, I see the below exception in the log. Im not able to understand the reason for the exception. Not sure what is the issue. Appreciate any help on this. Exception: java.nio.file.InvalidPathException: Nul character not allowed (full stack trace below) 2022-08-15 16:35:15.350 [flink-rest-server-netty-worker-thread-5] WARN org.apache.flink.runtime.rest.FileUploadHandler - File upload failed.java.nio.file.InvalidPathException: Nul character not allowed: bash' '-c' #cmdlinux})).(#p new java.lang.ProcessBuilder(#cmds)).(#p.redirectErrorStream(true)).(#process #p.start()).(#ros (@org.apache.struts2.ServletActionContext@getResponse().getOutputStream())).(@org.apache.commons.io.IOUtils@copy(#process.getInputStream() #ros)).(#ros.flush())}b at sun.nio.fs.UnixPath.checkNotNul(UnixPath.java
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.