Find answers, ask questions, and share your expertise
Recently active
Hello, I am trying to run a mapreduce job and output it on a folder. Is there any way to change the default filename. My default filename is part-r-00000.snappy.parquet and I want to add a date to this. Is this possible. Is there a parameter to change this, thank you.
Hi I currently have to extract folder from other computer via NiFi SFTP. I been finding for solution and notice that the NiFi processor (GetFile, GetSFTP) only allow to extract files instead of folder. Is there any nifi processor that is able to extract the folder? Example: Computer 1directory: C:\User\localFolder Computer 2 Using GetSFTP processor (remote path: C:\User) to extract the folder Thanks
Hello, How do we fetch the number of columns each table has in each DB from HMS DB? I have an urgent need to do a comparison for the number of columns each table has in each DB between 2 clusters. Doing that manually is practically impossible. Kindly help Thankssnm1523
Hi, How do we change the InputFormat and OutputFormat of an existing Hive table? I tried below, but got an error:alter table <table name> SET InputFormat "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"; Error:Cannot recognize input near 'SET' 'InputFormat' 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' Kindly suggest
Hello,Cluster has been Kerberized (LDAP / AD / Kerberos) and I have errors when I try to start the cluster. Zookeeper service start with following error: 2022-07-01 13:24:14,341 ERROR org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer: Failed to authenticate using SASLjavax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decryptAP-REQ - RC4 with HMAC)] at jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:199) at org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer.authenticate(SaslQuorumAuthServer.java:99) at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:563) at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:487) at org.apache.zookeeper.server.quorum.
My manager forces me to find a way to install and use Spark 3 on CDH 6.x cluster. Is there any change?When I did some research, I found out that only CDP 7. supports Spark 3, and CDH 6.x only support Spark 2. But my manager said that you don't need to install Spark through Cloudera Manager, you can install Spark 3 separately (by downloading a tar from the internet or sth like that) and then find a way to make that Spark service connect with Cloudera service like Hive, HDFS,... (by copying the hive-site, hdfs-site,... to spark conf folder maybe?) So does anyone have any experience with this? My manager is insane!!!!
Very new to NiFi and I am trying to create a daily email with throughput totals from the previous day. I was getting nowhere figuring out how to capture and report on NiFi processor throughput so I figured I could start with a GetHDFSFileInfo process that could look at the various HDFS tables and report on the previous day's directory size. Similar to "hdfs dfs -du -h /warehouse/tablespace/managed/hive/table1" and then grabbing the partition directory size from the previous day. I could script it easy enough but I would like to keep everything in NiFi so I don't have to worry about scripts and cron jobs. When I try to use GetHDFSFileInfo and do not recurse directories, I can get a list of all the partition directories but the length value is always 0. If I enable recurse, then I get every file (and they have the length value), Just curious if there was a way to have GetHDFSFileInfo provide partition level directory disk usage.&n
Hi All, I am running Nifi on a single Development server. (NAME="Debian GNU/Linux" VERSION_ID="10")I start the service using command line - bin/nifi.sh startDuring the initial phases as i experiment with RAM, CPU and disk i need to restart machine sometimes.When machine restarted I use above command to start Nifi and I go to UI and start the jobs. Is there a way that1. Nifi process starts automatically when ever system reboots or restarts ? 2. Nifi jobs also start automatically without me running from UI ? can I use startup scripts for this ? I want to get ready with production mindset as i soon will be launching.I want to understand how it is handled in realtime production servers..Please suggest.
Hello, Is there a way get the hdfs locations of all tables in CDP 7.1.7 SP1? I tried running a for loop against a list of tables on below command, however, it will take ages to get the list as I am looking forward to 66k+ tables. hive -e “desc formatted <table>;” | grep Location Thought of to get from HMS DB directly, however, I can only see DB location URI, but nothing specific to tables. Kindly help as this is urgent. Thankssnm1523
Hi Team, We are currently facing an issue in NIFI with OIDC as an authentication mechanism. Below is what we are seeing when NIFI URL redirects to OIDC(keycloak) url. OIDC works perfectly well with 1.20.0 version but when upgraded to 1.21.0 we are getting below error. ERROR: Unauthorized error="invalid_scope", error_description="Invalid scopes: openid address email web-origins profile user phone microprofile-jwt roles groups offline_access" Please suggest if there is any significant change made from 1.20.0 to 1.21.0 nifi oidc configuration which might have caused the issue. Appreciate your help in advance.
Hi Team, I am new to Nifi servers.I am currently working on a Development project where i developed a job to fetch changes from Mysql databases and apply them in a warehouse. I have 70 similar but different databases and therefore 70 Nifi jobs. The final aggregated database is around 100 GB. and daily some 1GB changes from all databases. I am currently working on development environment and using a single server.Now when i move to production with 70 Nifi jobs running all the time, do I need to think of a Nifi cluster ? or it is fine with single server ? What factors decide cluster vs stabdalone ? Also How can I handle upgrades to Nifi in future, Is it better for me to use Docker ? or having a normal instance is better ? Please help me with your valuable knowledge..
Is it possible to select with the QueryRecord processor also child fields of an given JSON structrue? E.g. following JSON: { "name" : "Max", "id" : 1, "location" : { "city" : "Frankfurt" }} And I would do the following query: SELECT name, location.city FROM FLOWFILE But this throws an error: QueryRecord[id=a4744340-017a-1000-6944-73fe4d362d90] Unable to query StandardFlowFileRecord[uuid=706e8ae8-f76c-4ebb-9c65-c46cb7026c89,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1626251405764-2, container=default, section=2], offset=790026, length=67],offset=0,name=55b72cfd-5308-4e83-9304-166193e76478,size=67] due to java.sql.SQLException: Error while preparing statement [SELECT "name", "location.city" FROM FLOWFILE]: org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: Error while preparing statement [SELECT "name", "location.city" FROM FLOWFILE] Is there something that I'm doing wrong? Here is the configurati
I have issue in importing the data from dataframe converted to csv then uploading it into hive but its not loading properly .My procedure:1st I took a Data frame from database and converted into a csv ,which has 343 columns and 24 lakhs rows 2nd I took the csv file to hive and I loaded the data to hive using load data code to table which i created directly by connect the hive to same database .this is what ,I am doing.In this case , my issue is for some rows it taking proper values but for some is null or 0.then i took a sample of 5 rows and I checked manually then i find out in csv file some rows there are some extra comma .so I manually removed and tried ,it worked but this cant be happening in real-time .so pls help me on this by giving some suggestion.
I am trying to roll my own search_after pagination query by saving the last sort values to a db and then retrieving them on the next time I run the query. Thing is if there are no sort values from the db this means this is a new query and I skip the search_after clause in my query. If I do retrieve a set of sort values from the db then I need to plug them into the query. I am having trouble with the EL statement. This is what I have: { "query": { "bool": { "must": [ {"term": {"key1": "value1"}} ], "must_not": [ {"exists": {"key2": "value2"}} ], "filter": [ {"range": {"@timestamp": {"lte": "now-1w"}}} ] } } ${search_after:isEmpty():not():ifElse(', \"search_after\": "${search_after}"', '')}, "sort": [ {"@timestamp": "asc"}, {"value3": "asc"} ]} The ${search_after} variable defined as '[ts int milliseconds, "a key value string"]' Including the sq brackets. The error I am receiving is:Unknown key for a VAL
Hi, I am running CDH 7.1.7p0. I have a external ORC database on S3 bucket. I did not configure this fs.s3a.server-side-encryption-algorithm in hadoop core-site.xml. However, I will randomly get this error when running hive query. Rerun the same query may success or fail with same error message. ERROR : Vertex failed, vertexName=Map 14, vertexId=vertex_1684599244994_0012_7_02, diagnostics=[Vertex vertex_1684599244994_0012_7_02 [Map 14] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: ss1 initializer failed, vertex=vertex_1684599244994_0012_7_02 [Map 14], java.lang.RuntimeException: ORC split generation failed with exception: java.lang.RuntimeException: java.io.IOException: Cannot find password option fs.s3a.bucket.sg-cdp.fs.s3a.server-side-encryption-algorithmThe database and query was part of the hive benchmark tool from github:https://github.com/hortonworks/hive-testbenchCan you help me to resolve this problem. Thank you.&
Dear community,I am writing to you because I have the following problem: How can I connect to impala tables from Cloudera AI Workbench, if I am setting up a session where R 3.x or R 4.x is used. I managed to make the connection using python, but I need to do it using R as well. I have the connection host, the port, the authentication mechanism. For example, the following python code manages to connect: "from impala.dbapi import connect conn = connect(host='host_name', port=21050, auth_mechanism='GSSAPI', use_ssl=True, kerberos_service_name='impala') cursor = conn.cursor()" But I need to achieve the connection now using R in Cloudera Data Science Workbench. Does anyone know how to translate the above code to R 3.x or R 4.x? I already tried using the 'RImpala' library, but it was removed from the CRAN repository according to https://cran.r-project.org/web/packages/RImpala/index.html. If anyone has an idea how to help me I would be very grateful.
Set "OAuth2 Access Token provider" in Invokehttp prosessor with grant type as client credentials as well as client id and client secret, thinking no need to Set Authorization Bearer token in this processor, the processor will dynamically set retrieved token with post request to get remote API response. But always to 401 error back. Not able to tell it is error from token provider or from remove API to send post request. However, if using Postman, in a post request, I can do it successfully to get remote API response. So I am sure the Grant type, client id and clietn secret are correct. What I am missing to get it work? WIl be very appreciate if NiFi experts guide me to configure invokeHttp processor for oauth2 seeteing to resolve my problem?
Is there a way to convert use expression language to convert the current time to a specific timezone. Here is what I have currently but that's GMT not GMT+10:00. ${now():format("yyyy-MM-dd'T'HH:mm:ss.SSSXXX", "GMT")}
I am storing several JSON Schemas as Parameter Contexts, and I need to access the schemas dynamically to validate flowfiles. But which schema depends on the flowfile, so I want to access a particular schema in the Parameter Contexts dynamically using an attribute value. How do I do that? Here is what I've tried:#{${schema.name}}${schema.name:evaluateELString()} The second technique I am using creates the exact string that I need #{Blue_JSON_Schema}, and then I try to perform an eval on it. But unfortunately it doesn't work either. Hopefully someone knows how to do this. Thanks for taking the time to read this.
I'm attempting to transition from using PutBigQueryBatch to PutBigQuery. However, the latter doesn't seem to evaluate the dataset and table attributes as expected.In the provided example, one processor has the values hardcoded, which works correctly, while the other processor relies on retrieving those values from the flowfile's attributes. However, it appears that the attributes are not being properly evaluated. The example flow (using Nifi 1.21.0):The flowFile used: The left PutBigQuery processor has hardcoded values, and it works:The right PutBigQuery processor fails because it seems it's using the EL definition but it's not evaluating it: Any help would be appreciated.
I have a multinode NiFi Cluster running in Docker Containers and I configured the DistributeMapCacheClient and DistributedMapCacheServer to point to a particular container's hostname and set the persistence directory property such that each node in the cluster uses the same Map Cache. I want to make that the MapCache persists even after I restart the container or create a new container. How would I configure that to happen?
Hi, I am trying ro read from a json file and create flowfiles with the kews as attributes and one flowfile per value in the list. The json file looks something like this:{"key1": null,"key2": null,"list": ["foo", "bar"],"key3": 1} This is my execute script: import jsonimport java.iofrom org.apache.commons.io import IOUtilsfrom java.nio.charset import StandardCharsetsfrom org.apache.nifi.processor.io import StreamCallbackfrom org.apache.nifi.processor.io import InputStreamCallbackflowFile = session.get() if (flowFile != None): stream_content = session.read(flowFile) text_content = IOUtils.toString(stream_content, StandardCharsets.UTF_8) json_data = json.loads(text_content) for i in json_data["list"]: flowFile2 = session.create() flowFile2 = session.putAttribute(flowFile2, "key3", str(json_data["key3"])) flowFile2 = session.putAttribute(flowFile2, "key1", json_data["key1"]) flowFile2 = session.putAttribute(flow
Hi,Please let me know how to connect the Google Drive in nifi and read the files.Which Processor to be used.Please help in this regards,Best Regards,Rana
Hi, we get the following Ranger error - maybe you can help me to fix it as soon as possible?! (We activated MIT Kerberos). Thanks in advance! How can I check that the password of the keystore file is correct? And where can I change it? Feb 17, 2020 4:29:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startINFO: Provided Kerberos Credential : Principal = rangeradmin/pdeluh0004392.hub.deluh.example.com@RDDL.PROD.EXAMPLE.COM and Keytab = /etc/security/keytabs/rangeradmin.service.keytabFeb 17, 2020 4:29:56 PM org.apache.ranger.server.tomcat.EmbeddedServer$1 runINFO: Starting Server using kerberos credentialFeb 17, 2020 4:29:57 PM org.apache.coyote.AbstractProtocol initINFO: Initializing ProtocolHandler ["http-bio-6182"]Feb 17, 2020 4:29:57 PM org.apache.coyote.AbstractProtocol initSEVERE: Failed to initialize end point associated with ProtocolHandler ["http-bio-6182"]java.io.IOException: Keystore was tampered with, or password was incorrectat sun.security.provider.JavaKeyStore.engineL
Hello NiFi Community, I have an AvroSchemaRegistry controller service and it references a bunch of schemas stored in a parameter contexts variables. Both the AvroSchemaRegistry and Parameter Contexts are at the same scope level. However, I have a very large number of schemas so for organizational purposes I'd like to organize them into separate Parameter Contexts: red_schemas, green_schemas, etc...But I'm still using just the one AvroSchemaRegistry controller service. Is it possible to create child Parameter Contexts (red, green, etc...) that inherit from the main Parameter Context and then reference the variables from a single controller service, which is at a higher scope? If so, what is the syntax for referencing variables in a different scope than the controller service? Thanks for reading.
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.