Q&A

I need your help in figuring out why this Hive Server 2 are not working in ambari, Please find the log information Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/ODP/1.0/services/HIVE/package/scripts/hive_server.py", line 143, in <module> HiveServer().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/ODP/1.0/services/HIVE/package/scripts/hive_server.py", line 53, in start hive_service('hiveserver2', action = 'start', upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/stacks/ODP/1.0/services/HIVE/package/scripts/hive_service.py", line 64, in hive_service check_fs_root(params.hive_server_conf_dir, params.execute_path) File "/var/lib/ambari-agent/cache/stacks/ODP/1.0/services/HIVE/package/scripts/hive_service.py", line 173, in check_fs_root environment = {'PATH': execution_path} File "/usr/lib/ambari-

Hi everyone, our Hadoop cluster has recently encountered a strange issue. After starting the cluster, Spark jobs run normally, but after running for more than a week, job submission timeouts occur. Specifically, the jobs are all in the ACCEPTED state, but fail due to timeout after 2 minutes. The error log shows:Caused by: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/134.1.1.3:45866 remote=hdp144019.bigdata.com/134.1.1.19:45454]I found an app id, and then went to check the corresponding NodeManager log for attempting to start the AM. I found that the AM was never started at all. At the time when the job submission timed out, the NodeManager log reported the following log:2024-03-15 00:04:17,668 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:handle(1607)) - couldn't find application application_1709910180593_38018 while processing FINISH_APPS event. The R

I am trying to copy table rows from one oracle database to another oracle table. PutDatabaseRecord cannot find the second table. I checked many times that table  exists. But it is still giving error. Does anybody know how to solve it?  PutDatabaseRecord[id=cb461cc1-018d-1000-21fe-ec8db1afeca1] Failed to put Records to database for FlowFile[filename=040d505a-1d6c-4f36-8a2b-38bcf577a059]. Routing to failure.: org.apache.nifi.processors.standard.db.TableNotFoundException: Table fsc_bank_dim_zz not found, ensure the Catalog, Schema, and/or Table Names match those in the database exactly 

I am trying to Deploy  Unravel Product Parcels  over CDP 719 ( RHEL 8.7) , I am getting below exceptions while distributing  the parcels. Any pointers to resolve the issue ?Error when distributing to cdpvm1.unraveldata.com : Ignoring non-compliant parcel manifest: UNRAVEL_SENSOR-1.0.4791002_cdh7.1_h3.1_s2.4.I am attaching the screenshot of the error 

Hi all,  I have been trying to store my Nifi workflows (via Nifi Registry) onto Azure Devops. I need little help in identifying the appropriate "flowPersistenceProvider" for Azure Devops that needs to be modified in providers.xml file.  Ps: so far, I have connected Nifi Registry to Github and its a success. I need same functionality for Azure Devops now.  Looking forward to any guidelines. 

We have recently migrated NIFI to AKS and it was working perfectly. Recently there was AKS version upgrade to 1.27.7 and since then UAT cluster has issues. The error while accessing Nifi UI.javax.net.ssl.SSLException: Connection reset When I get the logs of 3 pods, I am getting errorERROR [Timer-Driven Process Thread-3] o.a.n.c.s.StandardControllerServiceNode Failed to invoke @OnEnabled method of DBCPConnectionPool[id=12d922eb-87dc-3b72-9550-5e49f6849617] due to org.apache.nifi.reporting.InitializationException: Can't load Database DriverNeed suggestions to fix this error

When try to use ExecuteSQL 1.25.0  to query  data from oracle 11g  , we received the below errors  can you guide me please , we have Apache NiFi in docker .Caused by: java.sql.SQLException: ORA-00604: error occurred at recursive SQL level 1ORA-01882: timezone region not found  

The minimum hardware & Software requirements for the trial version of Cloudera Private Cloud Base in a non-production environment

We have a requirement to merge files in a specific HDFS path. The merged file should also be created in the same HDFS path but with a different name. Could you please suggest how we can achieve this using the Nifi Processor?

Hi everyone,I need to create HUE table separating "|" for insert like following values into table with removing all not required """94112605011"|"DEFAULT"|"FTTH_WEB FAMILY PLUS!PDL&Ec5@1706725800;94112605011;1704047400"|"COUNTER"|"u"|"0E-12"|"2023-12-31 18:30:00.000000"|"2024-01-31 18:30:00.000000"|"0"|"{""data"":{""o"":{""-t"":""ECS_BalanceContextInfo"",""m"":[{""-c"":""57a2"",""-v"":""1"",""a"":[{""-n"":""unitType"",""v"":{""-s"":""VOLUME""}},{""-n"":""offerId"",""v"":{""-s"":""15""}},{""-n"":""offerName"",""v"":{""-s"":""FTTH_WEB FAMILY PLUS""}},{""-n"":""offerQuota"",""v"":{""-s"":""42949672960""}},{""-n"":""quotaThreshold"",""v"":{""-s"":""0""}}]}],""o"":[{""-t"":""ECS_BalanceContextInfoBase"",""m"":[{""-c"":""5136"",""-v"":""1""},{""-c"":""8d18"",""-v"":""2"",""a"":[{""-n"":""subscriptionId"",""v"":{""-s"":""9411260501111152164097540010""}}]}]},{""-t"":""ECS""},{""-t"":""AVS"",""m"":[{""-c"":""e3a0"",""-v"":""1""}]}]}}}"|"{""a"":[""0""]}"|"BYTES"|"0E-12"i HAVE USED FOLLOWING

Hello guys,I currently work on a company that does provide hive 3.1 servers by using knox or zookeeper (kerberos) authentication methods. I am able to connect and query data by using a odbc connection on my personal computer. Now I need to set the connection on a virtual jupiter notebook server with pyodbc , so, I am not able to install the ODBC (and probably the server is based on Linux anyway).I´ve seen many options as pyhive, implya and others, but everything I find is regarding Hive 2 servers.So, what could I use to set the python connection to the Hive 3 servers?

Hello everyone, I am currently getting acquainted with Cloudera. I have successfully installed CDP Private Cloud Base on an Ubuntu virtual machine. However, when I want to install CDP Private Cloud Data Service to perform SparkJob creation and run it on the web interface as shown in the Video Demo on the Cloudera website under the Engineer section, I see a requirement to install it on OpenShift Container with a different configuration. I don't understand this, could you please guide me or share any step-by-step tutorials for installing both components? Thank you very much.

Hello,"I have a QueryDatabaseTable processor where I've written custom SQL code and max value property ti pick only latest data from source. Suppose I'm creating the Dim1 flow. I want to ensure that once Dim1 is finished, Dim2 should start. However, QueryDatabaseTable does not accept input. How can I enable this solution?  

I have a kerberized hadoop cluster. Installed knox in kubernetes container with its keytab. Configured knox topology with shiro provider and ldap authentication. I'm able to access webhdfs fine with ldap credentials but Hive is failing with HTTP 401 error.Knox is configured with self-signed cert and hive is not configured with SSL.topology.xml -- Configured with shiro provider and ldap info<service><role>WEBHDFS</role><url>http://name1_hostname:50070/webhdfs</url><url>http://name12_hostname:50070/webhdfs</url></service><service><role>HIVE</role><url>http://<host_name>:10004/cliservice</url><param><name>replayBufferSize</name><value>8</value></param></service>Configured hive-site.xml with below configs in HS2 hosthive.server2.thrift.http.port = 10004hive.server2.thrift.http.path = cliservicehive.server2.transport.mode = httphive.server2.allow.user.substitution =

Hello,I'm currently running Nifi version 1.8.0, starting from last week I noticed that I'm missing PutSysLog message from my syslog system.  Some logs would appear and some wouldn't write to the syslog system.  I checked my syslog system and was unable to find any issue.  No other systems are experiencing issue writing to the syslog system.  I enabled DEBUG on PutSysLog and it shows it was successful in sending the message to the syslog system but no entry exists. At the same time other PutSysLog does record the entries in syslog system and the file flows through other processors.      Any suggestions on how to troubleshoot this issue?Thanks!

Hello Exeprts, I am using hive-jdbc jar in my application to connect Hive and Delta (via Thrift server)  from Nifi flow using a custom processor - https://mvnrepository.com/artifact/org.apache.hive/hive-jdbc Now the problem is this has not released any major version in recent times and all the available older versions have velnurables in it.(Even I have tried '4.0.0-beta-1' , I don't see any velnurables with this, but looks like they have removed some internal dependencies, so it did not work for me) So to get rid of velnurbles I am checking to see if there is any alternate option for me to use. Thanks Mahendra

We have the below requirements in the Nifi flowa) Fetch a file from HDFS and transfer it to a Linux Server. I have achieved this using FetchHDFS and PutSFTP NiFi Processors, and it is working fine as expected for a single file.Now, we have a requirement that this HDFS directory may contain multiple files, and all of them need to be transferred to the SFTP server. I tried passing the HDFS directory path in the "HDFS File Name" Property of the FetchHDFS processor; however, it seems to accept only file names. Could you please suggest an alternative processor/method to achieve this?I tried to use ListHDFS -> FetchHDFS -> PutSFTP Actually the HDFS files are output of sqoop jobs. I tried connecting RouterAtribute to ListHDFS, but that connection is not allowed.b) The second requirement is to combine all the files in a specific HDFS directory. Can we achieve this using the MergeContent processor?Could you please provide some inputs?

Hello,My team is about to deploy a clustered scaling NiFi, along with NiFi Registry and Zookeeper. We are deploying on Openshift Kubernetes.We are determining which directories to mount our persistent storage to on both NiFi and the Registry.1. Since we are using Registry, do we need to back anything up on the NiFi pod?2. What are all of the locations we would want mount persistent storage for both NiFi and Registry? 3. What are all of the locations we would want to establish a backup procedure for on NiFi and Registry if we are using local filesystem persistence?4. Do we still need to backup any locations if we use the Git and S3 persistence providers?5. What does the restoration process look like for bothA. local persistence providers onlyB. Git + S3 persistence6. Do we need to worry about Zookeeper at all for backups?Thanks ahead of time!

Installed CFM (Nifi) on Cloudera 6.2 using parcels. I see default 512 M being allocated for Heap size and want to increase it. I did not see nifi.properties file where I could do the same...    Any guidance would be appreciated.. 

I am getting the following error message when I try to refresh my datasource in Tableau. I have gotten this error when I try to refresh in an existing workbook and when I try to connect in a new workbook.Has anyone seen this error before?An error occurred while communicating with the Other Databases (JDBC) data source 'ShowData'Bad Connection: Tableau could not connect to the data source.Error Code: B19090E0[Cloudera][HiveJDBCDriver](500051) ERROR processing query/statement. Error Code: 10002,SQL state: TStatus(statusCode:ERROR_STATUS,infoMessages:[*org.apache.hive.service.cli.HiveSQLException:Error while compiling statement:FAILED: SemanticException [Error 10002]: Line 1:20Invalid column reference 'tableausql.fieldname':38:37, org.apache.hive.service.cli.operation.Operation:toSQLException:Operation.java:335, org.apache.hive.service.cli.operation.SQLOperation:prepare:SQLOperation.java:199, org.apache.hive.service.cli.operation.SQLOperation:runInternal:SQLOperation.java:260,org.apache.h

I'm new to NiFi. I have a flow that's consuming messages from Kafka and producing them to Redis. The issue I'm experiencing is the MergedContent queue is filling up within seconds, which is slowing down the process of getting the records into Redis. I'm wondering what the recommended approach is to improve things (being new to Redis, I'm aware that I'm most likely missing some obvious things). My flow looks like this: 

I want to implement Apache Nifi on my webapp and i need to customize a little bit the UI to make it more compliant with my app. I have some troubles in understanding where is the code of frontend in the Source2.0.0-M2 package you can download from the official website.Where can I find the code? Thank you in advance.I have tried to change the js code in multiple folders but nothing has changed.

Hello,I am in the process of transferring data from an Oracle database to HDFS, formatted as Parquet. The workflow, as depicted in the attached screenshot, employs the GeneratedTableFetch method for data ingestion in segments. Additionally, the ExecuteSQL processor runs the generated queries, UpdateAttribute is utilized to add an attribute, and QueryRecord is used to introduce a new column in the flow files.The source table contains approximately 22 million records. In this procedure, the 'Date' column is set as the 'Maximum-value Column' with the GenerateTableFetch processor, and a partition size of one million rows has been configured. This approach has allowed for the successful transfer and storage of all 22 million records into HDFS.However, during data quality checks, I encountered some issues: there were missing rows and instances of duplicate records in the HDFS storage, a situation not mirrored in the source database, which maintained an accurate and duplicate-free record coun

I am getting the following error message when running a refresh. Do I have an invalid date field causing the issue? Unable to complete actionNot all extracts were refreshedAn error occurred while refreshing outputtable[Cloudera][HiveJDBCDriver](500051) ERROR processing query/statement. Error Code: 40000, SQL state: TStatus(statusCode:ERROR_STATUS, infoMessages:[*org.apache.hive.service.cli.HiveSQLException:Error while compiling statement: FAILED: ParseException line 1:7 cannot recognize input near '{' 'fn' 'TIMESTAMPADD' in select clause:38:37, org.apache.hive.service.cli.operation.Operation:toSQLException:Operation.java:335, org.apache.hive.service.cli.operation.SQLOperation:prepare:SQLOperation.java:199, org.apache.hive.service.cli.operation.SQLOperation:runInternal:SQLOperation.java:260, org.apache.hive.service.cli.operation.Operation:run:Operation.java:247, org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementInternal:HiveSessionImpl.java:541, org.apache.hive.se

Hi there,Maybe some one can help us out. Searching on the web for similar error's doesn't give us a lead on what is going on.Let me sketch out what is happening. Last week we were cleaning up some old buckets with flows from the nifi-registry manually thru the web gui. At some point we received an error that pointed us the registry logs. Actually didn't see anything there. To test if everything is still working, we tried to commit a change to a versioned flow. Which failed. Unfortunately I don't have the error anymore. So decided to restart the registry hoping that this will help. And now it has gotten worse, the registry isn't starting anymore and is showing the error below.2024-03-26 09:40:28,551 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getFlowPersistenceProvider' defined in class path resource [org/apache/nifi/registry/provider/StandardProviderFactory.class]: Bean in