Cloudera Community
Althotta
user rank
Rising Star

Create Hive Tables in NIFI using Schema from schem...

by Rising Star in Support Questions
‎12-05-2022 01:54 AM
‎12-05-2022 01:54 AM
Is there a way in NIFI to create Hive tables from using the schema from CDP schema registry. ... View more
Labels:

NIFI Infer Schema trying to convert string(alpha) ...

by Rising Star in Support Questions
‎09-22-2022 09:03 PM
‎09-22-2022 09:03 PM
While converting the input flow file JSON to any other format using query record (CSVwriter or AVRO Writer) using inferschema strategy the NIFI processor is trying to convert to Date based on first few characters of the incoming string. NIFI Error Output Successfully parsed a JSON object from input but failed to convert into a Record object with the given schema\n- Caused by: org.apache.nifi.serialization.MalformedRecordException: Successfully parsed a JSON object from input but failed to convert into a Record object with the given schema\n- Caused by: org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Failed Conversion of Field [sharepoint_documents_name__c] from String [10-7-20 2x10GE LAN PHY XXXXXXXXXXXX] to LocalDate\n- Caused by: java.time.format.DateTimeParseException: Text '10-7-20 2x10GE LAN PHY XXXXXXXXXXX' could not be parsed at index 0:     Any workaround/fixes available in the newer version. Iam using NIFI 16.1 ... View more
Labels:

Re: Restrict Threads for Process group

by Rising Star in Support Questions
‎07-30-2022 05:18 PM
‎07-30-2022 05:18 PM
I have achieved this using Wait and Noitfy which is working as per my use case. ... View more

Restrict Threads for Process group

by Rising Star in Support Questions
‎07-25-2022 02:31 AM
‎07-25-2022 02:31 AM
Is there a simple way to restrict number of threads to a NIFI process group?.  ... View more
Labels:

Re: Convert Json array into csv using Nifi produce...

by Rising Star in Support Questions
‎07-25-2022 02:26 AM
‎07-25-2022 02:26 AM
Not yet resolved. @SAMSAL  solution is a potential workaround. Iam looking for a right solution why Convert produces this as i have few places to change.  ... View more

Re: Convert Json array into csv using Nifi produce...

by Rising Star in Support Questions
‎07-15-2022 05:43 AM
‎07-15-2022 05:43 AM
@SAMSAL   I need to retain the JSON format as-is in the output. I have updated the updated the expected output.   Expected Output: aaaa|{country=CHINA, city=null, street=null, latitude=null, postalCode=null, geocodeAccuracy=null, state=null, longitude=null} ... View more

Convert Json array into csv using Nifi produces Ma...

by Rising Star in Support Questions
‎07-15-2022 04:00 AM
1 Kudo
‎07-15-2022 04:00 AM
1 Kudo
JsonTreeReader to CSV Writer produces MapRecord String for the inner Json.    Data "id" : "aaaa", "billingaddress" : { "city" : null, "country" : "CHINA", "geocodeAccuracy" : null, "latitude" : null, "longitude" : null, "postalCode" : null, "state" : null, "street" : null }   Produces CSV out with MapRecord String attached.  aaaa|MapRecord[{country=CHINA, city=null, street=null, latitude=null, postalCode=null, geocodeAccuracy=null, state=null, longitude=null}]   Expected Output: aaaa|{country=CHINA, city=null, street=null, latitude=null, postalCode=null, geocodeAccuracy=null, state=null, longitude=null}   I want to keep the inner JSON as-is in the CSV output. Any workarounds to this issue? ... View more
Labels:

NIFI : PutHive3Streaming Transaction state is not ...

by Rising Star in Support Questions
‎06-21-2022 03:24 AM
1 Kudo
‎06-21-2022 03:24 AM
1 Kudo
NIFI version 1.16.1 nifi-hive3-nar-1.16.2   While loading data to hive tables using PutHive3Streaming, some of the tables loads are getting failed with errors. I tried to change commit size, it didn't help.  out of 50 hive table loads only 5 gets failed with this error and this is repeatable. Record count for these tables are less than 1Million.   2022-06-21 09:53:23,114 ERROR [Timer-Driven Process Thread-21] o.a.n.processors.hive.PutHive3Streaming PutHive3Streaming[id=9c9e3916-2000-1698-a0da-2dc44149819f] Aborted transaction cannot be committed: Transaction txnid:10346416 already aborted org.apache.nifi.processors.hive.PutHive3Streaming$ShouldRetryException: Aborted transaction cannot be committed: Transaction txnid:10346416 already aborted at org.apache.nifi.processors.hive.PutHive3Streaming.lambda$onTrigger$0(PutHive3Streaming.java:512) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1662) at org.apache.nifi.processors.hive.PutHive3Streaming.onTrigger(PutHive3Streaming.java:412) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1283) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hive.streaming.TransactionError: Aborted transaction cannot be committed: Transaction txnid:10346416 already aborted at org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.commitImpl(HiveStreamingConnection.java:877) at org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.commit(HiveStreamingConnection.java:841) at org.apache.hive.streaming.HiveStreamingConnection.commitTransaction(HiveStreamingConnection.java:513) at org.apache.nifi.processors.hive.PutHive3Streaming.lambda$onTrigger$0(PutHive3Streaming.java:499) ... 16 common frames omitted Caused by: org.apache.hadoop.hive.metastore.api.TxnAbortedException: Transaction txnid:10346416 already aborted at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$commit_txn_result$commit_txn_resultStandardScheme.read(ThriftHiveMetastore.java) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$commit_txn_result$commit_txn_resultStandardScheme.read(ThriftHiveMetastore.java) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$commit_txn_result.read(ThriftHiveMetastore.java) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_commit_txn(ThriftHiveMetastore.java:5192) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.commit_txn(ThriftHiveMetastore.java:5179) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.commitTxn(HiveMetaStoreClient.java:2491) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208) at com.sun.proxy.$Proxy249.commitTxn(Unknown Source) at org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.commitImpl(HiveStreamingConnection.java:859) ... 19 common frames omitted 2022-06-21 09:53:23,114 ERROR [Timer-Driven Process Thread-21] o.a.n.processors.hive.PutHive3Streaming PutHive3Streaming[id=9c9e3916-2000-1698-a0da-2dc44149819f] Failed to abort Hive Streaming transaction { metaStoreUri: thrift://vsgcnredhad12.in.reach.com:9083,thrift://vsgcnredhad13.in.reach.com:9083, database: tigfin_nifi, table: t_ap_invoice_lines_all } due to exception org.apache.hive.streaming.StreamingException: Transaction state is not OPEN. Missing beginTransaction? at org.apache.hive.streaming.HiveStreamingConnection.checkState(HiveStreamingConnection.java:500) at org.apache.hive.streaming.HiveStreamingConnection.abortTransaction(HiveStreamingConnection.java:519) at org.apache.nifi.processors.hive.PutHive3Streaming.abortConnection(PutHive3Streaming.java:652) at org.apache.nifi.processors.hive.PutHive3Streaming.lambda$onTrigger$0(PutHive3Streaming.java:559) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1662) at org.apache.nifi.processors.hive.PutHive3Streaming.onTrigger(PutHive3Streaming.java:412) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1283) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ... View more
Labels:

Re: NIFI ExecuteSQL AVRO logical type converts all...

by Rising Star in Support Questions
‎06-15-2022 07:01 AM
‎06-15-2022 07:01 AM
I had a look into it but that's not the right solution to my opinion. ExecuteSQL just converts every date into UTC, lets say if the dates are already in UTC then it once again converts to UTC which is not right.  ... View more

Re: PutHive3Streaming java.lang.NoClassDefFoundErr...

by Rising Star in Support Questions
‎06-15-2022 01:54 AM
‎06-15-2022 01:54 AM
The version 1.16.2 works. I don't see an option to accept the soln.  ... View more
Contact Me
Online
Offline
Last Visited
‎11-06-2024 02:09 AM
Community Statistics
Member Since ‎02-11-2021 02:26 AM
Last Visited ‎11-06-2024 02:09 AM
Posts 31
Kudos received 5