Member since
04-05-2016
188
Posts
19
Kudos Received
11
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
909 | 10-30-2017 07:05 AM | |
1194 | 10-12-2017 07:03 AM | |
4849 | 10-12-2017 06:59 AM | |
7117 | 03-01-2017 09:56 AM | |
21282 | 01-26-2017 11:52 AM |
03-24-2017
08:44 PM
1 Kudo
I have been experiencing failures with my datanodes and the error is WRITE_BLOCK and READ_BLOCK. I have checked the data handlers and i have dfs.datanode.max.transfer.threads set to 16384. I run HDP 2.4.3 with 11 nodes. Please see error below; 2017-03-24 10:09:59,749 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dn:50010:DataXceiver error processing READ_BLOCK operation src: /ip_address:49591 dst: /ip_address:50010
2017-03-24 11:02:18,750 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dn:50010:DataXceiver error processing WRITE_BLOCK operation src: /ip_address:43052 dst: /ip_address:50010
... View more
Labels:
- Labels:
-
Apache Hadoop
03-20-2017
10:52 AM
Thank you @Matt Clarke.
... View more
03-17-2017
10:04 AM
I found a ListSFTP processor with error "Failed remote connection, input stream is closed". This was temporarily resolved by increasing the nproc to 10240 from 1024. Is this nproc count normal or should i be ready to increase the nproc to unlimited?
... View more
Labels:
- Labels:
-
Apache NiFi
03-02-2017
05:55 AM
There was a special character (') in one of columns that made the insert statements fail. A ReplaceText processor was used to replace the special character
with an empty string before processing the insert statements. Issue resolved. insert into TABLE(col1,col2,col3,col4,col5, col6)
values ('${csv.1}','${csv.2}', to_timestamp(' ${csv.3}', 'YYYY-MM-DD HH24:MI:SS,FF9'),'${csv.4}','${source}', to_timestamp(' ${current_timestamp}', 'YYYY-MM-DD HH24:MI:SS,FF9'))
... View more
03-01-2017
10:10 AM
I notice that my NiFi flow for loading into DB fails to load some records. The error message in nifi-app.log below. Forwarding the insert statements to retry does not help either... 2017-03-01 05:42:39,287 ERROR [Timer-Driven Process Thread-17] o.apache.nifi.processors.standard.PutSQL PutSQL[id=6f0f11db-015a-1000-ffff-ffffd3720836] Failed to update database for [StandardFlowFileRecord[uuid=d5fd46d2-3d33-4ed3-84f1-c5028106041a,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1488339757266-557, container=content1, section=557], offset=175256, length=302],offset=0,name=NPS_173_20170228.csv,size=302], StandardFlowFileRecord[uuid=93fe79f1-95ea-4d1e-b119-d678d75cb9a0,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1488339757266-557, container=content1, section=557], offset=175558, length=305],offset=0,name=NPS_173_20170228.csv,size=305], StandardFlowFileRecord[uuid=6af94c1e-379a-436b-a2cc-f068b3ed4751,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1488339757266-557, container=content1, section=557], offset=175863, length=304],offset=0,name=NPS_173_20170228.csv,size=304], StandardFlowFileRecord[uuid=5c792aac-6d7c-4099-b591-c0b49eb09105,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1488339757266-557, container=content1, section=557], offset=176167, length=301],offset=0,name=NPS_173_20170228.csv,size=301], StandardFlowFileRecord[uuid=51bf67f3-1b8f-4c87-ae8a-5085a95e4980,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1488339757266-557, container=content1, section=557], offset=176468, length=304],offset=0,name=NPS_173_20170228.csv,size=304], StandardFlowFileRecord[uuid=a06cb53b-bc96-49a8-a2a2-6e65e39b0ad0,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1488339757266-557, container=content1, section=557], offset=176772, length=305],offset=0,name=NPS_173_20170228.csv,size=305], StandardFlowFileRecord[uuid=ec9356fe-c0fb-4d06-bc61-6d2211de4367,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1488339757266-557, container=content1, section=557], offset=177077, length=301],offset=0,name=NPS_173_20170228.csv,size=301], StandardFlowFileRecord[uuid=9d3b1554-88f0-411b-bf2c-a51f8a90a06a,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1488339757266-557, container=content1, section=557], offset=177378, length=301],offset=0,name=NPS_173_20170228.csv,size=301], StandardFlowFileRecord[uuid=40137316-28a3-4262-a386-92226273605e,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1488339757266-557, container=content1, section=557], offset=177679, length=303],offset=0,name=NPS_173_20170228.csv,size=303]] due to java.sql.SQLException: Missing IN or OUT parameter at index:: 1; it is possible that retrying the operation will succeed, so routing to retry: java.sql.SQLException: Missing IN or OUT parameter at index:: 1
2017-03-01 05:42:39,296 ERROR [Timer-Driven Process Thread-14] o.apache.nifi.processors.standard.PutSQL
java.sql.SQLException: Missing IN or OUT parameter at index:: 1
at oracle.jdbc.driver.OraclePreparedStatement.processCompletedBindRow(OraclePreparedStatement.java:1878) ~[na:na]
at oracle.jdbc.driver.OraclePreparedStatement.addBatch(OraclePreparedStatement.java:10836) ~[na:na]
at oracle.jdbc.driver.OraclePreparedStatementWrapper.addBatch(OraclePreparedStatementWrapper.java:1369) ~[na:na]
at org.apache.commons.dbcp.DelegatingPreparedStatement.addBatch(DelegatingPreparedStatement.java:175) ~[na:na]
at org.apache.commons.dbcp.DelegatingPreparedStatement.addBatch(DelegatingPreparedStatement.java:175) ~[na:na]
at org.apache.nifi.processors.standard.PutSQL.onTrigger(PutSQL.java:276) ~[nifi-standard-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064) [nifi-framework-core-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_77]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
2017-03-01 05:42:39,296 ERROR [Timer-Driven Process Thread-17] o.apache.nifi.processors.standard.PutSQL
... View more
Labels:
- Labels:
-
Apache NiFi
03-01-2017
09:56 AM
1 Kudo
The issue was with a column that handled datetime in millseconds and needed the format as 'YYYY-MM-DD HH24:MI:SS,FF9' specified in NiFi and it worked.
... View more
02-23-2017
04:51 PM
@Timothy Spann I am able to access Oracle 12c and successfully run the insert statements generated in NiFi. SQL connection is fine, no errors. I have checked data provenance as well and nothing seems to be amiss but that it dropped the flowfile.
... View more
02-23-2017
01:50 PM
Hi @Pierre Villard, see the PutSQL error below; 2017-02-23 14:40:43,173 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 21071 nanos
2017-02-23 14:40:43,202 INFO [Timer-Driven Process Thread-18] o.a.n.controller.FileSystemSwapManager Successfully swapped out 10000 FlowFiles from FlowFileQueue[id=14c03e86-82c8-103d-984e-73d1ee3b19f6] to Swap File /apps1/flowfile-repo/swap/1487853642436-14c03e86-82c8-103d-984e-73d1ee3b19f6-2bfefc74-f917-4d3c-80a2-2707596f1dd8.swap
2017-02-23 14:40:43,617 INFO [Timer-Driven Process Thread-18] o.a.n.c.r.WriteAheadFlowFileRepository Successfully swapped out 10000 FlowFiles from FlowFileQueue[id=14c03e86-82c8-103d-984e-73d1ee3b19f6] to Swap File /apps1/flowfile-repo/swap/1487853642436-14c03e86-82c8-103d-984e-73d1ee3b19f6-2bfefc74-f917-4d3c-80a2-2707596f1dd8.swap
2017-02-23 14:40:44,580 INFO [StandardProcessScheduler Thread-3] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled PutSQL[id=c37c33b4-f4f6-165c-beb3-be6de80a4882] to run with 5 threads
2017-02-23 14:40:44,584 ERROR [Timer-Driven Process Thread-5] o.apache.nifi.processors.standard.PutSQL PutSQL[id=c37c33b4-f4f6-165c-beb3-be6de80a4882] Failed to update database due to a failed batch update. There were a total of 1 FlowFiles that failed, 0 that succeeded, and 0 that were not execute and will be routed to retry;
2017-02-23 14:40:44,584 ERROR [Timer-Driven Process Thread-8] o.apache.nifi.processors.standard.PutSQL PutSQL[id=c37c33b4-f4f6-165c-beb3-be6de80a4882] Failed to update database due to a failed batch update. There were a total of 1 FlowFiles that failed, 0 that succeeded, and 0 that were not execute and will be routed to retry;
2017-02-23 14:40:44,585 ERROR [Timer-Driven Process Thread-17] o.apache.nifi.processors.standard.PutSQL PutSQL[id=c37c33b4-f4f6-165c-beb3-be6de80a4882] Failed to update database due to a failed batch update. There were a total of 1 FlowFiles that failed, 0 that succeeded, and 0 that were not execute and will be routed to retry;
2017-02-23 14:40:44,585 ERROR [Timer-Driven Process Thread-10] o.apache.nifi.processors.standard.PutSQL PutSQL[id=c37c33b4-f4f6-165c-beb3-be6de80a4882] Failed to update database due to a failed batch update. There were a total of 1 FlowFiles that failed, 0 that succeeded, and 0 that were not execute and will be routed to retry;
2017-02-23 14:40:44,585 ERROR [Timer-Driven Process Thread-3] o.apache.nifi.processors.standard.PutSQL PutSQL[id=c37c33b4-f4f6-165c-beb3-be6de80a4882] Failed to update database due to a failed batch update. There were a total of 1 FlowFiles that failed, 0 that succeeded, and 0 that were not execute and will be routed to retry;
2017-02-23 14:40:44,586 ERROR [Timer-Driven Process Thread-17] o.apache.nifi.processors.standard.PutSQL PutSQL[id=c37c33b4-f4f6-165c-beb3-be6de80a4882] Failed to update database due to a failed batch update. There were a total of 1 FlowFiles that failed, 0 that succeeded, and 0 that were not execute and will be routed to retry;
2017-02-23 14:40:44,586 ERROR [Timer-Driven Process Thread-5] o.apache.nifi.processors.standard.PutSQL PutSQL[id=c37c33b4-f4f6-165c-beb3-be6de80a4882] Failed to update database due to a failed batch update. There were a total of 1 FlowFiles that failed, 0 that succeeded, and 0 that were not execute and will be routed to retry;
... View more
02-23-2017
01:36 PM
1 Kudo
Hi, i am loading records from a csv file into Oracle 12c. The PutSQL processor comes up with error message "failed to update database due to failed batch update". I am able to run the insert statements using sql developer but not sure why PutSQL is having issues. What could be wrong?
... View more
Labels:
- Labels:
-
Apache NiFi
02-23-2017
08:20 AM
I ran into the same issue of OOM on NiFi and discovered that the Init and Max memory allocation were set to 512M. Setting both to 4GB resolved my issue. Thanks @Sunile Manjee for recommending the article by @Matt Clarke.
... View more