Member since
05-17-2016
190
Posts
46
Kudos Received
11
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1387 | 09-07-2017 06:24 PM | |
1790 | 02-24-2017 06:33 AM | |
2575 | 02-10-2017 09:18 PM | |
7066 | 01-11-2017 08:55 PM | |
4707 | 12-15-2016 06:16 PM |
02-09-2017
04:01 PM
1 Kudo
Is there any document that talks about the number of input sources that a NiFi cluster can connect to?
I believe there is no such thing as "number of source systems". I have seen the sizing chart
here by @Matt Clarke.
Is it a fair assumption that - maximum 2 source systems can connect to a 2 node cluster under the condition that each source streams 500 message at 25 MB/s ?
Also, is there any supporting document for the data flows, processors and workload under which the sizing chart was created?
... View more
Labels:
- Labels:
-
Apache NiFi
02-02-2017
02:41 PM
@Matt Have been able to get this resolved. Thanks for your time. I figured out that any access issue gets redirected to a HTML page and that is why the XML parse error came up.
... View more
02-01-2017
09:43 PM
Hi All, I am trying to configure a ListS3 processor. However I get the below error and not sure what went wrong. failed to process session due to com.amazonaws.AmazonClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler: com.amazonaws.AmazonClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandle
My bucket name takes the form source/customername And I have configured the following fields,
bucket region Access Key Secret Key
Is there something that I am missing? EDIT The full Stack Trace is as below 2017-02-01 21:25:12,772 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@56017274 checkpointed with 2 Records and 0 Swap Files in 14 milliseconds (Stop-the-world time = 0 milliseconds, Clear Edit Logs time = 0 millis), max Transaction ID 5
2017-02-01 21:25:33,057 INFO [pool-8-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2017-02-01 21:25:33,129 INFO [pool-8-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@6ae5e1cf checkpointed with 103 Records and 0 Swap Files in 71 milliseconds (Stop-the-world time = 13 milliseconds, Clear Edit Logs time = 13 millis), max Transaction ID 40
2017-02-01 21:25:33,129 INFO [pool-8-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 103 records in 71 milliseconds
2017-02-01 21:25:45,904 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@4725617a // Another save pending = false
2017-02-01 21:25:49,512 INFO [StandardProcessScheduler Thread-3] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled ListS3[id=fb75523d-0159-1000-fa7e-9293836c0d73] to run with 1 threads
2017-02-01 21:25:49,911 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@4725617a // Another save pending = false
2017-02-01 21:25:49,986 ERROR [Timer-Driven Process Thread-2] org.apache.nifi.processors.aws.s3.ListS3 ListS3[id=fb75523d-0159-1000-fa7e-9293836c0d73] ListS3[id=fb75523d-0159-1000-fa7e-9293836c0d73] failed to process due to com.amazonaws.AmazonClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler; rolling back session: com.amazonaws.AmazonClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
2017-02-01 21:25:49,988 ERROR [Timer-Driven Process Thread-2] org.apache.nifi.processors.aws.s3.ListS3
com.amazonaws.AmazonClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:127) ~[aws-java-sdk-s3-1.11.8.jar:na]
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:270) ~[aws-java-sdk-s3-1.11.8.jar:na]
at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:67) ~[aws-java-sdk-s3-1.11.8.jar:na]
at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:57) ~[aws-java-sdk-s3-1.11.8.jar:na]
at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62) ~[aws-java-sdk-s3-1.11.8.jar:na]
at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31) ~[aws-java-sdk-s3-1.11.8.jar:na]
at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1244) ~[aws-java-sdk-core-1.11.8.jar:na]
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:885) ~[aws-java-sdk-core-1.11.8.jar:na]
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:695) ~[aws-java-sdk-core-1.11.8.jar:na]
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:447) ~[aws-java-sdk-core-1.11.8.jar:na]
at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:409) ~[aws-java-sdk-core-1.11.8.jar:na]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:358) ~[aws-java-sdk-core-1.11.8.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3787) ~[aws-java-sdk-s3-1.11.8.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3729) ~[aws-java-sdk-s3-1.11.8.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:606) ~[aws-java-sdk-s3-1.11.8.jar:na]
at org.apache.nifi.processors.aws.s3.ListS3.onTrigger(ListS3.java:184) ~[nifi-aws-processors-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064) [nifi-framework-core-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.0.0.2.0.1.0-12.jar:1.0.0.2.0.1.0-12]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Caused by: org.xml.sax.SAXParseException: Premature end of file.
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203) ~[na:1.8.0_121]
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177) ~[na:1.8.0_121]
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400) ~[na:1.8.0_121]
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327) ~[na:1.8.0_121]
at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1472) ~[na:1.8.0_121]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1014) ~[na:1.8.0_121]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602) ~[na:1.8.0_121]
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112) ~[na:1.8.0_121]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505) ~[na:1.8.0_121]
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:841) ~[na:1.8.0_121]
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:770) ~[na:1.8.0_121]
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) ~[na:1.8.0_121]
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213) ~[na:1.8.0_121]
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:114) ~[aws-java-sdk-s3-1.11.8.jar:na]
... 27 common frames omitted
2017-02-01 21:25:49,988 ERROR [Timer-Driven Process Thread-2] org.apache.nifi.processors.aws.s3.ListS3 ListS3[id=fb75523d-0159-1000-fa7e-9293836c0d73] ListS3[id=fb75523d-0159-1000-fa7e-9293836c0d73] failed to process session
... View more
Labels:
- Labels:
-
Apache NiFi
01-30-2017
06:35 PM
@Bryan Bende : Thanks for pointing the Jira.
... View more
01-30-2017
05:36 PM
Hi All,
I am doing an attribute to json conversion using NiFi AttributesToJSON Processor. What is observed is that any key value pair is mapped to String datatype in the resultant JSON, which I believe is the intended behavior.
However, is there any known approaches to specify the datatype for a Key-Value pair? Currently I am employing an ExecuteScript processor once the JSON conversion is done to convert the "String" data type to whatever datatype I want the key-value to be in.
Are there other better approaches?
... View more
Labels:
- Labels:
-
Apache NiFi
01-30-2017
04:33 PM
@bhagan : Sorry, I was looking at if timestamp data type could be used in NiFi AVRO related processor. Unfortunately not, with NiFi 1.1.1 (the one I am using) because Avro is still 1.7 and not yet updated to 1.8 there.
I believe this is what @Artem Ervits meant by his answer.
... View more
01-27-2017
09:39 PM
@Timothy Spann This is NiFi 1.1.1. I believe avro is still 1.7.7. Not sure if it is 1.8.
... View more
01-27-2017
06:45 PM
Thanks @Timothy Spann The Avro version is 1.7.7 (looking at the pom.xml).
Is there a timestamp type in avro 1.7.7? I guess not. I could see the logicalType timestamp-* in 1.8
Should I assume that the timestamp would not work.
... View more
01-27-2017
02:55 PM
@bhagan : Is timestamp a supported avro datatype?
... View more