Member since
09-13-2015
59
Posts
18
Kudos Received
6
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2682 | 12-19-2017 08:14 PM | |
2097 | 10-11-2017 02:21 PM | |
2890 | 06-12-2017 09:26 PM | |
4498 | 06-08-2017 01:36 PM | |
1702 | 11-04-2016 08:35 PM |
09-29-2017
08:49 PM
Both HDP and HDF are clustered. 2017-09-29 16:42:41,239 ERROR [StandardProcessScheduler Thread-6] o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1480)
at org.apache.nifi.controller.StandardProcessorNode.access$000(StandardProcessorNode.java:102)
at org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1301)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1463)
... 9 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47)
at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1305)
at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1301)
... 6 common frames omitted
Caused by: java.io.IOException: Login failure for NiFi_App_Prod from keytab /etc/security/keytabs/NiFi_App_Prod.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1183)
at org.apache.nifi.hadoop.SecurityUtil.loginKerberos(SecurityUtil.java:54)
at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.resetHDFSResources(AbstractHadoopProcessor.java:271)
at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.abstractOnScheduled(AbstractHadoopProcessor.java:204)
at org.apache.nifi.processors.hadoop.PutHDFS.onScheduled(PutHDFS.java:191)
... 16 common frames omitted
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
at sun.reflect.GeneratedMethodAccessor609.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1172)
... 20 common frames omitted
... View more
09-29-2017
07:15 PM
1 Kudo
I am getting a login failure when trying to use the PutHDFS processor. These are the things I have verified: nifi.kerberos.krb5.file is set correctly nifi account has read permissions to keytab and krb5.conf I am able to kinit using the keytab/principal from the nifi account Environment is HDF 3.0.1 (NiFi 1.2) to HDP 2.6.x Is there something I am missing?
... View more
Labels:
- Labels:
-
Apache NiFi
09-13-2017
06:47 PM
My apologies, I misunderstood the question. You say the import is working for the table but sqoop-merge is failing. Are you running a merge against a dataset that had null defined as \N and one where null was defined as \n?
... View more
09-13-2017
05:49 PM
It looks like you have \n defined for null-non-string, but in the input it has a null as \N. Changing the null-non-string to \\N should resolve the issue (unless you have both \n and \N in the source, in which case you need to cleanse your data). Hopefully this helps!
... View more
06-13-2017
06:40 PM
Yep, that should work just fine. Longer format is useful if you want to use custom SerDes such as the OpenCSV serde, but the table created using STORED AS AVRO will be identical as the one created explicitly using the org.apache.hadoop.hive.serde2.avro.AvroSerDe.
... View more
06-12-2017
09:26 PM
1 Kudo
They are the same. Hive has a few shortcuts for common SerDes and Avro is one of them. You can use one of the following, or specify your own INPUTFORMAT and OUTPUTFORMAT classes. file_format:
: SEQUENCEFILE
| TEXTFILE -- (Default, depending on hive.default.fileformat configuration)
| RCFILE -- (Note: Available in Hive 0.6.0 and later)
| ORC -- (Note: Available in Hive 0.11.0 and later)
| PARQUET -- (Note: Available in Hive 0.13.0 and later)
| AVRO -- (Note: Available in Hive 0.14.0 and later)
| INPUTFORMAT input_format_classname OUTPUTFORMAT output_format_classname
... View more
06-09-2017
02:47 PM
Is there a work around for this? I have no way of controlling the response headers, and the flowfile is being terminated after transmission. Currently it is just filling the nifi-app.log with unnecessary errors.
... View more
06-08-2017
06:09 PM
I am attempting to sent a POST request to a PowerBI web API. I am using InvokeHTTP processor, and it is sending data (PowerBI is receiving the data). However, every flowfile is routed to Failure with the following error: 2017-06-08 14:01:37,368 ERROR Timer-Driven Process Thread-232 - o.a.nifi.processors.standard.InvokeHTTP
org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from buffer(okio.GzipSource@252df454).inputStream() for StandardFlowFileRecord[uuid=85bfdcdc-e0d6-41f9-8492-1d71e5732e10,claim=,offset=0,name=1963726412837939,size=0] due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.io.EOFException
at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2690) ~[na:na]
at org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:700) ~[nifi-standard-processors-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_91]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_91]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.io.EOFException
at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2683) ~[na:na]
... 13 common frames omitted
Caused by: java.io.EOFException: null
at okio.RealBufferedSource.require(RealBufferedSource.java:64) ~[okio-1.6.0.jar:na]
at okio.GzipSource.consumeHeader(GzipSource.java:114) ~[okio-1.6.0.jar:na]
at okio.GzipSource.read(GzipSource.java:73) ~[okio-1.6.0.jar:na]
at okio.RealBufferedSource$1.read(RealBufferedSource.java:371) ~[okio-1.6.0.jar:na]
at java.io.InputStream.read(InputStream.java:101) ~[na:1.8.0_91]
at org.apache.nifi.stream.io.StreamUtils.copy(StreamUtils.java:35) ~[nifi-utils-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.controller.repository.FileSystemRepository.importFrom(FileSystemRepository.java:700) ~[na:na]
at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2680) ~[na:na]
... 13 common frames omitted Any idea what would cause this exception, and how to fix it?
... View more
Labels:
- Labels:
-
Apache NiFi