Member since
09-13-2015
59
Posts
18
Kudos Received
6
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1633 | 12-19-2017 08:14 PM | |
1230 | 10-11-2017 02:21 PM | |
1760 | 06-12-2017 09:26 PM | |
3056 | 06-08-2017 01:36 PM | |
1028 | 11-04-2016 08:35 PM |
08-21-2018
06:50 PM
2 Kudos
Restarting NiFi nodes fails with an error message such as the one below:
<br>File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 278, in
Master().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute
method(env)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 993, in restart
self.start(env, upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 152, in start
nifi_cli.create_or_update_reg_client(params.nifi_registry_host, params.nifi_registry_url)
File "/u/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_cli.py", line 175, in create_or_update_reg_client
existing_clients = list_reg_clients()
File "/u/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_cli.py", line 144, in list_reg_clients
outputType="json"
File "/u/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_cli.py", line 73, in nifi_cli
raise Fail("Failed to execute nifi cli.sh command")
Cause: This occurs when the nifi.initial.admin.identity does not have permission to access the /controller API. Workaround: Create a local user in Ranger with the name of the initial admin idenity Add a Ranger Policy for NiFi with Read/Write permissions for /controller
... View more
- Find more articles tagged with:
- Cloud & Operations
- FAQ
- hdf-3.2.0
- NiFi
- nifi-registry
- upgrade
Labels:
07-27-2018
08:10 PM
Toad for Hadoop has been discontinued and is no longer available from Dell.
... View more
07-12-2018
04:46 PM
I believe this doc contains the information you're looking for: https://github.com/apache/ambari/blob/branch-2.1/ambari-server/docs/api/v1/alert-definitions.md#metric METRIC
METRIC source fields are used to define JMX endpoints that can be queried for values. The source/reporting and jmx/value fields are parameterized to match the property_list specified.
"source" : {
"jmx" : {
"property_list" : [
"java.lang:type=OperatingSystem/SystemCpuLoad",
"java.lang:type=OperatingSystem/AvailableProcessors"
],
"value" : "{0} * 100"
},
"reporting" : {
"ok" : {
"text" : "{1} CPU, load {0:.1%}"
},
"warning" : {
"text" : "{1} CPU, load {0:.1%}",
"value" : 200.0
},
"critical" : {
"text" : "{1} CPU, load {0:.1%}",
"value" : 250.0
},
"units" : "%"
},
"type" : "METRIC",
"uri" : {
"http" : "{{hdfs-site/dfs.namenode.http-address}}",
"https" : "{{hdfs-site/dfs.namenode.https-address}}",
"https_property" : "{{hdfs-site/dfs.http.policy}}",
"https_property_value" : "HTTPS_ONLY",
"default_port" : 0.0,
"high_availability" : {
"nameservice" : "{{hdfs-site/dfs.nameservices}}",
"alias_key" : "{{hdfs-site/dfs.ha.namenodes.{{ha-nameservice}}}}",
"http_pattern" : "{{hdfs-site/dfs.namenode.http-address.{{ha-nameservice}}.{{alias}}}}",
"https_pattern" : "{{hdfs-site/dfs.namenode.https-address.{{ha-nameservice}}.{{alias}}}}"
}
}
}
... View more
02-28-2018
11:02 PM
Unfortunately the jolt spec isn't working as intended. Is is replacing the field names with their values. On to Plan B...
... View more
02-28-2018
10:51 PM
Thank you @Matt Burgess. I will give this a shot. My alternate strategy is to infer an avro schema, and then use @Timothy Spann 's attribute cleaner and then convert back to JSON. https://github.com/tspannhw/nifi-attributecleaner-processor
... View more
02-28-2018
09:34 PM
I have json content streaming in, and I would like to rename the field names, specifically I need to remove "." as I am inserting into MongoDB which does not support periods in key names. Requirements: Only field names are modified, not content. So {"host.ip":"192.168.1.1"} -> {"host_ip":"192.168.1.1"} Field names are unknown, so I can't explicitly replace I would prefer not to split the json array into individual flow files. Has anyone tackled an issue like this before?
... View more
Labels:
- Labels:
-
Apache NiFi
01-02-2018
10:51 PM
It looks like SSH is working. This usually is DNS related; do you have the hosts file set on all the nodes? Do you have the firewall/iptables disabled?
... View more
01-02-2018
10:22 PM
The connection string looks correct to me. Do you have the license jar in the same directory as the jdbc jar?
... View more
12-19-2017
08:47 PM
So if I am understanding this correctly, you want to do the following: Windows Files Share ----> NiFi -----> Hadoop Off the top of my head I can think of a couple ways to do it. 1) Setup the Windows directory to share via FTP. This can be done using IIS on the Windows machine, or a 3rd party FTP server 2) Install MiNiFi or NiFi on the Windows machine to transmit data using site-to-site protocol to the NiFi Cluster Is there a particular reason you don't want to mount the share to the NiFi host?
... View more
12-19-2017
08:14 PM
1 Kudo
There is no native processor to open access databases files. You can handle it a couple ways... Easiest would be to export the data from access into csv and ingest that. The other method would be to connect to the server running MS Access via JDBC. However, I believe this would require a 3rd party driver, as Microsoft only has ODBC drivers for Access (but this may no longer be the case)
... View more
11-21-2017
07:58 PM
I am attempting to query hive_table entities in Atlas using the Atlas webUI. One of the queries I would like to do is filter by db. However, db is not a filter option, and querying where db="name" throws an exception. The only workaround I have found is filtering on the qualifiedName begins_with dbname. Is there something I am missing
... View more
Labels:
- Labels:
-
Apache Atlas
11-15-2017
07:00 PM
@Chad Woodhead Yes, it was resolved. But I don't have a good explanation. It actually resolved itself by simply restarting the NiFi process on all nodes. After the restart they all connected and worked as expected.
... View more
11-08-2017
09:26 PM
It is important to note dataplane itself is just a platform for additional services. At the moment those services are Data Lifecycle Manager (DLM) and the upcoming Data Steward Studio (DSS) and Cloud Manager. 1) It doesn't contain those products (they are contained in HDP) but it leverages them through their APIs to make the data available to dataplane services 2) It allows a single pane of glass for data management across multiple clusters. DLM for disaster recovery and data movement between clusters. DSS for metadata management and auditing. 3) DLM is currently available, DSS is in tech preview. Currently they are only available to Hortonworks customers, I don't know if there is a roadmap to connect to other hadoop distributions. 4) There is no sandbox environment for Dataplane at this time. I hope this helps. I would also recommend checking out the webinar here: https://hortonworks.com/webinar/global-data-management-multi-cloud-hybrid-world/
... View more
10-11-2017
02:21 PM
2 Kudos
I am not aware of any way to do it natively in NiFi. You could always use a REST interface with MongoDB to allow interaction via InvokeHTTP. https://docs.mongodb.com/ecosystem/tools/http-interfaces/#rest-interface
... View more
09-29-2017
09:14 PM
The existing keytab should not require a password. The following command works, and doesn't prompt for a password. kinit -kt /etc/security/keytabs/NiFi_App_Prod.keytab NiFi_App_Prod@<REALM>.com Also, a similar configuration works in the dev environment without issue.
... View more
09-29-2017
08:49 PM
Both HDP and HDF are clustered. 2017-09-29 16:42:41,239 ERROR [StandardProcessScheduler Thread-6] o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1480)
at org.apache.nifi.controller.StandardProcessorNode.access$000(StandardProcessorNode.java:102)
at org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1301)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1463)
... 9 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47)
at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1305)
at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1301)
... 6 common frames omitted
Caused by: java.io.IOException: Login failure for NiFi_App_Prod from keytab /etc/security/keytabs/NiFi_App_Prod.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1183)
at org.apache.nifi.hadoop.SecurityUtil.loginKerberos(SecurityUtil.java:54)
at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.resetHDFSResources(AbstractHadoopProcessor.java:271)
at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.abstractOnScheduled(AbstractHadoopProcessor.java:204)
at org.apache.nifi.processors.hadoop.PutHDFS.onScheduled(PutHDFS.java:191)
... 16 common frames omitted
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
at sun.reflect.GeneratedMethodAccessor609.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1172)
... 20 common frames omitted
... View more
09-29-2017
07:15 PM
1 Kudo
I am getting a login failure when trying to use the PutHDFS processor. These are the things I have verified: nifi.kerberos.krb5.file is set correctly nifi account has read permissions to keytab and krb5.conf I am able to kinit using the keytab/principal from the nifi account Environment is HDF 3.0.1 (NiFi 1.2) to HDP 2.6.x Is there something I am missing?
... View more
Labels:
- Labels:
-
Apache NiFi
09-29-2017
06:25 PM
1 Kudo
It does support ACID: https://cwiki.apache.org/confluence/display/Hive/LLAP#LLAP-ACIDSupport Are you using HDP 2.6?
... View more
09-18-2017
08:38 PM
Please see the following: https://community.hortonworks.com/content/supportkb/49444/can-we-provide-restricted-access-to-resource-manag.html
... View more
09-13-2017
06:47 PM
My apologies, I misunderstood the question. You say the import is working for the table but sqoop-merge is failing. Are you running a merge against a dataset that had null defined as \N and one where null was defined as \n?
... View more
09-13-2017
05:49 PM
It looks like you have \n defined for null-non-string, but in the input it has a null as \N. Changing the null-non-string to \\N should resolve the issue (unless you have both \n and \N in the source, in which case you need to cleanse your data). Hopefully this helps!
... View more
06-13-2017
06:40 PM
Yep, that should work just fine. Longer format is useful if you want to use custom SerDes such as the OpenCSV serde, but the table created using STORED AS AVRO will be identical as the one created explicitly using the org.apache.hadoop.hive.serde2.avro.AvroSerDe.
... View more
06-12-2017
09:26 PM
1 Kudo
They are the same. Hive has a few shortcuts for common SerDes and Avro is one of them. You can use one of the following, or specify your own INPUTFORMAT and OUTPUTFORMAT classes. file_format:
: SEQUENCEFILE
| TEXTFILE -- (Default, depending on hive.default.fileformat configuration)
| RCFILE -- (Note: Available in Hive 0.6.0 and later)
| ORC -- (Note: Available in Hive 0.11.0 and later)
| PARQUET -- (Note: Available in Hive 0.13.0 and later)
| AVRO -- (Note: Available in Hive 0.14.0 and later)
| INPUTFORMAT input_format_classname OUTPUTFORMAT output_format_classname
... View more
06-09-2017
02:47 PM
Is there a work around for this? I have no way of controlling the response headers, and the flowfile is being terminated after transmission. Currently it is just filling the nifi-app.log with unnecessary errors.
... View more
06-08-2017
06:09 PM
I am attempting to sent a POST request to a PowerBI web API. I am using InvokeHTTP processor, and it is sending data (PowerBI is receiving the data). However, every flowfile is routed to Failure with the following error: 2017-06-08 14:01:37,368 ERROR Timer-Driven Process Thread-232 - o.a.nifi.processors.standard.InvokeHTTP
org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from buffer(okio.GzipSource@252df454).inputStream() for StandardFlowFileRecord[uuid=85bfdcdc-e0d6-41f9-8492-1d71e5732e10,claim=,offset=0,name=1963726412837939,size=0] due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.io.EOFException
at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2690) ~[na:na]
at org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:700) ~[nifi-standard-processors-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_91]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_91]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.io.EOFException
at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2683) ~[na:na]
... 13 common frames omitted
Caused by: java.io.EOFException: null
at okio.RealBufferedSource.require(RealBufferedSource.java:64) ~[okio-1.6.0.jar:na]
at okio.GzipSource.consumeHeader(GzipSource.java:114) ~[okio-1.6.0.jar:na]
at okio.GzipSource.read(GzipSource.java:73) ~[okio-1.6.0.jar:na]
at okio.RealBufferedSource$1.read(RealBufferedSource.java:371) ~[okio-1.6.0.jar:na]
at java.io.InputStream.read(InputStream.java:101) ~[na:1.8.0_91]
at org.apache.nifi.stream.io.StreamUtils.copy(StreamUtils.java:35) ~[nifi-utils-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.controller.repository.FileSystemRepository.importFrom(FileSystemRepository.java:700) ~[na:na]
at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2680) ~[na:na]
... 13 common frames omitted Any idea what would cause this exception, and how to fix it?
... View more
Labels:
- Labels:
-
Apache NiFi
06-08-2017
01:36 PM
It appears that this is expected behavior. I discovered a jira pertaining to it here: https://issues.apache.org/jira/browse/HIVE-13865
... View more
06-07-2017
07:18 PM
I have noticed an issue in Hive 1.2 where char data is rtrimmed when converted to string. This doesn't occur when converting varchar to string. Is this a bug? or a feature? select length(cast('abc ' as char(10))),
length(cast('abc ' as varchar(10))),
length(cast('abc ' as string));
_c0,_c1,_c2
3,7,7
In the above example I would expect the length of the char field to be 10.
... View more
- Tags:
- Data Processing
- Hive
Labels:
- Labels:
-
Apache Hive
06-06-2017
07:41 PM
The individual file sizes is not stored in the metastore, so there is not a way to query them directly From within the hive shell you can execute HDFS commands such as dfs -ls /path/to/table to see the individual files and their sizes. If you're interested in the total data size of the table, you can execute: DESCRIBE FORMATTED table_name; and look for the table parameter named totalSize.
... View more