Running HDF 3.1.0, NiFi 1.5.0. Flow contains, among other things, a ListSFTP, then a FetchSFTP, then some processing, where only after succes I'd like to delete the file from the SFTP, and since I don't have a DeleteSFTP available, I do another FetchSFTP with completetion strategy set to Delete (first fetch has None for completion strategy). The issue is that this last FetchSFTP (which actually shouldn't fetch, but just delete) often fails with an error that the file is not found, even though the path is valid/present and just used succesfully for the previous FetchSFTP. ListSFTP is running only on primary node, and the same for the two FetchSFTP. Sometimes the last FetchFTP succeeds after a retry, but it seems if I don't run the flow very slowly, like through a RateControl, then it starts failing more and more, to the point that even repeated retries to fetch/delete the file fails on that last cleanup step. Any ideas on what the reason can be? I don't suspect I need to treat spaces in file names specifically, right? As it works perfectly fine on the previous FetchSFTP, but on second one, reusing the sftp.remote.filename, it then (often) fails. I know there could be other patterns to try out, like move/rename/delete etc, but I think the above should work too. Please help. ... View more

In NiFi (1.5) I need the best way to prepare JSON so it can be parsed by the Hive json serde ('org.apache.hive.hcatalog.data.JsonSerDe). I have an array of, here two, records in this format (from a REST API response). [   {         "id": 1,         "call_status": "OK",         "result": 0.0239,         "explanation": [             "some_var",             "another_var"         ],         "foo": "OK"     },     {         "id": 2,         "call_status": "OK",         "result": 0.0239,         "explanation": [             "some_var",             "another_var"         ],         "foo": "OK"     } ] It seems it should be transformed to this format for the serde to work, which I tried doing manually with succes. No array brackets and one record per line. { "id": 1, "call_status": "OK", "result": 0.0239, "explanation": [ "some_var", "another_var" ], "foo": "OK" } { "id": 2, "call_status": "OK", "result": 0.0239, "explanation": [ "some_var", "another_var" ], "foo": "OK" } What is the recommended / most efficient way of doing this transformation? Preferably without having to input the schema, as it would be nice with a generic solution. If a schema _is_ required, let's say for Record processors, I'll live with that. ... View more

I have a problem inserting NiFi Records with null values into Sybase ASE and IQ with PutDatabaseRecord (version 1.5.0.3). I am using jconn4.jar and com.sybase.jdbc4.jdbc.SybDriver for connecting. The affected column in the database is set to allow null, and works fine when inserting values if not null. But if null in the Record, like this [ { "field1" : "abcd", "field2" : null } ] it fails with this message: PutDatabaseRecord[id=d1fb33c6-8b72-18b0-95f9-6c437879ff44] Failed to process StandardFlowFileRecord[uuid=8bb1975a-c169-4a6f-8b48-98225f346a75,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1551948944283-259989, container=default, section=917], offset=16640, length=47016],offset=0,name=XXX,size=47016] due to JZ0SE: Invalid object type (or null object) specified for setObject().: That error is described by Sybase as JZ0SE: Invalid object type specified for setObject(). Description: Illegal type argument passed to PreparedStatement.setObject. Action: Check the JDBC documentation. The argument must be a constant from java.sql.Types. Is there somewhere a bug where setNull() should have been called for NULL values instead of setObject()? No clue where or how to fix. Can anyone help? ... View more

Using ConsumeKafka processor in NiFi - can the Topic Name(s) property list be dynamically updated (without a processor restart)? I have a need to expose a web service trough NiFi that can retrieve metadata on which topic names to consume/ingest. So when receiving a new one topic consume request, I need to update the Topic Name(s) property list of the ConsumeKafka processor (adding the new topic to the list). How do I achieve that? Does it afterwards require a restart of the ConsumeKafka processor (through the NiFi REST API)? The topic list is planned to be persisted somewhere. Could simply be a flat file or in RDBMS, HBASE or just about anywhere. Any preferences recommended? ... View more

In PutHDFS I have set Conflict Resolution Strategy to Fail as I don't want to overwrite existing files. But for error handling and logging/notification, I need to differentiate file-exist fails from other types of fails from this processor. How is that possible? In bulletin board I can see a text message from the processor indicating when file exists, but how do I get that info in the flow itself? Is the message / fail type available for flow control handling somehow? Suggestions? ... View more

I'm seeing processors with very little input generate a tsunami of tasks (thousands within a couple of seconds) when Run Schedule is set to 0ms (run duration to 0 also). I have the understanding that 0ms in Run Schedule should be interpreted as "always on" / "continuously", like a HTTP request handler or similar listener, always ready for handling requests individually and immediately when received. I have a case with an UpdateAttribute processor, also with Run Schedule to 0ms. It generates no tasks when no incoming flowfiles, and only generates as few as incoming flowfiles require (this case, very few, say 10 for 10 incoming test flow files). But if the FlowFiles are penalized (this instance a full day, 1d, coming from PutHDFS) then the task generation goes crazy (like thousands per seconds), without actually doing any work (no flow files move through). Why the high number of tasks generations? Seems to affect the cluster. We notice similar tsunami task generation on other processors, like HTTP request handler. It's unexpected behaviour to me. Doesn't seem as a robust reaction. Is it intended/expected outcome or a bug? ... View more

Using HDP 2.6.4 and Sqoop 1.4.6. So not Sqoop 2, which I understand is dropped. I'm importing from a postgresql with simple username and password (no Kerberos) to Hive/HDFS which does use Kerberos. Besides doing a kinit first, do I need to tell sqoop somehow that the underlying import work to Hive/HDFS (initiated by the sqoop import command) needs to use Kerberos? If so, how? ... View more

As my own user (not system user) I cannot create an external table via a .hql script with hive -f. use db; create external table `table` ( columns...) stored as orc LOCATION '/external/path/to/my/db/table' TBLPROPERTIES ("orc.compress"="ZLIB"); I get this $ hive -f ddl_create_db.table.hql FAILED: SemanticException MetaException(message:org.apache.hadoop.security.AccessControlException: Permission denied: user=<my-user>, access=EXECUTE, inode="/apps/hive/warehouse/db.db":hdfs:hdfs:d--------- at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:292) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:238) at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:428) at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:365) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:108) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:4142) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1137) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:866) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347) ) Notice it's tryign to access /apps/hive/warehouse/db.db, which I correctly don't have access to with this user, therefore the design requirement to use external tables with another path which I do have access to. It works fine from Ambari, but not from command line from where I need to script/automate it. ... View more

Using Hive 1.2.1000.2.6.4.0-91. When running a 'hive -e' from command line, I get a message about -chmod and hadoop fs. See below. Any idea why I get this, and how to fix it? The -S doesn't suppress it either, as shown. [ ~] hive -S -e 'select max(entry_timestamp) from xx.yy' log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender. -chmod: chmod : mode '0' does not match the expected pattern. Usage: hadoop fs [generic options] [-appendToFile <localsrc> ... <dst>] [-cat [-ignoreCrc] <src> ...] [-checksum <src> ...] [-chgrp [-R] GROUP PATH...] [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...] [-chown [-R] [OWNER][:[GROUP]] PATH...] [-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>] [-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] [-count [-q] [-h] [-v] [-t [<storage type>]] [-u] <path> ...] [-cp [-f] [-p | -p[topax]] <src> ... <dst>] [-createSnapshot <snapshotDir> [<snapshotName>]] [-deleteSnapshot <snapshotDir> <snapshotName>] [-df [-h] [<path> ...]] [-du [-s] [-h] <path> ...] [-expunge] [-find <path> ... <expression> ...] [-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] [-getfacl [-R] <path>] [-getfattr [-R] {-n name | -d} [-e en] <path>] [-getmerge [-nl] <src> <localdst>] [-help [cmd ...]] [-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [<path> ...]] [-mkdir [-p] <path> ...] [-moveFromLocal <localsrc> ... <dst>] [-moveToLocal <src> <localdst>] [-mv <src> ... <dst>] [-put [-f] [-p] [-l] <localsrc> ... <dst>] [-renameSnapshot <snapshotDir> <oldName> <newName>] [-rm [-f] [-r|-R] [-skipTrash] [-safely] <src> ...] [-rmdir [--ignore-fail-on-non-empty] <dir> ...] [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]] [-setfattr {-n name [-v value] | -x name} <path>] [-setrep [-R] [-w] <rep> <path> ...] [-stat [format] <path> ...] [-tail [-f] <file>] [-test -[defsz] <path>] [-text [-ignoreCrc] <src> ...] [-touchz <path> ...] [-truncate [-w] <length> <path> ...] [-usage [cmd ...]] Generic options supported are -conf <configuration file> specify an application configuration file -D <property=value> use value for given property -fs <local|namenode:port> specify a namenode -jt <local|resourcemanager:port> specify a ResourceManager -files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath. -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines. The general command line syntax is bin/hadoop command [genericOptions] [commandOptions] Usage: hadoop fs [generic options] -chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH... [ ~] 2018-08-31 12:42:14.41040 ... View more

Using Sybase Adaptive Server Enterprise (ASE) 15.7, Sqoop 1.4.6.2.5.0.0-1245 and jconn4 as driver. I have a table in Sybase ASE with some columns of type 'char' with various length (sometimes called precision in metadata dump), some at 10, some at 1 - so basically char(10) and char(1). When importing with scoop with table creation it maps the char columns wrongly. Doing something like this sqoop import --driver com.sybase.jdbc4.jdbc.SybDriver --connect "jdbc:sybase:Tds:XXX/XXX" --username XXX --password-file XXX --table XXX -m 1 --create-hcatalog-table ... I notice it gets the column type readout wrong for the char columns. It says 0 for Precision, not 10, 1 or whatever is chosen in Sybase. 18/07/24 10:18:36 INFO hcat.SqoopHCatUtilities: Database column name - info map : ... XXX : [Type : 1,Precision : 0,Scale : 0] ... Later, at create table phase, it decides to map the char columns to char(65535) no matter the actual length in Sybase, like this 18/07/24 10:18:36 INFO hcat.SqoopHCatUtilities: HCatalog Create table statement: create table `XXX`.`XXX` ( `XXX` char(65535), ... That then fails, as char is only valid in Hive between 1 and 255. 18/07/24 10:18:42 INFO hcat.SqoopHCatUtilities: FAILED: RuntimeException Char length 65535 out of allowed range [1, 255] I can override the wrong automatic mapping manually with --map-column-hive XXX="char(10)", but I want to avoid that as I need it automatically to work for a lot of tables. Hope anyone can help fix this issue. Thanks. ... View more