In NiFi (1.5) I need the best way to prepare JSON so it can be parsed by the Hive json serde ('org.apache.hive.hcatalog.data.JsonSerDe). I have an array of, here two, records in this format (from a REST API response). [   {         "id": 1,         "call_status": "OK",         "result": 0.0239,         "explanation": [             "some_var",             "another_var"         ],         "foo": "OK"     },     {         "id": 2,         "call_status": "OK",         "result": 0.0239,         "explanation": [             "some_var",             "another_var"         ],         "foo": "OK"     } ] It seems it should be transformed to this format for the serde to work, which I tried doing manually with succes. No array brackets and one record per line. { "id": 1, "call_status": "OK", "result": 0.0239, "explanation": [ "some_var", "another_var" ], "foo": "OK" } { "id": 2, "call_status": "OK", "result": 0.0239, "explanation": [ "some_var", "another_var" ], "foo": "OK" } What is the recommended / most efficient way of doing this transformation? Preferably without having to input the schema, as it would be nice with a generic solution. If a schema _is_ required, let's say for Record processors, I'll live with that. ... View more

Using ConsumeKafka processor in NiFi - can the Topic Name(s) property list be dynamically updated (without a processor restart)? I have a need to expose a web service trough NiFi that can retrieve metadata on which topic names to consume/ingest. So when receiving a new one topic consume request, I need to update the Topic Name(s) property list of the ConsumeKafka processor (adding the new topic to the list). How do I achieve that? Does it afterwards require a restart of the ConsumeKafka processor (through the NiFi REST API)? The topic list is planned to be persisted somewhere. Could simply be a flat file or in RDBMS, HBASE or just about anywhere. Any preferences recommended? ... View more

In PutHDFS I have set Conflict Resolution Strategy to Fail as I don't want to overwrite existing files. But for error handling and logging/notification, I need to differentiate file-exist fails from other types of fails from this processor. How is that possible? In bulletin board I can see a text message from the processor indicating when file exists, but how do I get that info in the flow itself? Is the message / fail type available for flow control handling somehow? Suggestions? ... View more

I'm seeing processors with very little input generate a tsunami of tasks (thousands within a couple of seconds) when Run Schedule is set to 0ms (run duration to 0 also). I have the understanding that 0ms in Run Schedule should be interpreted as "always on" / "continuously", like a HTTP request handler or similar listener, always ready for handling requests individually and immediately when received. I have a case with an UpdateAttribute processor, also with Run Schedule to 0ms. It generates no tasks when no incoming flowfiles, and only generates as few as incoming flowfiles require (this case, very few, say 10 for 10 incoming test flow files). But if the FlowFiles are penalized (this instance a full day, 1d, coming from PutHDFS) then the task generation goes crazy (like thousands per seconds), without actually doing any work (no flow files move through). Why the high number of tasks generations? Seems to affect the cluster. We notice similar tsunami task generation on other processors, like HTTP request handler. It's unexpected behaviour to me. Doesn't seem as a robust reaction. Is it intended/expected outcome or a bug? ... View more

Using HDP 2.6.4 and Sqoop 1.4.6. So not Sqoop 2, which I understand is dropped. I'm importing from a postgresql with simple username and password (no Kerberos) to Hive/HDFS which does use Kerberos. Besides doing a kinit first, do I need to tell sqoop somehow that the underlying import work to Hive/HDFS (initiated by the sqoop import command) needs to use Kerberos? If so, how? ... View more

As my own user (not system user) I cannot create an external table via a .hql script with hive -f. use db; create external table `table` ( columns...) stored as orc LOCATION '/external/path/to/my/db/table' TBLPROPERTIES ("orc.compress"="ZLIB"); I get this $ hive -f ddl_create_db.table.hql FAILED: SemanticException MetaException(message:org.apache.hadoop.security.AccessControlException: Permission denied: user=<my-user>, access=EXECUTE, inode="/apps/hive/warehouse/db.db":hdfs:hdfs:d--------- at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:292) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:238) at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:428) at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:365) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:108) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:4142) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1137) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:866) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347) ) Notice it's tryign to access /apps/hive/warehouse/db.db, which I correctly don't have access to with this user, therefore the design requirement to use external tables with another path which I do have access to. It works fine from Ambari, but not from command line from where I need to script/automate it. ... View more

Using Hive 1.2.1000.2.6.4.0-91. When running a 'hive -e' from command line, I get a message about -chmod and hadoop fs. See below. Any idea why I get this, and how to fix it? The -S doesn't suppress it either, as shown. [ ~] hive -S -e 'select max(entry_timestamp) from xx.yy' log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender. -chmod: chmod : mode '0' does not match the expected pattern. Usage: hadoop fs [generic options] [-appendToFile <localsrc> ... <dst>] [-cat [-ignoreCrc] <src> ...] [-checksum <src> ...] [-chgrp [-R] GROUP PATH...] [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...] [-chown [-R] [OWNER][:[GROUP]] PATH...] [-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>] [-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] [-count [-q] [-h] [-v] [-t [<storage type>]] [-u] <path> ...] [-cp [-f] [-p | -p[topax]] <src> ... <dst>] [-createSnapshot <snapshotDir> [<snapshotName>]] [-deleteSnapshot <snapshotDir> <snapshotName>] [-df [-h] [<path> ...]] [-du [-s] [-h] <path> ...] [-expunge] [-find <path> ... <expression> ...] [-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] [-getfacl [-R] <path>] [-getfattr [-R] {-n name | -d} [-e en] <path>] [-getmerge [-nl] <src> <localdst>] [-help [cmd ...]] [-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [<path> ...]] [-mkdir [-p] <path> ...] [-moveFromLocal <localsrc> ... <dst>] [-moveToLocal <src> <localdst>] [-mv <src> ... <dst>] [-put [-f] [-p] [-l] <localsrc> ... <dst>] [-renameSnapshot <snapshotDir> <oldName> <newName>] [-rm [-f] [-r|-R] [-skipTrash] [-safely] <src> ...] [-rmdir [--ignore-fail-on-non-empty] <dir> ...] [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]] [-setfattr {-n name [-v value] | -x name} <path>] [-setrep [-R] [-w] <rep> <path> ...] [-stat [format] <path> ...] [-tail [-f] <file>] [-test -[defsz] <path>] [-text [-ignoreCrc] <src> ...] [-touchz <path> ...] [-truncate [-w] <length> <path> ...] [-usage [cmd ...]] Generic options supported are -conf <configuration file> specify an application configuration file -D <property=value> use value for given property -fs <local|namenode:port> specify a namenode -jt <local|resourcemanager:port> specify a ResourceManager -files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath. -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines. The general command line syntax is bin/hadoop command [genericOptions] [commandOptions] Usage: hadoop fs [generic options] -chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH... [ ~] 2018-08-31 12:42:14.41040 ... View more

Using Sybase Adaptive Server Enterprise (ASE) 15.7, Sqoop 1.4.6.2.5.0.0-1245 and jconn4 as driver. I have a table in Sybase ASE with some columns of type 'char' with various length (sometimes called precision in metadata dump), some at 10, some at 1 - so basically char(10) and char(1). When importing with scoop with table creation it maps the char columns wrongly. Doing something like this sqoop import --driver com.sybase.jdbc4.jdbc.SybDriver --connect "jdbc:sybase:Tds:XXX/XXX" --username XXX --password-file XXX --table XXX -m 1 --create-hcatalog-table ... I notice it gets the column type readout wrong for the char columns. It says 0 for Precision, not 10, 1 or whatever is chosen in Sybase. 18/07/24 10:18:36 INFO hcat.SqoopHCatUtilities: Database column name - info map : ... XXX : [Type : 1,Precision : 0,Scale : 0] ... Later, at create table phase, it decides to map the char columns to char(65535) no matter the actual length in Sybase, like this 18/07/24 10:18:36 INFO hcat.SqoopHCatUtilities: HCatalog Create table statement: create table `XXX`.`XXX` ( `XXX` char(65535), ... That then fails, as char is only valid in Hive between 1 and 255. 18/07/24 10:18:42 INFO hcat.SqoopHCatUtilities: FAILED: RuntimeException Char length 65535 out of allowed range [1, 255] I can override the wrong automatic mapping manually with --map-column-hive XXX="char(10)", but I want to avoid that as I need it automatically to work for a lot of tables. Hope anyone can help fix this issue. Thanks. ... View more

Is there a way to use the --hcatalog options with sqoop import while maintaining control over destination dir, like --warehouse-dir or --destination-dir? These options appear to be incompatible, but perhaps there's a trick. I'd like to import and create table definitions for Hive in a single go, storing as avro or orc, while controlling the destination dir, so I'm not forced to Hive's default/managed directory. There are various reasons for not placing in default warehouse dir, one being I don't currently have permissions to do to (so getting an error in sqoop import with --hcatalog options), another being we have a dir placement strategy not using the default dir. ... View more