Q&A

I cannot get the following query run using PutHiveQL processor, it fails with the permission denied exception. I see that the processor emulates the same behavior as in Beeline. But, I have been able to run the query from Hive CLI and writes to a file as expected. So, we know that Hive shell is an option, but can you let me know if there is any specific settings that causes this behavior in Beeline (Hive2) preventing to write to local filesystem? insert overwrite local directory '/tmp' select current_date from dual Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [xxxxx] does not have [WRITE] privilege on [/tmp] (state=42000,code=40000)

How to capture the filesize and the filename, which can be found in the details section of the flowfile? I understand that the filename is listed in the attribute section, but is there a way to know the variable name for filesize? kafka-flow-file.jpg

Hello, First time posting here so sorry if this is in the wrong section / wrong format. First, some background : We started a POC using NiFi 1.0.0. We're using a 3 node cluster with limited ressources (this is a POC...). Each of the node has 16 cores, 32gb of ram and 2 volumes : a raid 1 volume for the OS and a Raid 10 volume on 2.5in splindles. I know this is not a recommended setup but the content repo, the provenance repo, the flow files, everything basically, is on the same raid 10 array. The disks are heavily used right now. Content Repo archiving is disabled. Now here's the thing : every 2-3 days, the disk fills up. Right now, the UI reports that we have, in queue : 450 000 (3.21gb). I would expect to have roughly the same amount of data in the nifi/content_repository folder but it's not the case : On one of the node, the content_repo folder is 73gb. I can't tell how big the 2 others nodes are since the "du -h" operation is still running after 10minutes but using "df", I can esti

Hi, I have a case where i have to route incoming flow files to different folders based on the File names. Lets say i have 1000 files that need to send to 5 folders. i thought this would work with Advanced settings of updateattribute where i can write the rules. but the folder name is not being populated.anything wrong i am doing .?? here my test flow..List-->Fetch-->UpdateAttribute-->PutFile and i tried to use the Folder attribute in the putfile ..i tried without single quotes also.. or is there any better approach..?? i need to route 8000 files in to 18 different folders.basically 70 files to 17 folders and remaining 7100+ files to a default folder..(per day 8K files , i need to load data for 18 months) Thanks for your help.. Regards, Sai

Im trying to do an app that takes a json from a web filter the url attribute of the JSON and get the html of this url. My problem is that I can’t set in the property remote url in InvokeHTTP the url that I got in SplitJSON. This is the config of splitJSON (it works, the output is the url that i want) And this is the conf of the InvokeHTTP (it work if I set manually the url) but I dont know how to set a dinamic url.

Hi all, Someone can explain me about "auto Terminate Relationships" in the processor ? thanks

Hello I have a scenario where I am getting confused, - I have Secured 3 node NiFi cluster [Kerberos] - Logged in as user 'admin' trying to do file-based authorization with out Ranger - Created a Process group named Group1 --> 5 processors with in the group - Set modify privilege on just GetSFTP processor for user 'john' in the Group1. - Once I set the privilege for GetSFTP, entire process group is allocated with the modify access for john, not just GetSFTP processor. I am thinking this is not an expected behavior, someone experienced this before? Thanks, Jobin George

Hi,I want to use standalone NiFi 0.6 to ingest data into Kafka within a kerberized HDP2.3.4 cluster.Base config ran smooth and additional config from https://community.hortonworks.com/articles/28180/how-to-configure-hdf-12-to-send-to-and-get-data-fr.html , "Section 2" has been applied.To start playing around I added a GetFile processor and connected it to PutKafka, but after starting the flow, PutKafka fails with the following error:2016-09-20 11:44:38,076 INFO [StandardProcessScheduler Thread-6] o.a.kafka.clients.producer.KafkaProducer Closing the Kafka producer with timeoutMillis = 0 ms.2016-09-20 11:44:38,076 ERROR [StandardProcessScheduler Thread-6] o.apache.nifi.processors.kafka.PutKafka PutKafka[id=9293cf22-d7c2-40a6-ba7e-a6d41942fc30] PutKafka[id=9293cf22-d7c2-40a6-ba7e-a6d41942fc30] failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.; processor will not be scheduled to run for 30000 milliseconds: ja

how to copy files from local desktop lo hdfs location using NiFi?

I have a flow that works on my desktop and I would like to move it to a single server that has 16 cores. My question is there anything I need to configure in NiFi to light up all the cores. I see documentation on clustering servers, which doesn't apply to me, or does NiFI use everything available, which is the behavior I'm looking for?

Storage limit of NN and total storage capacity?

In a typical HDFS HA deployment, Zookeeper, ZKFailoverController, DataNodes, NameNodes and JournalNodes, are all installed. What's the best order of starting/stoping these HDFS services to avoid running into services dependency issues?

Hi, Reading the latest NiFi manual from the Hortworks web site: http://docs.hortonworks.com/HDPDocuments/HDF1/HDF-1.2.0.1/bk_AdminGuide/content/index.html (this uses NiFi version 0.6.0.) The manual talks about creating a 'state' directory you need to use, when you want to use "embedded zookeeper' for a clustered NiFi environment: When using an embedded ZooKeeper, the ./conf/zookeeper.properties file has a property nameddataDir. By default, this value is set to./state/zookeeper. If more than one NiFi node is running an embedded ZooKeeper, it is important to tell the server which one it is. This is accomplished by creating a file named myid and placing it in ZooKeeper's data directory. The contents of this file should be the index of the server as specific by theserver.<number>. So for one of the ZooKeeper servers, we will accomplish this by performing the following commands: cd $NIFI_HOME mkdir state mkdir state/zookeeper echo 1 > state/zookeeper/myidWhen I started looking arou

I am new to HDFS Transparent encryption feature. I am using cloudera CDH 5.4.9 and trying to use hdfs encryption in following manner.# I am using Java KeyStore and configured KMS service and integrated HDFS to use this JAVA KMS# Create two usersuseradd -m user1passwd user1useradd -m user2passwd user2 # As a user1 perform following operations# Create key and create user1 encryption zone# link user1 zone to created keysu user1hadoop key create user1key1hadoop fs -mkdir /tmp/user1zone1su hdfshdfs crypto -createZone -keyName user1key1 -path /tmp/user1zone1 # verify zone is createdhdfs crypto -listZones # create file with user1 credential and put into user1 encryption zoneecho "Hello World" > /tmp/helloWorld.txthadoop fs -put /tmp/helloWorld.txt /tmp/user1zone1hadoop fs -cat /tmp/user1zone1/helloWorld.txtHello World # Verify if file is encryptedsu hdfshadoop fs -cat /.reserved/raw/tmp/zone1/helloWorld.txtT▒▒6▒5▒▒7̼[ # Now login as another usersu user2hadoop fs -cat /tmp/user1zone1/helloW

I am looking deepen my understanding on type of storage disk used for data nodes. outside of single point of failure (SAN box goes down) are they any other reason not to use SAN storage on data nodes? Spindle dedicated on SAN? Is that even possible? How is performance san vs das (dedicated attached storage)? Any insights you can share would be appreciated.

Hello, Hiveserver2 is non-responsive and the log reveals below error. Cluster is Kerberos enabled and there is no HA on Hive. hive.cluster.delegation.token.store.class - org.apache.hadoop.hive.thrift.ZooKeeperTokenStoreAmbari V 2.0HDP V 2.2.4.2 2016-01-19 23:20:18,466 ERROR [HiveServer2-Handler-Pool: Thread-159]: metadata.Hive (Hive.java:getDelegationToken(2910)) - java.lang.UnsupportedOperationException: getDelegationToken() can be called only in thrift (non local) mode at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDelegationToken(HiveMetaStoreClient.java:1664) at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90) at com.sun.proxy.$Proxy12.getDelegationToken(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.ge

Whats the difference between the two?

I use a kerberized cluster and once in a while I notice following error in my zookeeper client logs:   15/11/15 15:46:53 ERROR client.ZooKeeperSaslClient: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Connection reset)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state. 15/11/15 15:46:53 ERROR zookeeper.ClientCnxn: SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Connection reset)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state.   So, I had following doubt with this:   It is showing

Is there a way to configure a non-default replication factor for a HDFS directory such that all future files and sub-directories in that directory use that specific replication factor? Currently, we are using a work around of running a daemon process to set the replication factor for all files in the required directory. Is there a better way to do this? while true; do hdfs dfs -setrep -w 2 /tmp/ sleep 30done I see at one point there was this JIRA https://issues.apache.org/jira/browse/HDFS-199 opened but is blocked by this JIRA [https://issues.apache.org/jira/browse/HADOOP-4771]

Is there a way of restricting access to an HBase coprocessor in a multi tenant environment ? What should i be taking into consideration when using coprocessor ?

Hi, I am setting up a Hadoop cluster and I am having trouble with Zookeeper health tests. Our devops team has closed all ports in the machines except those used by zookeeper, namely: 2181, 3181, and 4181. That makes the Zookeeper canary test to fail. If we open all ports in the machines, the canary test passes. I have seen two types in problems in the logs: WARN org.apache.zookeeper.server.quorum.Learner: Unexpected exception, tries=2, connecting to myhostname/10.2.3.23:3181java.net.ConnectException: Connection refused. Which is odd, because port 3181 is open. and: INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.2.2.11:50561INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14ba997bb9d0e82 with negotiated timeout 30000 for client /10.2.2.11:50561 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.2.2.11:50561 which had sessionid 0x14ba997bb9d0e82WARN com.cloudera.cmf.event.publ

Hello,   I  want to do a folder that can be accessed in r, w,x by all the users of the group 'developer' that I created. I also want that all the folders/files created by these users in this folder can be accessed in r,w,x by the users from this group.  All the other users should access the folder in r and x.   What should I do?   I tried:   - to create my folder with these rights: drwxrwxr-t+   - the folder that I created is owned by admin who is part of the developer group.   All I think is missing is to add the default group of my users the developer group : do something like hadoop def -usermod -G developer Peter But this is not working!      Thank you!    

Hello, We are trying to execute a MapReduce job to write to HBASE using OOZIE Workflow. We have implemented Kerberos in our Cluster and that is when these issues started.... The Job is failing after 10 minutes with an error: We had a similar issue with beeline not able to run HBASE Commands and the solution was - I think the issue is that HBase does not allow hive to impersonate users.  So you'll need to setup hive as a proxy user in HBase.  Can you try the following: - Go to the HDFS service configuration in CM. - Go to Service-Wide->Advanced and add the following to "Cluster-wide Configuration Safety Valve for core-site.xml": I have added similar entry for OOZIE - HDFS Service-Wide Advanced properties and add the following to Cluster-wide ConfigurationSafety Valve for core-site.xml Here is what i have in the logs - Oozie Launcher failed, finishing Hadoop job gracefully