Member since
05-16-2016
76
Posts
44
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
973 | 03-10-2016 08:52 PM |
10-27-2016
06:38 PM
1 Kudo
Is there any better way to pull HDFS data into Kafka? I could use a flume exec source and kafka sink, but any alternative?
... View more
Labels:
- Labels:
-
Apache Kafka
10-17-2016
03:52 AM
Sunile. For daemons such as NM and RM, I can hard-code JMS port number. However, how to set different JMX port number on a number of JVM containers running on the same slave node? For example, I can configure 18 container per node, how could I set the port number to 8001-8018? Maybe I can use ZK, or some kind of script.
... View more
10-17-2016
03:45 AM
My problem is how to set different JMX ports on the containers running on the same slave node.
... View more
10-14-2016
07:29 PM
@Timothy Spann Tim, Thanks for the info. I can pull daemon JMX for sure since they are running all the time. However, I am more interested on per job based container. How to query YARN restful api to get a list of container for a job?
... View more
10-14-2016
02:52 PM
@mhendricks I have the exact same interest on monitoring all the containers of the job via JMX. However, my question is how to get the list of containers for a job assuming we can turn ON JMX for yarn containers?
... View more
10-14-2016
02:41 PM
Great article. Is there any way to pull individual yarn container JMX? It would be great for me to see what is going on when I start a job and monitor the job containers at real time.
... View more
09-07-2016
09:20 PM
The more, the better, to some extent.
... View more
09-06-2016
09:34 PM
3 Kudos
Should 3 be sufficient for a 3 rack cluster with one ZK per rack? Does increasing ZK nodes to 5 make sense? My understanding is that for fault tolerant 3 ZKs are good enough. Having 2 ZK nodes on the same rack doesn't increase HA.
... View more
08-29-2016
02:49 PM
This is NOT about HiveServer2, but only on Hive metastore. Do we have only two choices on Hive metastore property hive.security.metastore.authorization.manager? StorageBasedAuthorizationProvider DefaultHiveMetastoreAuthorizationProvider
... View more
Labels:
- Labels:
-
Apache Hive
07-28-2016
04:47 PM
# H2 Settings nifi.database.directory=./database_repository
nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE Is something to be added here?
... View more
07-28-2016
04:45 PM
1 Kudo
I installed NIFI in two separate location nifi and nifi2 with different ports - one without kerberized ZK and one with kerberized ZK. However, both web UI (8080 and 9080) see the same nifi flow graphs. How could they share the same internal database such as H2?
... View more
Labels:
- Labels:
-
Apache NiFi
07-27-2016
04:53 PM
Yes. However, we use adkeytab from Centrify. Here is a blog on this (he used adkeytab to add spn and kutil to merge with no detailed steps). https://www.centrify.com/activate/cmz3c60glnnmc1tnlwnv0srcpkfct5p9/
... View more
07-27-2016
04:29 PM
http://community.centrify.com/t5/Centrify-Insight/Difference-between-adkeytab-and-ktutil/td-p/14281 Did you use ktutil to merge?
... View more
07-26-2016
03:25 PM
Thanks Sunile. We got that already. This is for adding two principals in the same key tab using AD, not MIT kdc.
... View more
07-26-2016
02:39 PM
Anyone knows how to do with AD adkeytab command? https://community.hortonworks.com/articles/35019/oozie-ha-configuration-with-kerberos.html step 8 requires to add more than more principals in the same keytab.
[root@kk2 ~]# klist -ket /etc/security/keytabs/spnego.service.keytab Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab KVNO TimestampPrincipal ----------------------------------------------------------------------------- 305/03/1616:44:05 HTTP/kk2.hwxblr.com@HWX.COM (aes256-cts-hmac-sha1-96) 305/03/1616:44:05 HTTP/kk2.hwxblr.com@HWX.COM (aes128-cts-hmac-sha1-96) 305/03/1616:44:05 HTTP/kk2.hwxblr.com@HWX.COM (des3-cbc-sha1) 305/03/1616:44:05 HTTP/kk2.hwxblr.com@HWX.COM (arcfour-hmac) 505/03/1616:44:11 HTTP/test1-ambari-server.hwxblr.com@HWX.COM (aes256-cts-hmac-sha1-96) 505/03/1616:44:11 HTTP/test1-ambari-server.hwxblr.com@HWX.COM (aes128-cts-hmac-sha1-96) 505/03/1616:44:11 HTTP/test1-ambari-server.hwxblr.com@HWX.COM (des3-cbc-sha1) 505/03/1616:44:11 HTTP/test1-ambari-server.hwxblr.com@HWX.COM (arcfour-hmac)
... View more
Labels:
- Labels:
-
Apache Oozie
07-25-2016
02:12 PM
Do you mean that each nifi user should have his/her own instance? There could be too many. Also is there any way to specify multiple instances per host?
... View more
07-20-2016
04:31 PM
Currently HDF 1.2 is running as user root. I am using HDP 2.3.2 so NIFI is not part of Ambari controlled list. Since it is run as root, getHDFS or putHDFS processor uses a user keytab. How could I prevent a user A to create a processor to use user B's keytab? In HDP 2.4 or later, do we have NIFI service user ID provisioned?
... View more
Labels:
- Labels:
-
Apache NiFi
07-20-2016
02:33 PM
There are three directories to be on NFS mount. Conf directory should not take much space. I checked retry directory and it was empty. Will it be automatically cleaned up? I checked graph storage and I had under 1 mb since I have only a few feeds. Will it keep growing as I add more feeds? How to get a good estimation on this? https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_data_governance/content/ch_appendix_data_gov_config_ha.html
*.retry.recorder.path: This location should be an NFS-mounted directory that is owned by Falcon, and with permissions set to 755. *.falcon.graph.storage.directory: This location should also be an NFS-mounted directory that is owned by Falcon, and with permissions set to 755. Falcon conf directory: The default location of this directory is <falcon_home>/conf, which is symbolically linked to/etc/falcon/conf. This directory must point to an NFS-mounted directory to ensure that the changes made on the primary Falcon server are populated to the back-up server.
... View more
Labels:
- Labels:
-
Apache Falcon
07-11-2016
06:35 PM
I don't think it works. Tried with /hdfsDataLocation only, retention doesn't work.
... View more
07-11-2016
04:14 PM
https://falcon.apache.org/EntitySpecification.html#Feed_Specification
<cluster name="test-cluster">
<validity start="2012-07-20T03:00Z" end="2099-07-16T00:00Z"/>
<retention limit="days(10)" action="delete"/>
<sla slaLow="hours(3)" slaHigh="hours(4)"/>
<locations>
<location type="data" path="/hdfsDataLocation/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
<location type="stats" path="/projects/falcon/clicksStats" />
<location type="meta" path="/projects/falcon/clicksMetaData" />
</locations>
</cluster> For location as data, do we have to specify the date format? Could I just have a general location as /hdfsDataLocation?
... View more
Labels:
- Labels:
-
Apache Falcon
07-11-2016
03:28 PM
Thanks Sunile. The replication Oozie WF is defined in target cluster, and WF will be run by RMs in target cluster, so there should be no problem then.
... View more
07-11-2016
02:32 PM
Here is the reference: https://falcon.apache.org/FalconNativeScheduler.html I am on HDP 2.3.2 which has Falcon 0.6. Is the version ready for Falcon native scheduler? If not, how about HDP 2.4.2? Currently native scheduler doesn't support Oracle. Will this be supported soon?
... View more
Labels:
- Labels:
-
Apache Falcon
07-07-2016
02:57 PM
For replication across the clusters, Falcon uses pull mode which Ooize/RM on target cluster should be used. If the RM specified is DOWN, not standby, do we have any problem for Falcon?
... View more
07-06-2016
07:45 PM
Where could I find a complete guide on Falcon configuration for HA services on both source and target clusters? So for NN HA, the solution is clear - define both clusters in hdfs-site.xml in source and target clusters. How about RM HA? With both RM1 and RM2 as active and passive, should I just pick one for source cluster such as in the following: https://community.hortonworks.com/questions/9416/falcon-with-ha-resource-manager.html <interface type="execute" endpoint="RM1:8050" version="2.2.0" /> What should I do for remote cluster RM? Just pick one? How do we define "hive.metastore.kerberos.principal" and "hive.metastore.uris" for multiple Hive metastore instances? How do we define for multiple oozie instances?
... View more
Labels:
- Labels:
-
Apache Falcon
06-27-2016
02:24 PM
Is there a Hive policy that we can enable a user to read schema only without accessing the data? Since Ranger plugin is in HS2 only, could Hive metastore be a choice?
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Ranger
05-25-2016
06:26 PM
1 Kudo
My cluster NN HA was implemented w/o securing the access to ZK. I am following the document to enable SASL: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/hdfs_configuration.html Here is the existing Znode used by Hadoop NN for HA. I was ablet to setACL on /hadoop-ha to sasl:nn:rwcda, but haadmin failover command failed. [zk: zkserver1:2181(CONNECTED) 5] getAcl /hadoop-ha 'world,'anyone : cdrwa If I delete the old one, will NN automatically create a new Znode? I also tried to do "hdfs zkfc -formatZK" to create a new one, but the command was not using SASL.
... View more
Labels:
- Labels:
-
Apache Hadoop
05-17-2016
07:11 PM
That should work. My original thought is to grant a user the ability to kill his own job. It is OK to promote him to the admin of the Yarn queue.
... View more
05-16-2016
09:44 PM
When a user submits a query via Beeline, HS2 in turn runs the query with user hive as doas is set to false. How could a user kill the job? It should be a command via beeline to ask HS2 to kill on behalf of a user. Snapshot: XYZ $ yarn application -kill
application_1461543063850_12340 16/05/16 11:56:13 INFO impl.TimelineClientImpl: Timeline service
address: ... 16/05/16 11:56:13 INFO client.RMProxy: Connecting to ResourceManager at ... Killing application application_1461543063850_12340 Exception in thread "main"
org.apache.hadoop.yarn.exceptions.YarnException:
java.security.AccessControlException: User XYZ cannot perform operation
MODIFY_APP on application_1461543063850_12340
... View more
Labels:
- Labels:
-
Apache Hive
05-13-2016
09:04 PM
1 Kudo
Here is what I found for YARN and HDFS to access ZK using SASL. YARN configuration makes perfect sense. However, is there something missing for HDFS such as configuring its own hdfs_jaas.conf? How about Hive? https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/securing_zookeeper_access.html
... View more
Labels: