About ScipioTheElder

ScipioTheElder · ‎10-27-2016

Is there any better way to pull HDFS data into Kafka? I could use a flume exec source and kafka sink, but any alternative?

ScipioTheElder · ‎10-17-2016

Sunile. For daemons such as NM and RM, I can hard-code JMS port number. However, how to set different JMX port number on a number of JVM containers running on the same slave node? For example, I can configure 18 container per node, how could I set the port number to 8001-8018? Maybe I can use ZK, or some kind of script.

ScipioTheElder · ‎10-17-2016

My problem is how to set different JMX ports on the containers running on the same slave node.

ScipioTheElder · ‎10-14-2016

@Timothy Spann Tim, Thanks for the info. I can pull daemon JMX for sure since they are running all the time. However, I am more interested on per job based container. How to query YARN restful api to get a list of container for a job?

ScipioTheElder · ‎10-14-2016

@mhendricks I have the exact same interest on monitoring all the containers of the job via JMX. However, my question is how to get the list of containers for a job assuming we can turn ON JMX for yarn containers?

ScipioTheElder · ‎10-14-2016

Great article. Is there any way to pull individual yarn container JMX? It would be great for me to see what is going on when I start a job and monitor the job containers at real time.

ScipioTheElder · ‎09-07-2016

The more, the better, to some extent.

ScipioTheElder · ‎09-06-2016

Should 3 be sufficient for a 3 rack cluster with one ZK per rack? Does increasing ZK nodes to 5 make sense? My understanding is that for fault tolerant 3 ZKs are good enough. Having 2 ZK nodes on the same rack doesn't increase HA.

ScipioTheElder · ‎08-29-2016

This is NOT about HiveServer2, but only on Hive metastore. Do we have only two choices on Hive metastore property hive.security.metastore.authorization.manager? StorageBasedAuthorizationProvider DefaultHiveMetastoreAuthorizationProvider

ScipioTheElder · ‎07-28-2016

# H2 Settings nifi.database.directory=./database_repository nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE Is something to be added here?

ScipioTheElder · ‎07-28-2016

I installed NIFI in two separate location nifi and nifi2 with different ports - one without kerberized ZK and one with kerberized ZK. However, both web UI (8080 and 9080) see the same nifi flow graphs. How could they share the same internal database such as H2?

ScipioTheElder · ‎07-27-2016

Yes. However, we use adkeytab from Centrify. Here is a blog on this (he used adkeytab to add spn and kutil to merge with no detailed steps). https://www.centrify.com/activate/cmz3c60glnnmc1tnlwnv0srcpkfct5p9/

ScipioTheElder · ‎07-27-2016

http://community.centrify.com/t5/Centrify-Insight/Difference-between-adkeytab-and-ktutil/td-p/14281 Did you use ktutil to merge?

ScipioTheElder · ‎07-26-2016

Thanks Sunile. We got that already. This is for adding two principals in the same key tab using AD, not MIT kdc.

ScipioTheElder · ‎07-26-2016

Anyone knows how to do with AD adkeytab command? https://community.hortonworks.com/articles/35019/oozie-ha-configuration-with-kerberos.html step 8 requires to add more than more principals in the same keytab. [root@kk2 ~]# klist -ket /etc/security/keytabs/spnego.service.keytab Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab KVNO TimestampPrincipal ----------------------------------------------------------------------------- 305/03/1616:44:05 HTTP/kk2.hwxblr.com@HWX.COM (aes256-cts-hmac-sha1-96) 305/03/1616:44:05 HTTP/kk2.hwxblr.com@HWX.COM (aes128-cts-hmac-sha1-96) 305/03/1616:44:05 HTTP/kk2.hwxblr.com@HWX.COM (des3-cbc-sha1) 305/03/1616:44:05 HTTP/kk2.hwxblr.com@HWX.COM (arcfour-hmac) 505/03/1616:44:11 HTTP/test1-ambari-server.hwxblr.com@HWX.COM (aes256-cts-hmac-sha1-96) 505/03/1616:44:11 HTTP/test1-ambari-server.hwxblr.com@HWX.COM (aes128-cts-hmac-sha1-96) 505/03/1616:44:11 HTTP/test1-ambari-server.hwxblr.com@HWX.COM (des3-cbc-sha1) 505/03/1616:44:11 HTTP/test1-ambari-server.hwxblr.com@HWX.COM (arcfour-hmac)

ScipioTheElder · ‎07-25-2016

Do you mean that each nifi user should have his/her own instance? There could be too many. Also is there any way to specify multiple instances per host?

ScipioTheElder · ‎07-20-2016

Currently HDF 1.2 is running as user root. I am using HDP 2.3.2 so NIFI is not part of Ambari controlled list. Since it is run as root, getHDFS or putHDFS processor uses a user keytab. How could I prevent a user A to create a processor to use user B's keytab? In HDP 2.4 or later, do we have NIFI service user ID provisioned?

ScipioTheElder · ‎07-20-2016

There are three directories to be on NFS mount. Conf directory should not take much space. I checked retry directory and it was empty. Will it be automatically cleaned up? I checked graph storage and I had under 1 mb since I have only a few feeds. Will it keep growing as I add more feeds? How to get a good estimation on this? https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_data_governance/content/ch_appendix_data_gov_config_ha.html *.retry.recorder.path: This location should be an NFS-mounted directory that is owned by Falcon, and with permissions set to 755. *.falcon.graph.storage.directory: This location should also be an NFS-mounted directory that is owned by Falcon, and with permissions set to 755. Falcon conf directory: The default location of this directory is <falcon_home>/conf, which is symbolically linked to/etc/falcon/conf. This directory must point to an NFS-mounted directory to ensure that the changes made on the primary Falcon server are populated to the back-up server.

ScipioTheElder · ‎07-11-2016

I don't think it works. Tried with /hdfsDataLocation only, retention doesn't work.

ScipioTheElder · ‎07-11-2016

https://falcon.apache.org/EntitySpecification.html#Feed_Specification <cluster name="test-cluster"> <validity start="2012-07-20T03:00Z" end="2099-07-16T00:00Z"/> <retention limit="days(10)" action="delete"/> <sla slaLow="hours(3)" slaHigh="hours(4)"/> <locations> <location type="data" path="/hdfsDataLocation/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/> <location type="stats" path="/projects/falcon/clicksStats" /> <location type="meta" path="/projects/falcon/clicksMetaData" /> </locations> </cluster> For location as data, do we have to specify the date format? Could I just have a general location as /hdfsDataLocation?

ScipioTheElder · ‎07-11-2016

Thanks Sunile. The replication Oozie WF is defined in target cluster, and WF will be run by RMs in target cluster, so there should be no problem then.

ScipioTheElder · ‎07-11-2016

Here is the reference: https://falcon.apache.org/FalconNativeScheduler.html I am on HDP 2.3.2 which has Falcon 0.6. Is the version ready for Falcon native scheduler? If not, how about HDP 2.4.2? Currently native scheduler doesn't support Oracle. Will this be supported soon?

ScipioTheElder · ‎07-07-2016

For replication across the clusters, Falcon uses pull mode which Ooize/RM on target cluster should be used. If the RM specified is DOWN, not standby, do we have any problem for Falcon?

ScipioTheElder · ‎07-06-2016

Where could I find a complete guide on Falcon configuration for HA services on both source and target clusters? So for NN HA, the solution is clear - define both clusters in hdfs-site.xml in source and target clusters. How about RM HA? With both RM1 and RM2 as active and passive, should I just pick one for source cluster such as in the following: https://community.hortonworks.com/questions/9416/falcon-with-ha-resource-manager.html <interface type="execute" endpoint="RM1:8050" version="2.2.0" /> What should I do for remote cluster RM? Just pick one? How do we define "hive.metastore.kerberos.principal" and "hive.metastore.uris" for multiple Hive metastore instances? How do we define for multiple oozie instances?

ScipioTheElder · ‎06-30-2016

Is there any example with Kerberos enabled cluster?

ScipioTheElder · ‎06-27-2016

Is there a Hive policy that we can enable a user to read schema only without accessing the data? Since Ranger plugin is in HS2 only, could Hive metastore be a choice?

ScipioTheElder · ‎05-25-2016

My cluster NN HA was implemented w/o securing the access to ZK. I am following the document to enable SASL: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/hdfs_configuration.html Here is the existing Znode used by Hadoop NN for HA. I was ablet to setACL on /hadoop-ha to sasl:nn:rwcda, but haadmin failover command failed. [zk: zkserver1:2181(CONNECTED) 5] getAcl /hadoop-ha 'world,'anyone : cdrwa If I delete the old one, will NN automatically create a new Znode? I also tried to do "hdfs zkfc -formatZK" to create a new one, but the command was not using SASL.

ScipioTheElder · ‎05-17-2016

That should work. My original thought is to grant a user the ability to kill his own job. It is OK to promote him to the admin of the Yarn queue.

ScipioTheElder · ‎05-16-2016

When a user submits a query via Beeline, HS2 in turn runs the query with user hive as doas is set to false. How could a user kill the job? It should be a command via beeline to ask HS2 to kill on behalf of a user. Snapshot: XYZ $ yarn application -kill application_1461543063850_12340 16/05/16 11:56:13 INFO impl.TimelineClientImpl: Timeline service address: ... 16/05/16 11:56:13 INFO client.RMProxy: Connecting to ResourceManager at ... Killing application application_1461543063850_12340 Exception in thread "main" org.apache.hadoop.yarn.exceptions.YarnException: java.security.AccessControlException: User XYZ cannot perform operation MODIFY_APP on application_1461543063850_12340

ScipioTheElder · ‎05-13-2016

Here is what I found for YARN and HDFS to access ZK using SASL. YARN configuration makes perfect sense. However, is there something missing for HDFS such as configuring its own hdfs_jaas.conf? How about Hive? https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/securing_zookeeper_access.html

Online	Offline
Last Visited	‎08-22-2019 12:57 PM

Date Registered	‎05-16-2016 12:18 PM
Last Visited	‎08-22-2019 12:57 PM
Total Messages Posted	76
Total Kudos Received	44

Re: Sanbox 2.3.2 hive beeline error

How to ingest HDFS data into Kafka

Re: Enable JMX metrics on hadoop using jmxterm

Re: Access Running Job Metrics via JMX

Re: Access Running Job Metrics via JMX

Re: Access Running Job Metrics via JMX

Re: Enable JMX metrics on hadoop using jmxterm

Re: Number of Zookeepers in a 3 rack cluster with ...

Number of Zookeepers in a 3 rack cluster with data...

Hive metastore authorization

Re: Why multiple nifi instances on the same host s...

Why multiple nifi instances on the same host share...

Re: AD how to add multiple principals in the same ...

Re: AD how to add multiple principals in the same ...

Re: AD how to add multiple principals in the same ...

AD how to add multiple principals in the same keyt...

Re: NIFI service user ID

NIFI service user ID

Falcon HA shared NFS drive sizing

Re: Falcon Feed File System Storage format

Falcon Feed File System Storage format

Re: Falcon support on Hadoop HA services

Falcon native scheduler to replace Oozie

Re: Falcon support on Hadoop HA services

Falcon support on Hadoop HA services

Re: Hadoop Data Pipeline using Falcon

Ranger Hive policy for schema only

Switch NameNode HA Zookeeper access from no secur...

Re: How to kill Hive job with individual user ID w...

How to kill Hive job with individual user ID when ...

Zookeeper kerberos authentication