About ScipioTheElder

ScipioTheElder · ‎10-27-2016

Is there any better way to pull HDFS data into Kafka? I could use a flume exec source and kafka sink, but any alternative?

ScipioTheElder · ‎10-17-2016

Sunile. For daemons such as NM and RM, I can hard-code JMS port number. However, how to set different JMX port number on a number of JVM containers running on the same slave node? For example, I can configure 18 container per node, how could I set the port number to 8001-8018? Maybe I can use ZK, or some kind of script.

ScipioTheElder · ‎10-17-2016

My problem is how to set different JMX ports on the containers running on the same slave node.

ScipioTheElder · ‎10-14-2016

@Timothy Spann Tim, Thanks for the info. I can pull daemon JMX for sure since they are running all the time. However, I am more interested on per job based container. How to query YARN restful api to get a list of container for a job?

ScipioTheElder · ‎10-14-2016

@mhendricks I have the exact same interest on monitoring all the containers of the job via JMX. However, my question is how to get the list of containers for a job assuming we can turn ON JMX for yarn containers?

ScipioTheElder · ‎10-14-2016

Great article. Is there any way to pull individual yarn container JMX? It would be great for me to see what is going on when I start a job and monitor the job containers at real time.

ScipioTheElder · ‎09-07-2016

The more, the better, to some extent.

ScipioTheElder · ‎09-06-2016

Should 3 be sufficient for a 3 rack cluster with one ZK per rack? Does increasing ZK nodes to 5 make sense? My understanding is that for fault tolerant 3 ZKs are good enough. Having 2 ZK nodes on the same rack doesn't increase HA.

ScipioTheElder · ‎08-29-2016

This is NOT about HiveServer2, but only on Hive metastore. Do we have only two choices on Hive metastore property hive.security.metastore.authorization.manager? StorageBasedAuthorizationProvider DefaultHiveMetastoreAuthorizationProvider

ScipioTheElder · ‎08-15-2016

I am on Atlas 0.5 with Kerberos secured cluster. Is there any way to enable a user to log in the dashboard by changing the rules on atlas.http.authentication.kerberos.name.rules. RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*// Also is there any way to do user authorization to decide who can do what?

ScipioTheElder · ‎07-28-2016

# H2 Settings nifi.database.directory=./database_repository nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE Is something to be added here?

ScipioTheElder · ‎07-28-2016

I installed NIFI in two separate location nifi and nifi2 with different ports - one without kerberized ZK and one with kerberized ZK. However, both web UI (8080 and 9080) see the same nifi flow graphs. How could they share the same internal database such as H2?

ScipioTheElder · ‎07-27-2016

Yes. However, we use adkeytab from Centrify. Here is a blog on this (he used adkeytab to add spn and kutil to merge with no detailed steps). https://www.centrify.com/activate/cmz3c60glnnmc1tnlwnv0srcpkfct5p9/

ScipioTheElder · ‎07-27-2016

http://community.centrify.com/t5/Centrify-Insight/Difference-between-adkeytab-and-ktutil/td-p/14281 Did you use ktutil to merge?

ScipioTheElder · ‎07-26-2016

Thanks Sunile. We got that already. This is for adding two principals in the same key tab using AD, not MIT kdc.

ScipioTheElder · ‎07-26-2016

Anyone knows how to do with AD adkeytab command? https://community.hortonworks.com/articles/35019/oozie-ha-configuration-with-kerberos.html step 8 requires to add more than more principals in the same keytab. [root@kk2 ~]# klist -ket /etc/security/keytabs/spnego.service.keytab Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab KVNO TimestampPrincipal ----------------------------------------------------------------------------- 305/03/1616:44:05 HTTP/kk2.hwxblr.com@HWX.COM (aes256-cts-hmac-sha1-96) 305/03/1616:44:05 HTTP/kk2.hwxblr.com@HWX.COM (aes128-cts-hmac-sha1-96) 305/03/1616:44:05 HTTP/kk2.hwxblr.com@HWX.COM (des3-cbc-sha1) 305/03/1616:44:05 HTTP/kk2.hwxblr.com@HWX.COM (arcfour-hmac) 505/03/1616:44:11 HTTP/test1-ambari-server.hwxblr.com@HWX.COM (aes256-cts-hmac-sha1-96) 505/03/1616:44:11 HTTP/test1-ambari-server.hwxblr.com@HWX.COM (aes128-cts-hmac-sha1-96) 505/03/1616:44:11 HTTP/test1-ambari-server.hwxblr.com@HWX.COM (des3-cbc-sha1) 505/03/1616:44:11 HTTP/test1-ambari-server.hwxblr.com@HWX.COM (arcfour-hmac)

ScipioTheElder · ‎07-25-2016

Do you mean that each nifi user should have his/her own instance? There could be too many. Also is there any way to specify multiple instances per host?

ScipioTheElder · ‎07-20-2016

Currently HDF 1.2 is running as user root. I am using HDP 2.3.2 so NIFI is not part of Ambari controlled list. Since it is run as root, getHDFS or putHDFS processor uses a user keytab. How could I prevent a user A to create a processor to use user B's keytab? In HDP 2.4 or later, do we have NIFI service user ID provisioned?

ScipioTheElder · ‎07-20-2016

There are three directories to be on NFS mount. Conf directory should not take much space. I checked retry directory and it was empty. Will it be automatically cleaned up? I checked graph storage and I had under 1 mb since I have only a few feeds. Will it keep growing as I add more feeds? How to get a good estimation on this? https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_data_governance/content/ch_appendix_data_gov_config_ha.html *.retry.recorder.path: This location should be an NFS-mounted directory that is owned by Falcon, and with permissions set to 755. *.falcon.graph.storage.directory: This location should also be an NFS-mounted directory that is owned by Falcon, and with permissions set to 755. Falcon conf directory: The default location of this directory is <falcon_home>/conf, which is symbolically linked to/etc/falcon/conf. This directory must point to an NFS-mounted directory to ensure that the changes made on the primary Falcon server are populated to the back-up server.

ScipioTheElder · ‎07-18-2016

Is there any examples on using Kafka in Kerberos environment? I am using HDP 2.3.2 which ZK acl is set for user kafka only. How to set ACL properly to allow multiple users to pub/sub the topics? ZK-10] getAcl /kafka-acl 'sasl,'kafka : cdrwa 'sasl,'kafka : cdrwa

ScipioTheElder · ‎07-12-2016

I tested myself with a single cluster. Only saw two retention WF got created without any replication WF. So replication within a single cluster might have issues.

ScipioTheElder · ‎07-11-2016

@Sowmya Ramesh : could you confirm that we have to have dated partition format in director structure?

ScipioTheElder · ‎07-11-2016

I don't think it works. Tried with /hdfsDataLocation only, retention doesn't work.

ScipioTheElder · ‎07-11-2016

https://falcon.apache.org/EntitySpecification.html#Feed_Specification <cluster name="test-cluster"> <validity start="2012-07-20T03:00Z" end="2099-07-16T00:00Z"/> <retention limit="days(10)" action="delete"/> <sla slaLow="hours(3)" slaHigh="hours(4)"/> <locations> <location type="data" path="/hdfsDataLocation/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/> <location type="stats" path="/projects/falcon/clicksStats" /> <location type="meta" path="/projects/falcon/clicksMetaData" /> </locations> </cluster> For location as data, do we have to specify the date format? Could I just have a general location as /hdfsDataLocation?

ScipioTheElder · ‎07-11-2016

Thanks Sunile. The replication Oozie WF is defined in target cluster, and WF will be run by RMs in target cluster, so there should be no problem then.

ScipioTheElder · ‎07-11-2016

Here is the reference: https://falcon.apache.org/FalconNativeScheduler.html I am on HDP 2.3.2 which has Falcon 0.6. Is the version ready for Falcon native scheduler? If not, how about HDP 2.4.2? Currently native scheduler doesn't support Oracle. Will this be supported soon?

ScipioTheElder · ‎07-11-2016

I have a feed with both retention and replication definitions. Does Falcon create two Oozie workflows - one on source cluster for retention and another on target cluster for replication (pull mode)? If so, do these two workflows have the same name on source and target clusters? I tested with a single cluster where both cluster definitions point to the same cluster, only retention workflow works. I suspect the replication workflow is overwritten by the retention workflow in a single cluster. <?xml version="1.0" encoding="UTF-8"?> <feed description="Hourly Feed" name="testHourlyFeed" xmlns="uri:falcon:feed:0.1"> <tags>externalSystem=DataHub,classification=ETL</tags> <groups>ETL</groups> <frequency>minutes(5)</frequency> <timezone>GMT-06:00</timezone> <late-arrival cut-off="hours(1)"/> <clusters> <cluster name="labPrimary" type="source"> <validity start="2016-06-30T00:00Z" end="2016-08-01T00:00Z" /> <retention limit="hours(1)" action="delete"/> </cluster> <cluster name="labSecondary" type="target"> <validity start="2016-06-01T00:00Z" end="2016-08-01T00:00Z" /> <retention limit="days(7)" action="delete"/> <locations> <location type="data" path="/user/ambari-qa/data/repl-in/${YEAR}/${MONTH}/${DAY}/${HOUR}"/> </locations> </cluster> </clusters> <locations> <location type="data" path="/user/ambari-qa/data/in/${YEAR}/${MONTH}/${DAY}/${HOUR}"/> <location type="stats" path="/none"/> <location type="meta" path="/none"/> </locations> <ACL owner="ambari-qa" group="hdfs" permission="0755"/> <schema location="/none" provider="none"/> </feed>

ScipioTheElder · ‎07-08-2016

<?xml version="1.0" encoding="UTF-8"?> <feed description="Hourly Feed" name="testHourlyFeed" xmlns="uri:falcon:feed:0.1"> <tags>externalSystem=DataHub,classification=ETL</tags> <groups>ETL</groups> <frequency>minutes(5)</frequency> <timezone>GMT-06:00</timezone> <late-arrival cut-off="hours(1)"/> <clusters> <cluster name="labPrimary" type="source"> <validity start="2016-06-30T00:00Z" end="2016-08-01T00:00Z" /> <retention limit="hours(1)" action="delete"/> </cluster> <cluster name="labSecondary" type="target"> <validity start="2016-06-01T00:00Z" end="2016-08-01T00:00Z" /> <retention limit="days(7)" action="delete"/> <locations> <location type="data" path="/user/ambari-qa/data/repl-in/${YEAR}/${MONTH}/${DAY}/${HOUR}"/> </locations> </cluster> </clusters> <locations> <location type="data" path="/user/ambari-qa/data/in/${YEAR}/${MONTH}/${DAY}/${HOUR}"/> <location type="stats" path="/none"/> <location type="meta" path="/none"/> </locations> <ACL owner="ambari-qa" group="hdfs" permission="0755"/> <schema location="/none" provider="none"/> </feed>

ScipioTheElder · ‎07-08-2016

Both source and target clusters are the same cluster. I don't see any replication occurred, but retention seems to be working. One interesting thing is that the directory for the data source also gets deleted along with the file, such as /user/ambari-qa/data/repl-in/${YEAR}/${MONTH}/${DAY}/${HOUR} directory. <?xml version="1.0" encoding="UTF-8"?> <feed description="Hourly Feed" name="testHourlyFeed" xmlns="uri:falcon:feed:0.1"> <tags>externalSystem=DataHub,classification=ETL</tags> <groups>ETL</groups> <frequency>minutes(5)</frequency> <timezone>GMT-06:00</timezone> <late-arrival cut-off="hours(1)"/> <clusters> <cluster name="labPrimary" type="source"> <validity start="2016-06-30T00:00Z" end="2016-08-01T00:00Z" /> <retention limit="hours(1)" action="delete"/> </cluster> <cluster name="labSecondary" type="target"> <validity start="2016-06-01T00:00Z" end="2016-08-01T00:00Z" /> <retention limit="days(7)" action="delete"/> <locations> <location type="data" path="/user/ambari-qa/data/repl-in/${YEAR}/${MONTH}/${DAY}/${HOUR}"/> </locations> </cluster> </clusters> <locations> <location type="data" path="/user/ambari-qa/data/in/${YEAR}/${MONTH}/${DAY}/${HOUR}"/> <location type="stats" path="/none"/> <location type="meta" path="/none"/> </locations> <ACL owner="ambari-qa" group="hdfs" permission="0755"/> <schema location="/none" provider="none"/> </feed>

ScipioTheElder · ‎07-07-2016

For replication across the clusters, Falcon uses pull mode which Ooize/RM on target cluster should be used. If the RM specified is DOWN, not standby, do we have any problem for Falcon?

Online	Offline
Last Visited	‎08-22-2019 12:57 PM

Date Registered	‎05-16-2016 12:18 PM
Last Visited	‎08-22-2019 12:57 PM
Total Messages Posted	76
Total Kudos Received	44

Re: Sanbox 2.3.2 hive beeline error

How to ingest HDFS data into Kafka

Re: Enable JMX metrics on hadoop using jmxterm

Re: Access Running Job Metrics via JMX

Re: Access Running Job Metrics via JMX

Re: Access Running Job Metrics via JMX

Re: Enable JMX metrics on hadoop using jmxterm

Re: Number of Zookeepers in a 3 rack cluster with ...

Number of Zookeepers in a 3 rack cluster with data...

Hive metastore authorization

Atlas user authentication and user authorization

Re: Why multiple nifi instances on the same host s...

Why multiple nifi instances on the same host share...

Re: AD how to add multiple principals in the same ...

Re: AD how to add multiple principals in the same ...

Re: AD how to add multiple principals in the same ...

AD how to add multiple principals in the same keyt...

Re: NIFI service user ID

NIFI service user ID

Falcon HA shared NFS drive sizing

Kafka in a secured cluster

Re: Use Falcon only to implement hdfs data retenti...

Re: Use Falcon only to implement hdfs data retenti...

Re: Falcon Feed File System Storage format

Falcon Feed File System Storage format

Re: Falcon support on Hadoop HA services

Falcon native scheduler to replace Oozie

Falcon generated Oozie workflows for retention and...

Re: Falcon retention and replication feed in a sin...

Falcon retention and replication feed in a single ...

Re: Falcon support on Hadoop HA services