@pankaj chaturvedi inside your pig script do this: set exectype=tez; ... View more

@mark doutre difficult to provide recommendations without knowing the use case. One thing that stands out to me with ASR is the source data generating app will have to adhere to a schema. This ends up provide just another maintance issue on the app side. For me I rather have the data flow into NiFi and have it forks based on the type of data feed in. ... View more

@Sri Bandaru add your jars to .hiverc file. ... View more

@Benjamin Leonhardi did this work as expected? if we simply select one of the RMs in falcon, will it fail over automatically to secondary? I am trying to understand the impact if Falcon pointing to RM and that RM goes down. ... View more

Client, ApplicationMaster and NodeManager on RM failover When there are multiple RMs, the configuration (yarn-site.xml) used by clients and nodes is expected to list all the RMs. Clients, ApplicationMasters (AMs) and NodeManagers (NMs) try connecting to the RMs in a round-robin fashion until they hit the Active RM. If the Active goes down, they resume the round-robin polling until they hit the “new” Active. This default retry logic is implemented as org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider. You can override the logic by implementing org.apache.hadoop.yarn.client.RMFailoverProxyProvider and setting the value of yarn.client.failover-proxy-provider to the class name. ... View more

@Ahmad Debbas I have done this using storm to parse emails/pdfs using tika as documents land onto hdfs. You can use storm hdfs spout (info here). Once data is parsed, using another bolt to sink into solr. Pretty straight forward solution. NiFi is definitely a consideration. You will need a build a NiFi tiki processor. As each event is then run through processor --> parsed text--> into solr. this could work as well ... View more

@Kiran Jilla Are there any difference in pre-prod vs prod? db versions, services, kerberos, etc? ... View more