I ran into this timeout before. I found I had ranger enabled on kafka and the api would time out. can you verify you have ranger disabled on kafka topic, or give correct permission to atlas user. ... View more

I highly recommend you use Apache NiFi instead of flume for most if not all data movement into and out of hadoop. For your use case, there is a prebuilt nifi template to push tweets to hive and solr (for searching and trending) https://community.hortonworks.com/articles/1282/sample-hdfnifi-flow-to-push-tweets-into-solrbanana.html If you want further analysis (ie most used words), this can be done several ways. 1. for real time, use spark streaming with nifi. microbatch your counts 2. batch, run hive sql from nifi 3. batch, call hive script from nifi to calculate analysis every x internal 4. batch, setup oozie job to calculate analysis every internal (may be kicked off from nifi as well). so you have options. hope that helps. ... View more

You can install anaconda but will need to set the correct path during job execution PYSPARK_PYTHON=/opt/anaconda/bin/python spark-submit spark-job.py ... View more

@Sampat Budankayala can you add more details on what you are asking? I assume you might be asking on how to move data from Sharepoint to HDFS. I would use apache NiFi and pull data via sharepoint rest api (https://msdn.microsoft.com/en-us/library/office/jj860569.aspx). if it is bulk load option then a java client may be best. this processor does not exist yet but should be very easy https://github.com/OfficeDev/Office-365-SDK-for-Java ... View more

@hello hadoop I would check here to determine if option is available http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-upgrade/content/upgrading_hdp_stack.html From the docs above the option is available on 2.3. Also please check the pre reqs prior to upgrading http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-upgrade/content/upgrading_HDP_prerequisites.html ... View more