Support Questions
Find answers, ask questions, and share your expertise

Use MapReduce or Hadoop Streaming to Get Response of Urls in c#

Highlighted

Use MapReduce or Hadoop Streaming to Get Response of Urls in c#

New Contributor

I have following program which get more then 1,00,000 urls from database and use C# HttpWebRequest object to get Response and save into database.

To get response of 1,00,000 urls take long time to run in single server. I want divide this process into thread and want run in multiple server.

It is possible to use such type of code in MapReduce in C# with thread and TPL ?

using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading.Tasks;
using System.Runtime.Serialization.Json;
using System.Net;
using System.Runtime.Serialization;
using System.IO;


namespace Test
{
    class Program
    {
        static void Main(string[] args)
        {
            TestApp testapp = new TestApp();
            var urls = testapp.GetURls();


            //get string from urls


            foreach (var url in urls)
            {
                var responseString=testapp.GetURLString(url);


                //now save response into database
                testapp.saveResponseToDB(responseString);
            }
        }
    }


    public class TestApp
    {
        public void saveResponseToDB(string response)
        {
            //save into db
        }
        public List<string> GetURls()
        {
            var urls =new List<string>();


            //fetching more then 100000 urls from database.


            return urls;
        }


        public string GetURLString(string requestUrl)
        {
            HttpWebRequest request = WebRequest.Create(requestUrl) as HttpWebRequest;


            using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
            {
                StreamReader reader = new StreamReader(response.GetResponseStream());
                // Read the content.
                string responseFromServer = reader.ReadToEnd();
                // return the content.
                return responseFromServer;
            }
        }
    }
}

Don't have an account?