ASP.NET Core: Download from URL with WebClient

This article uses ASP.NET Core 3.1, but it works equally as well with .NET 5, .NET 6 and .NET 8. If using the older .NET Framework, see this article, though, not much has changed.

See this article if wanting to use HttpClient to download any datatype from a URL.

Using WebClient.DownloadFile() is a very efficient way to download a file and save it somewhere to be processed. Notice that a cookie is being sent to the server.

using (System.Net.WebClient wc = new System.Net.WebClient())
{
	wc.Headers.Add("Cookie: Authentication=user"); // NOTE: add a cookie header to the request
	try
	{
		string filename = System.Guid.NewGuid().ToString("N") + ".html"; // NOTE: use globally unique identifier for file name to avoid naming conflicts
		wc.DownloadFile("http://www.prowaretech.com/", filename); // NOTE: could add a file extension here
		// NOTE: do something with file
	}
	catch (System.Exception ex)
	{
		// NOTE: check exception object for the error
	}
}

WebClient.DownloadString() is a less efficient use of server memory, but it works well for downloading JSON.

using (System.Net.WebClient wc = new System.Net.WebClient())
{
	try
	{
		wc.Headers["Accept"] = "application/json"; // NOTE: accept JSON code (optional)
		string strJson = wc.DownloadString("http://www.prowaretech.com/json/some-file.json");
		// NOTE: do something with "strJson"
	}
	catch (System.Exception ex)
	{
		// NOTE: check exception object for the error
	}
}

WebClient.UploadString() is used to submit a POST request and download the results (as JSON in this case).

using (System.Net.WebClient wc = new System.Net.WebClient())
{
	try
	{
		wc.Headers["Accept"] = "application/json"; // NOTE: accept JSON code (optional)
		wc.Headers["User-Agent"] = "Custom User-Agent"; // NOTE: set the user-agent header (optional)
		string postBody = "field1=data1&field2=data2&field3=data3"; // NOTE: special characters in the data must be escaped
		string strResult = wc.UploadString("http://www.prowaretech.com/some/endpoint", postBody);
		// NOTE: do something with "strResult"
	}
	catch (System.Exception ex)
	{
		// NOTE: check exception object for the error
	}
}

WebClient.DownloadData() is another less efficient use of server memory. It downloads a file/endpoint/url into a byte array.

using (System.Net.WebClient wc = new System.Net.WebClient())
{
	try
	{
		byte[] data = wc.DownloadData("http://www.prowaretech.com/file.html");
		if (wc.ResponseHeaders != null)
		{
			string? contentType = wc.ResponseHeaders["Content-Type"];
			// NOTE: do something with data and contentType (Content-Type should BEGIN with "text/html" in this case)
		}
		else
		{
			// NOTE: do something with data
		}
	}
	catch (System.Exception ex)
	{
		// NOTE: check exception object for the error
	}
}

WebClient.OpenRead() is a less efficient use of server memory if reading the whole stream into memory. Use StreamReader.Read() to be conservative with memory particularly if working with large files on a server.

using (System.Net.WebClient wc = new System.Net.WebClient())
{
	try
	{
		using (System.IO.Stream stream = wc.OpenRead("http://www.prowaretech.com/"))
		{
			using (System.IO.StreamReader reader = new System.IO.StreamReader(stream))
			{
				string str = reader.ReadToEnd(); // NOTE: could read one character at a time with reader.Read() which is more efficient with regard to memory usage
				// NOTE: do something with str
			}
		}
	}
	catch (System.Exception ex)
	{
		// NOTE: check exception object for the error
	}
}

Comment