How to create a dynamic sitemap for your website in ASP.NET Core

.NET

About Sitemaps

A sitemap is a file that lists all of the pages on a website, along with additional information about each page. Sitemaps are primarily used to help search engines index a website more effectively, but they can also be useful for website owners and users. This information may include the last time the page was updated, the frequency with which it is updated, and its priority relative to other pages on the site. Sitemaps are typically written in XML format, although other formats may be used as well.

How do sitemaps work?

Sitemaps work by providing search engines with a comprehensive list of all the pages on a website. When a search engine crawls a website, it uses the information in the sitemap to identify all the pages on the site and determine how frequently they are updated. This can help ensure that all of the site's pages are properly indexed and ranked in search results.

To create a sitemap, website owners typically use a sitemap generator tool. These tools can automatically crawl a website and generate a sitemap in XML format. Once the sitemap has been created, it can be submitted to search engines through their respective webmaster tools. This alerts the search engine to the existence of the sitemap and helps ensure that all of the site's pages are properly indexed.

Create a dynamic sitemap - Endpoint

[HttpGet("sitemap.xml")]
public async Task GetSiteMap()
{
    var pages = await _apiClient.GetWebsitePages();
	var siteMapXml = GetSiteMapXml(pages);

    Response.Headers.ContentType = "application/xml; charset=UTF-8";
	await Response.WriteAsync(siteMapXml);
}

This endpoint is implementing an HTTP GET method with the URL path /sitemap.xml. When a client makes an HTTP GET request to this endpoint, it returns an XML file representing the sitemap of a website.
The method is marked as async and returns a Task. The method name is GetSiteMap(). It does not take any parameters.
The method starts by making an asynchronous call to the _apiClient.GetWebsitePages() method to retrieve a list of pages from the website. This method is presumably defined elsewhere in the code and returns a collection of website pages.
After obtaining the list of pages, the method calls the GetSiteMapXml() method to generate an XML string representing the sitemap. The GetSiteMapXml() method is presumably defined elsewhere in the code and returns an XML string.
The method sets the Content-Type header of the HTTP response to "application/xml; charset=UTF-8" to indicate that the response is an XML file with the UTF-8 character encoding.
Finally, the method writes the XML string to the response body using the Response.WriteAsync() method. The Response object is a property of the ControllerBase class, which this method is presumably a member of. The WriteAsync() method asynchronously writes a string to the response body stream.

Create a dynamic sitemap - Prepare encoders for a XML based sitemap

public class Utf8StringWriter : StringWriter
{
	public override Encoding Encoding
	{
		get { return new UTF8Encoding(false); }
	}
}

We need to define a custom class Utf8StringWriter that inherits from the StringWriter class. The StringWriter class is used to write text to a string.
The Utf8StringWriter class overrides the Encoding property of the StringWriter class. The Encoding property specifies the character encoding used by the writer. In this case, the overridden Encoding property returns a new instance of the UTF8Encoding class with the false parameter passed to its constructor.
The UTF8Encoding class is a built-in class in .NET that represents the UTF-8 character encoding. The false parameter passed to the constructor specifies that the encoder should not emit a byte order mark (BOM) at the beginning of the encoded output.

Create a dynamic sitemap - Build the sitemap

private string GetSiteMapXml(IEnumerable<string> pages)
{
    var baseUrl = "https://example-domain.com";

	using (var sw = new Utf8StringWriter())
	{
		using (var xml = XmlWriter.Create(sw, new XmlWriterSettings { Indent = true }))
		{
			xml.WriteStartDocument();
			xml.WriteStartElement("urlset", "http://www.sitemaps.org/schemas/sitemap/0.9");
			xml.WriteAttributeString("xmlns", "image", null, "http://www.google.com/schemas/sitemap-image/1.1");

			xml.WriteStartElement("url");
			xml.WriteElementString("loc", $"{baseUrl}");
			xml.WriteEndElement();

			foreach (var page in pages)
			{
				xml.WriteStartElement("url");
				xml.WriteElementString("loc", $"{baseUrl}/{page}");
				xml.WriteEndElement();
			}

			xml.WriteEndElement();
		}

		return sw.ToString();
	}
}

The GetSiteMapXml() method takes an IEnumerable<string> parameter named pages, which is a collection of page URLs on the website.
The method starts by defining a base URL for the website. This is the root URL of the website that will be used as the prefix for all the page URLs in the sitemap.
The method then creates a Utf8StringWriter object to generate the XML sitemap. The Utf8StringWriter class is a custom class that inherits from the StringWriter class, as explained above.
The method uses the XmlWriter class to generate the XML content. It creates an instance of the XmlWriter class using the Create() method, passing in the Utf8StringWriter object and an instance of XmlWriterSettings to specify the settings for the XML writer. The XmlWriterSettings is set to enable indentation of the XML elements.
The method then starts writing the XML document by calling the WriteStartDocument() method. It then writes the urlset element, which is the root element of the sitemap. The xmlns:image attribute specifies the schema for the images in the sitemap.
The method then writes a url element for the root page of the website by calling the WriteStartElement() method and WriteEndElement() method of the XmlWriter class.
After that, the method writes a url element for each page in the pages collection by looping through the collection using a foreach loop. For each page, it writes a loc element containing the full URL of the page, which is the concatenation of the baseUrl and the page URL.
Finally, the method closes the urlset element and returns the XML string representation of the sitemap by calling the ToString() method of the Utf8StringWriter object.

Conclusions

In conclusion, sitemaps are a useful tool for website owners and managers who want to ensure that their sites are properly indexed by search engines. By providing a comprehensive list of all the pages on a website, sitemaps can help search engines more effectively crawl and index a site, which can lead to higher rankings in search results. Additionally, sitemaps can also be useful for users, particularly on large or complex websites, by providing a clear and structured overview of all the site's pages. Overall, sitemaps are a valuable tool for anyone involved in website management or search engine optimization.