Sitemaps ping endpoint is going away
Google, Monday, June 26, 2023
The Sitemaps Protocol was introduced in 2005 to help search engines with the discovery of new URLs, and also to help with scheduling new crawls of already discovered URLs. It’s a wildly popular protocol that hasn’t changed for over 15 years. While the general idea is still useful, some aspects have become less practical in today’s internet.
To that end, we’re announcing deprecation of the sitemaps “ping” endpoint and providing additional recommendations for the use of the lastmod element.
Sitemap ping
The sitemap protocol defines an unauthenticated REST method for submitting sitemaps to search engines. Our internal studies—and also other search engines such as Bing—tell us that at this point these unauthenticated sitemap submissions are not very useful. In fact, in the case of Google Search, the vast majority of the submissions lead to spam. To wit, we’re deprecating our support for sitemaps ping and the endpoint will stop functioning in 6 months. You can still submit your sitemaps through robots.txt and Search Console, but the HTTP requests (“pings”) to the deprecated REST endpoint will result in a 404 error. Any existing code or plugins which use this endpoint will not cause problems for Google Search; you don’t need to make any changes (but using the endpoint will also not do anything useful).
The lastmod element
Over the years we’ve observed a varying level of usefulness of the lastmod element across the sites that provide it. This may have been the result of the kind of content that’s published, or perhaps the content management system, but nowadays lastmod is indeed useful in many cases and we’re using it as a signal for scheduling crawls to URLs that we previously discovered.
For the lastmod element to be useful, first it needs to be in a supported date format (which is documented on sitemaps.org); Search Console will tell you if it’s not once you submit your sitemap. Second, it needs to consistently match reality: if your page changed 7 years ago, but you’re telling us in the lastmod element that it changed yesterday, eventually we’re not going to believe you anymore when it comes to the last modified date of your pages.
You can use a lastmod element for all the pages in your sitemap, or just the ones you’re confident about. For instance, some site software may not be able to easily tell the last modification date of the homepage or a category page because it just aggregates the other pages on the site. In these cases it’s fine to leave out lastmod for those pages.
And when we say “last modification”, we actually mean “last significant modification”. If your CMS changed an insignificant piece of text in the sidebar or footer, you don’t have to update the lastmod value for that page. However if you changed the primary text, added or changed structured data, or updated some links, do update the lastmod value.
<urlset xmlns=“http://www.sitemaps.org/schemas/sitemap/0.9”>
<url>
<lastmod>2005-01-01</lastmod>
<loc>http://www.example.com/</loc>
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url>
</urlset>
Going on a small tangent, if you look at the xmlns attribute in the sitemap snippet, you’ll see that the URI is on HTTP, and not on HTTPS. This is working as intended: it’s a reference for parsers about the elements in the XML. Please don’t file more documentation feedback about this.
changefreq and priority
Google still doesn’t use the changefreq or priority elements at all. changefreq specifically is also conceptually overlapping with lastmod. The priority element is a heavily subjective field and based on our internal studies, it generally doesn’t accurately reflect the actual priority of a page relative to other pages on a site.
Want to read more about sitemaps? Check out our documentation, but also sitemaps.org, and if you want to just chat with us about sitemaps, you can find us in the Google Search Central forums and on Twitter.
(13)
