Thoughts on HTML Sitemaps / Hallway Pages

[Kookjay]

Hi guys,

What is everyones thought about using HTML sitemap / 'Hallway pages' in the footer of a site?

I have read a bit of mixed information about them. Some say that they are a black hat tactic and you can see some negative effects, but then some say there is noting wrong with them - you are helping the crawlers etc etc

Any thought would be appreciated!

 

[Sutra]

Never heard of an HTML sitemap being called a "hallway page" before. Just looked it up after reading this post.

Anyway, I use an HTML sitemap. Shows all the posts and pages on my site, which gives users a visual way of quickly finding everything on the site. Plus by having it makes sure that things don't accidentally get buried 3+ clicks deep within the architecture - thus easier for google to find. Not such a big deal if you have only a few articles, but is a bigger deal when you have multiple categories and 1000+ posts.

You mentioned the sitemap being in the footer. I assume you mean a link to the sitemap in the footer (which is what I do), and not an actual sitemap in the footer.

 

[Ryuzaki]

It's definitely not black hat to link your website to your website's own pages.

I like this term "Hallway page." I'd not heard that, but the distinction against "Doorway page" is immediately obvious. Doorway pages are bad.

Let's be frank. When people talk about HTML sitemaps, it's 100% about Google and nothing to do with users. No user want's to read a list of post titles. Maybe you decide to spruce it up with featured images and an excerpt. Maybe you decide to paginate them so you aren't showing 500 per page, but 25 instead. Congratulations, we've discovered the category page.

All of these problems are already solved, as seen above with category pages and below:

Most people, when they talk about HTML Sitemaps, are talking about nothing more than an ordered or unordered list of posts and pages. All of them from a site. Can this be useful to users? Maybe if you have 30-50 posts, but themed categories are still better. If you have 1000+, it's worthless to users. What's actually helpful in that case is a search function.

You may think having an HTML Sitemap linked sitewide is good for crawling. All you're really doing is creating a page that will suck up a ton of page rank juice, redistribute a lot it around your site but not all, and end up with so many links Google won't crawl them all on one page, so you have to paginate. Now you're bloating up the index with pages that are nothing but links and losing page rank to these pages. This isn't good.

The reason people do this is because they're concerned with crawlability and crawl budget. 99% of sites don't need to worry about crawl budget. And for crawlability, we already have solutions to this too. They're backlinks to deep pages, interlinking between your pages, and submitting your sitemap to Google so they regularly revisit every single URL on your site, especially if you update them.

You don't need to help crawlers in this fashion anyways. What happens is your HTML sitemap page is seen in the footer of the first page crawled, and then marked as "crawled" and not revisited on every single page that the crawler pulls up. They will time-stamp sitewide links and only crawl them once. So you're not really sending spiders to that page as much as you'd think.

The other thing is that a crawler isn't going to jump through every single one of those links just because it landed on the page. Not the first time, not every time, etc. Big spiders know how to save resources. They might do (I'm guessing) up to 50 URLs in one shot, but not 1,000. They might schedule a return at a later time to look at the rest of the URLs. But again, this is what an XML sitemap is for.

My philosophy is, if you're doing this just for spiders and not humans, then use an XML sitemap, put it in your robots.txt, submit it to Search Console, and you're good. The rest is a waste of your time and a waste of time for any human that ends up in it.

Another option is to create some kind of feature like "Recommended Articles" or whatever at the bottom, and link backwards within a category. Link to the previous 4 or 5 articles in that category, statically instead of letting it get randomized with PHP or whatever, so it's always the same ones. And then on your next article, link back to the previous 4 or 5. And in this way, you'll have every post being internally linked many times. And in the future, you can always link to the 3 most previous then jump way back in time to the 100th most previous just to shoot the spiders back there.

But again, XML Sitemaps cover every single bit of this for you. Categories are already like HTML sitemaps but for more organized. Thanks for reading my incoherent rambling. I'm here every day!