How to check for broken links using Xenu?

Posted on July 4, 2008 at 5:41 am

Do you often have problems with broken links? Are you tired of seeing the 404 or “File or Directory Not Found” error page? Wether you are a web developer or a user that is irritated by a website with this broken link issue, this free tool is for you.

Ironically, the Wikipedia page for Xenu’s Link Sleuth has a broken link on its footnote section:

http://www.curriculum.edu.au/SCIS/connections/cnetw02/43xenu.htm

Written by a programmer that’s as fed up as you with broken links, Xenu’s Link Sleuth crawls the entire website and checks for broken links and then creates a summarized report of all the broken links found. The name Xenu is derived from a Scientology god which does bad things – he is like the Satan of the Scientology world – Tilman, the programmer, is an avid cult and Scientology critic so I think he used “Xenu” to mock the fanatic cult followers.

Although interesting and a good Digg topic we will depart discussing the sensational background of this software.

To start the link checker, open Xenu and then click File > New URL…

Enter the URL of the website that you want to check on the prompt. On this example, I will check the links coming from the project’s homepage: http://home.snafu.de/tilman/xenulink.html

If you need to check for external sites check the box otherwise make sure to uncheck it to avoid crawling external sites (it could take time and possibly slow down other servers if you crawl sites that you don’t intend to).

External sites means sites that are hosted on another domain. Also, you can treat external sites as “internal” by adding the beginning address on the list of “Consider URLs beginning with this as ‘internal’”. On the same manner, you can bypass URLs that you don’t want to crawl like forums and dynamic pages. After you are done setting the URL, click OK to start the crawler.

During the crawling process you will see the program checking each page along with the status, type of file, size and HTML title:

Depending on the speed of your connection and the number of files linked from the URL you entered above, it could take from a few minutes to an hour. The software crawls the web using a multi-threaded technique so it could check for multiple pages concurrently. The number of threads is adjustable using the Options panel.

Click OK once the crawling is done:

The final output is a very comprehensive report in HTML format with the following contents:

* Broken links, ordered by link
* Broken links, ordered by page
* List of redirected URLs
* List of valid URLs you can submit to a search engine
* Site Map of HTML pages with a Title
* Broken page-local links
* Orphan files
* Statistics for managers

The most important part of the report is the list of broken links so we can identify them:

Click the first item “Broken links, ordered by link” to see the list of broken links:

To interpret the report see below guide:

From a web developer point of view, this can be fixed by:

1. making the missing page available OR

2. removing the broken link from the pages that point to the missing page.

Given the case above and considering what is the best way to finish the job, looks like step 1 is more practical to implement since only one file will be affected instead of many files.

What if you are only the user and don’t have direct control of these files? Well, first you can try to find the “Contact Us” page or email the admin if you have their contact details. Quality websites usually fix the issue very quickly – they don’t want to have irritated readers.

Now, to get the overall picture of the whole site scroll to the bottom part of the report to see the aggregated result:

So, from the report I can see that out of 6299 URLs only 17 are not found or only .27%. Depending on your standard or level of discomfort towards broken links this could be interpreted as either good, bad or in the middle – don’t really matter.

This tool enables you to find quality shortcomings of websites that don’t check their pages for broken links. With a tool like Xenu’s Link Sleuth, link checking is only a few clicks away.

Ben Carigtan is a new contributor writer. A Software Engineer with more than 7 years of technical experience, he will be writing articles covering practical advices and tips for computer users.

Enjoyed this post? Subscribe to Online Tech Tips via RSS Feed or via Email and receive free daily productivity tips.

» Filed Under Cool Websites

Save this page	Stir it up on Mixx
Add to Reddit	Stumble this page

2 Responses to “How to check for broken links using Xenu?”

Siddharth said on : July 4th, 2008 at 11:28 am

This is quite cool
Firefox has got a bunch of cool plugins for web developers

Christina Scull said on : September 22nd, 2008 at 3:53 pm

Can you use Xenu to check only 1 folder on a website or des it only crawl the whole site? I can’t seem to get it to contain itself to one folder no matter what I do.