Transact SQL Other Articles Software Reviews
Creating an ASP Search Engine for your Website
A guide to using The Website Utility to create an ASP powered server-side Search Engine for your website.
Why Websites need Search Facilities
Once a website grows beyond a couple of dozen pages then it can sometimes be difficult to create a site navigation scheme that allows users to quickly find exactly what they're looking for. One way to improve site navigation is to add a search facility to the website. Adding a search facility brings major benefits to a website, making it easier for users to find information as well as adding an additional method of navigating a website. Search facilities are generally well used, and will often appear within the top ten most requested pages on a website.
Search Engines Allow Visitors to Search your Content
One of the easiest ways to add a facility for searching the pages in your website is to link to search results for your website from one of the major search engines. Google and other major search engines allow you to do this. However, using this method it can be difficult to integrate the search results with the design of your website. It also carries the obvious risk of a website visitor leaving your website and not returning! Even worse, your website visitors may see an advert for a competitor on the search results page, and so go and do their business elsewhere!
Building Your Own Search Engine
There are a number of software solutions that allow you to put your own search engine on your website. These include server-side search solutions available such as Microsoft's Index Server or ht://Dig. Although they allow sophisticated search facilities to be created, they generally require a high level of technical knowledge to install and configure. Another complication is that your chosen Internet Service Provider or hosting company may not support the search software required, or you may not be able to install additional software onto the server hosting the website.
A useful alternative is to use a server-side solution that only makes use of ASP, and does not require any other components to be installed onto a web server.
The Website Utility Builds ASP Search Engines
The walkthrough below shows the steps involved in creating an ASP based search facility using The Website Utility:
Configuring The Website Utility to Produce ASP Search Engines
Note that if your website uses query strings then it is a good idea to tick the checkbox under Web Robot Settings called Use URL Query Strings. This will ensure that in the search results pages with different query strings will be treated as different search results pages. So for example www.mywebsite.net/news.php?ID=12 will link to a different news article from www.mywebsite.net/news.php?ID=21 and so The Website Utility will ensure they are indexed separately.
Running The Website Utility
Clicking on the Run button will start The Website Utility's web robot. This web robot start at a user specified page in the website and will automatically crawl all of the pages in that website. The Website Utility extracts all of the words from these pages, and finds the most relevant pages in the website for each word. Common English words (e.g. got, like, then) are removed, as are words of one or two characters. Word rankings depend on many factors, including their distribution through the entire website and their distribution in the content of a specific page.
Incorporating the ASP Search Facility into any Website
The Website Utility creates two ASP include files that can be used on the website's search results page:
Note that the data and code files are created in JScript, but they can easily be used from within ASP pages that use VBScript.
Pages are sorted in search results according to their ranking for the particular word or words being searched for. The ranking scale goes from 0 to 99. Rank is higher for pages that most closely match the search term. In general, searching for words that are common on the site will produce search results with a lower rank than very specific words that occur on only one or two pages.
The search facility also requires a search form and a search results page. The search form can either be put on a separate search page on the site, or the search form could be added to all of the pages in a website (e.g. in the top right hand corner of the website's navigation). The HTML code for a typical search form is shown below. The search form needs a text box called TWUQuery. The form should use either the GET or POST form submission method to submit to the search results page.
The search results page (i.e. searchresults.asp) needs to include references to the two ASP include files created by The Website Utility (TWUSearchData_JScript.asp and TWUSearchCode_JScript.asp):
The search results page can of course be customised to fit in with the existing design of your website. If you want to display the search terms the user was searching for, then use this ASP code (assuming VBScript is the default scripting language):
To display the search results, place this ASP code where you want the search results to appear:
The search results page also defines four ASP JScript variables that can be used to change the output:
A server-side ASP search engine is obviously going to have a larger performance overhead compared to using a dedicated search solution such as Microsoft's Index Server or SQL Server's full text indexing capabilities. The size of the TWUSearchData_JScript.asp include file will depend on the number of pages in the website indexed, and also the amount of content on each page in the website. It is also dependent upon the nature of the website itself - websites with pages about similar subjects will tend to require a smaller file than a website with pages about different subjects.
To create a search facility for medium to large websites of 1000+ pages, or to allow the simultaneous searching of several websites at once, then Microsoft's Indexing Service is recommended. The Indexing Service Companion may also be used to extend the functionality of Indexing Services.
The evaluation version of The Website Utility will allow you to determine whether the ASP Search Engines it creates are suitable for use on your own websites:
Sample Search Facilities
Purchase The Website Utility