Reasons your site may not be included: Your pages are dynamically generated. We are able to index dynamically generated pages. However, because our web crawler can easily overwhelm and crash sites serving dynamic content, we limit the amount of dynamic pages we index.
Google information for webmaster [google.com]

Another use for URL rewriting is to deal with “hotlinks”. Bob pointed me to a site nice site discussing hotlinks which are bascially links from other websites directly to your content, used for display on their page. I’ve had people do this in forums where they like a photo of mine and instead of linking to the particular page it’s hosted on and sending folks my way, they link directly to the image and use my bandwidth for their gain Here’s an example of someone doing that to me. It’s not a huge deal and thankfully I put the jdhodges.com URL on all of my images… Following the table is a new program called coldlink that is specifically targeted at preventing damage from hotlinks. First, here is another description of hotlinks:

…the practice of building web pages that contain unauthorized content links, known as a hotlink, direct link, or remote link to files hosted by another site. Notice that we said content links and not navigation links that lead to another site. Content links are file references that the browser fetches to draw the page such as images, style sheets, scripts or even complete web pages that are rendered within a frame. In other words, these are embedded content or embedded objects within an html page .

The result of hotlinking is that the offending site is able to present it’s pages without paying for the bandwidth needed to serve up the stolen content. The victim site ends up paying the bandwidth expense for serving up the files without gaining any page views.
Source: wordworx.com

IIS Rewrite is the granddady of rewrite filters and it is the first thorough implemenation that I came across. Now ISAPI_Rewrite appears to be taking the lead but they are still very comparable products. I would like to mention that I have corresponded personally with IIS Rewrite’s creator in the past and he was very helpful and responsive. I don’t have personal experience with ISAPI_Rewrite but many other users are atarted to praise the features it brings to the table.

Below is a table that I compiled attempting to compare the available products, I tried not to come to any incorrect conclusions but for some of the less well know rewriters out there you will want to do some more of your own research. If you don run across and errors or you have a new rewriter to add to the list please email me at jd_hodges_ii AT hotmail.com Finally, if you find this information useful and end up purchasing a filter, please let them know that you found out about them here. Thanks for reading!

ColdLink

I wasn’t aware of an IIS solution to the hotlinking problem, but Bob knew of one called ColdLink which claims to “puts hotlinks on ice” hehe, now that’s a nice slogan!

ColdLink Bandwidth Protection Software gives you instant relief when hotlinks from bandwidth bandits get you hot under the collar.

The ColdLink iis isapi filter or apache module transparently maps and rewrites protected urls dynamically in realtime within every page served from your site. Your site continues to work normally, even when the pages are linked from search engines or other sites. Users can even bookmark your pages for later return because the page will contain new valid links.
Source: coldlink.com

Update: For those wanting to check if their current anti-hotlinking measures are working, here is an online anti-hotlinking testing tool. Just enter the URL of a protected image, on their easy to use web form and hit “check now” Finally, here is an example of a hotlink protected image now go to the original source page of that image to view it in its original context(you may have to refresh your browser.) Pretty nifty huh?! I’m not sure what software they’re using but it works well

IIS Rewrite

What I’ve been using on my site for the past year, it has easily supported over a million rewrites without giving me an problems.

IISRewrite is a rule-based rewriting engine that allows a webmaster to manipulate URLs on the fly in IIS.

URLs are rewritten before IIS has handed over the request to be processed, so requests for HTML files, graphics, program files, and even entire directory structures can be rewritten before they are passed to ASP scripts for processing.

[. . . ]

IISRewrite is a stripped down implementation of Apache’s mod_rewrite modules for IIS. Webmasters who have used Apache’s mod_rewrite in the past will find that much of the configuration and functionality is the same.

OpUrl

ISAPI URL Rewrite

An ISAPI filter that provides powerful, integrated URL re-writing for IIS. OpUrl has many benefits, including helping to ensure search engines crawl even the dynamic parts of your site. Most search engine crawlers don’t index dynamic pages, e.g. page.asp?item=1 so OpUrl allows you to use static URLs instead. The functionality is very similar to Apache’s mod_rewrite

ISAPI Rewrite

ISAPI_Rewrite is a powerful URL manipulation engine based on regular expressions. It acts mostly like Apache’s mod_Rewrite, but is designed specifically for Microsoft’s Internet Information Server (IIS). ISAPI_Rewrite is an ISAPI filter written in pure C/C++ so it is extremely fast. ISAPI_Rewrite gives you the freedom to go beyond the standard URL schemes and develop your own scheme.

Here is what Joshua thought about ISAPI_Rewrite:

I’ve tried ISAPI_ReWrite (www.isapirewrite.com) and I like it. It seems comparable to IISReWrite, but here are some features I really like:
1. It’s free in it’s most basic form (one set of rules for every domain on the box). You have to buy it to set up rules for virtual domains, but it’s cheap.
2. The help documentation is really complete in regards to explaining the RegEx rules. I actually use the documentation as my daily RegEx reference.
3. Their support forums are decent. I got a couple answers within 24 hours.
4. * When you edit the config file, the changes are immediate * That means no having to restart anything when a rule changes. I know that IIS ReWrite requires a reboot of IIS every time you make a rule set change. The reason I like not having to restart anything is because I plan on making an online application to create rules on the fly.

Over on Web Master World forums another user also had good things to say about ISAPI_Rewrite.

ISAPI_Rewrite would be my suggestion. We’ve tested both IIS Rewrite and ISAPI. The ISAPI_Rewrite product outperforms the others due to its ease of configuration and a bunch of other little extras that come in handy. One of the biggest benefits we’ve found with ISAPI_Rewrite is that you don’t have to restart IIS each time you make a change to the .ini file

The ISAPI_Rewrite featureset sounds complete, and the documentation is thorough but not moreso than IIS_Rewrite.

URL Replacer

A new freeware rewriter that I just found out about, I don’t have any experience with it but if you’re looking for something to try on the cheap it might be your ticket.

The ISAPI filter replaces defined parts of URL from browser. It enables url to scripts (.asp, .cgi, .idc) with parameters look like static html pages or specify exact download filename generated by script.

Update 2006.March.28

LinkFreeze Description from the Helicon Tech website:

LinkFreeze is a fast and easy solution to optimize dynamic or database-driven website for search engines. It optimizes all dynamic links on your site removing all unwanted characters and making links look like static files. […] the main benefit of LinkFreeze is that you don’t need to change anything in your application to make it search engines friendly. LinkFreeze will modify all links directly before the server will send response to the client. So everybody – users, search engine crawlers or other robots will see the same nice statically looking links. And when somebody will navigate to this link LinkFreeze will intercept this request on the server and rewrite URL back to the dynamic form, so your applications will still thinking it is working with old dynamic links.

HotLink Blocker Description from the Helicon Tech website:

Hotlink Blocker is an IIS plug-in that will protect your web site from bad people stealing your traffic by directly linking to the content on your server. If you have an image collection, video or document archives, Flash games or any other downloadable content HotlinkBlocker is a right product for you. After installing HotlinkBlocker your site will continue working as before, all search engines and backward links will still be valid, all user’s favorites will remain working. But if someone tries to embed your stuff directly on their site, they will only see an error message or your site’s logo. You can save tons of traffic and thus the money. You can even turn these hotlinks into real visitors by redirecting hotlink request to your site’s homepage.

Updated 2003.Dec.05 with anti-hotlinking testing tool