Slides for talk given at IWMW 1999 held at Goldsmiths College on 7-9 September 1999. http://www.ukoln.ac.uk/web-focus/events/workshops/webmaster-sep1999/materials/dsullivan/

1. Getting Your Web Site Listed
Danny Sullivan
Editor, Search Engine Watch
http://searchenginewatch.com/

2. Session’s Goal
 Cover the best way to increase the
percentage of your web site listed with
crawler-based search engines
 Discuss importance of submitting to
human-powered directories
 Not covering issues of improving ranking!

3. Types of Search Services
 “Search engines” crawl the web and read
your web pages
 “Directories” like Yahoo depend on
human-provided information to categorize
sites
 Most search services are a hybrid of both,
but more dominant in one area

4. How Search Engines Work
 Read pages they find on the web
 Store text in an “index”
 When you search, they look for pages
with matching text
 Other factors involved in “ranking” those
pages, such as “link popularity”

5. The Major Search Engines
 Northern Light (big, but low traffic)
 FAST Search (big, but even less traffic)
 AltaVista (big and popular)
 Inktomi (HotBot, UK Max)
 Google (Netscape Search)
 Excite (WebCrawler)
 Go (Infoseek)
 Lycos (links at end)

6. Submitting To Search Engines
 Search engines should find you naturally,
but submitting helps speed the process
and can increase your representation
 Look for Add URL link at bottom of home
page
 Submit your home page and a few key
“section” pages
 Turnaround from a few days to 2 months

7. Deep Crawlers
 AltaVista, Inktomi, Northern Light will add
the most, usually within a month
 Excite, Go (Infoseek) will gather a fair
amount; Lycos gathers little
 Index sizes are going up, but the web is
outpacing them…nor is size everything
 Here are more actions to help even the
odds…

8. “Deep” Submit
 A “deep” submit is directly submitting pages
from “inside” the web site – can help improve
the odds these will get listed.
 At Go, you can email hundreds of URLs.
Consider doing this.
 At HotBot/Inktomi, you can submit up to 50
pages per day. Possibly worth doing.
 At AltaVista, you can submit up to 5 pages per
day. Probably not worth the effort.
 Elsewhere, not worth doing a “deep” submit.

9. Big Site? Split It Up
 Expect search engines to max out at
around 500 pages from any particular site
 Increase representation by subdividing
large sites logically into subdomains
 Search engines will crawl each subsite to
more depth
 Here’s an example...

10. Subdomains vs. Subdirectories
I nstead of Do this
gold.ac.uk/science/ science.gold.ac.uk
gold.ac.uk/english/ english.gold.ac.uk
gold.ac.uk/admin/ admin.gold.ac.uk

11. I Was Framed
 Don't use them. Period.
 If you do use them, search engines will
have difficulty crawling your site.

12. Dynamic Roadblocks
 Dynamic delivery systems that use ? symbols in
the URL string prevent search engines from
getting to your pages
 http://www.nike.com/ObjectBuilder/ObjectBuilder.iwx?
ProcessName=IndexPage&Section_Id=17200&
NewApplication=t
 Eliminate the ? symbol, and your life will be rosy
 Look for workarounds, such as Apache rewrite
or Cold Fusion alternatives
 Before you move to a dynamic delivery system,
check out any potential problems.

13. How Directories Work
 Editors find sites, describe them, put them
in a category
 Site owners can also submit to be listed
 A short description represents the entire
web site
 Usually has secondary results from a
crawler-based search engine

14. The Major Directories
 Yahoo
 The Open Directory
 (Netscape, Lycos, AOL Search, others)
 LookSmart
 UK Plus
 Snap

15. Submitting To Directories
 Directories probably won't find you or may list
you badly unless you submit
 Find the right category (more in a moment), then
use Add URL link at top or bottom of page
 Write down who submitted (and email address),
when submitted, which category submitted to
and other details
 You’ll need this info for the inevitable resubmission
attempt – it will save you time.

16. Submitting To Directories
 Take your time and submit to these right
 Write 3 descriptions: 15, 20 and 25 words long,
which incorporate your key terms
 Search for the most important term you want to
be found for and submit to first category that's
listed which seems appropriate for your site
 Be sure to note the contact name and email
address you provided on the submit form
 If you don't get in, keep trying

17. Subdomain Advantage
 Directories tend not to list subsections of
a web site.
 In contrast, they do tend to see
subdomains as independent web sites
deserving their own listings
 So, another reason to go with subdomains
over subdirectories

18. Other Tips / Misc
 Under represented? Get in touch!
 Need Search Engine Links? See:
http://searchenginewatch.com/links/