PublishSense, the book · Chapter 1 of 21 · free to read

Directories, Spiders and the First Tricks

For the first few years of the web, the best search engine in the world was two graduate students with a list.

In early 1994, Jerry Yang and David Filo, electrical engineering students at Stanford who were supposed to be finishing their dissertations, started keeping a list of websites they liked. They sorted it into categories. They called it "Jerry and David's Guide to the World Wide Web," which is the least marketable name in the history of technology, and by April they had renamed it Yahoo. Every site on the list had been looked at by a person and filed by a person. If you wanted your business included, you submitted it and waited for an editor to decide whether you belonged under Business and Economy, then Companies, then Health. Ranking, in the sense we mean it now, did not exist. You were in the directory or you were not.

That sounds quaint. It was also, in one respect, the most honest search product ever built, because a human being had read the page before recommending it. Hold that thought. The rest of this book is about the thirty years it took machines to get back there.

Counting words

The directory could not keep up. The web was growing faster than editors could read it, and a handful of people were building programs to read it instead. Matthew Gray's World Wide Web Wanderer had crawled the web to measure it in 1993. WebCrawler, launched in April 1994, was the first engine to index the full text of every page it found, so you could search for any word on a page and not just its title. Lycos followed that summer out of Carnegie Mellon. Infoseek and Excite arrived. Then, in December 1995, Digital Equipment Corporation launched AltaVista, which was fast, indexed nearly everything, and for a few years was the closest thing the web had to what Google would become.

How did these engines decide which page to show first? They counted. If you searched for "chiropractor," the engine looked for pages that contained the word, and it preferred pages that contained it more often, earlier, in the title, in the headings, and in a special tag hidden in the page's code called the meta keywords tag, which was designed so a site owner could tell the engine what the page was about.

Read that sentence again, because every trick of the next decade is inside it. The engine had no way to tell whether a page was good. It could only tell whether a page matched. And it let the page describe itself.

The first tricks, and why they were rational

If you were a business owner in 1997 and someone told you the engine counted the word "chiropractor," what would you do? You would use the word "chiropractor" more. That was not cheating. That was reading the instructions.

So keyword density became a discipline. People calculated the percentage of words on a page that were the target phrase and argued about the ideal number the way lifters argue about sets and reps. The meta keywords tag, meant to hold a few descriptive terms, swelled to hundreds of words, including competitors' brand names and, on the more ambitious sites, terms with nothing to do with the business at all, because traffic was traffic. When the visible text got embarrassing, the words went invisible: white text on a white background, text in a one-pixel font, text pushed off the edge of the screen. The page read fine to a human and shouted to the machine.

Then came doorway pages: dozens of near-identical pages, each built around one keyword variation, each existing only to catch a search and bounce the visitor somewhere else. And cloaking, where the server showed one page to the crawler and a different page to the person. If the engine was going to trust the page's description of itself, why not describe it as whatever the engine wanted to hear?

Here is the thing I want you to notice. None of this was done by criminals. It was done by webmasters and small business owners and the first generation of people who called themselves search engine optimizers, a phrase that appeared around 1997, the same year Danny Sullivan started Search Engine Watch to explain the engines to the people trying to please them. They were following the rules of the game as the game existed. The engine counted words; they gave it words. My old roommate ate a pound of bacon a day because the research said fat was fine, and he was mocked for it, and he turned out to be closer to right than the food pyramid. The keyword stuffers were his opposite: they did exactly what the experts said, and it worked, right up until it didn't.

The market arrives

While the engines were counting words, somebody noticed that the words were worth money. In 1998, Bill Gross's GoTo.com let advertisers bid for position in its results, openly, by the click. It was the first honest version of the market that keyword stuffing had been running in secret: if you wanted to be first for "New Jersey chiropractor," you paid for it, and the price was public. GoTo became Overture, Overture powered the paid results on Yahoo and others, and the idea it proved, that attention has a price and the price can be auctioned, became the engine of the largest advertising business in history. Keep it in mind. Every time a free ranking signal has been discovered since, somebody has tried to sell it, and the selling is usually what gets the signal discounted.

At the same time, the engines themselves lost interest in search. Excite, Lycos and the rest decided they were "portals," home pages with news and weather and email, where search was one box among twenty. The one company whose founders thought search was the whole point was still working out of a garage. That inattention is the reason the next chapter exists.

The question that was thirty years early

One more engine from that era deserves a paragraph, because it made a promise the rest of this book is about. In 1997, Ask Jeeves launched with a butler on the home page and a pitch that sounded like science fiction: don't type keywords, ask a question in plain English, and Jeeves will answer it. Behind the butler was a team of editors matching common questions to hand-picked pages, which is to say it was Yahoo's directory with better manners, and it could not scale any better than Yahoo's directory could. The butler was eventually retired.

But the promise was right. It was just early. A search engine that answers the question instead of listing pages that contain the words is exactly what Google's AI Mode is, and we will get there in Chapter 3. What Jeeves lacked was not the idea. It was a machine that could read.

Why it stopped working

Keyword counting stopped for the simplest reason a strategy ever stops: the engine's job was to satisfy the searcher, and the tricks made the searcher unhappy.

Picture the AltaVista user in 1998 who types "New Jersey chiropractor" and gets a page whose visible text is normal but whose hidden text lists four hundred towns. Or a doorway page that flashes for a second and dumps them on a home page about something else. The engine's results were being decided by whoever was most willing to lie to it. Users noticed, and drifted. The engines noticed the users drifting, and began to discount the very signals they had told everyone to optimize. Stuffed meta tags were ignored. Hidden text became a reason to drop a page rather than a reason to rank it. Google, when it arrived, never used the meta keywords tag for ranking at all, and in 2009 said so publicly for anyone who had not gotten the memo.

There was a second reason, and it matters more for where this book is going. Counting words could tell you a page was about chiropractors. It could not tell you whether the page was written by one, whether the information was current, whether anyone had found it useful, or whether the business behind it existed. A different kind of signal was needed, one that came from outside the page, that the page's author could not simply type in. Two other Stanford students were about to find it.

What the directory got right

Before we leave 1994, one more thing about Yahoo's list, because it is the point of this chapter and, quietly, of this book.

The directory worked, for as long as it could scale, because a human had read the page and decided it belonged. Everything the engines built afterward was an attempt to approximate that decision with math: first by counting words, then by counting links, then by modeling language, and now by having a machine read the page and answer the question itself. Thirty years of engineering, all of it aimed at one target: what would a knowledgeable person say this page is worth?

That is why gaming the algorithm has always been a short-term trade. You are not competing against a rule. You are competing against an ever-improving imitation of a reader. Every trick that works is a gap between the imitation and the reader, and every update closes a gap.

Which leaves you with the question this chapter exists to ask: if the engine is spending billions of dollars a year trying to become a good reader, what is the one strategy that never has to change?

The lesson for an automated blog: a system that counts what the engine counts is one update from worthless. Build it to satisfy the reader the engine is imitating, and the counting takes care of itself.