It is known as the Google dance, a delicate struggle between technicians at the world's largest internet search engine and the spin doctors who manipulate the worldwide web for commercial ends. Every day one group tries to prevent the other from abusing Google's index of more than 8bn web pages.
The stakes are high. As Britain's online shoppers spend £150m every day in the run-up to Christmas, the value of a high ranking on Google is potentially worth millions to retailers, website owners and criminals alike. Web search has become the focus for a legion of marketers over the past decade, and their tactics - known as search engine optimisation (SEO) - are the basis of a multibillion pound worldwide industry. Most are legitimate businesses, but some so-called "black hat" SEOs use unethical strategies to boost their clients.
Spoof site
To test the effectiveness of these tactics, the Guardian created a spoof site and tried to force it up Google's rankings. Over one week, a number of tricks - some similar to those used by black-hat firms - were used to successfully push it to the top.
The spoof site was set up to promote eco-friendly flip-flops, a bogus product promising zero harmful emissions. The simple page featured a disclaimer to make the nature of the experiment clear, and a picture of the goods. At the start of the experiment, there were more than 11,500 results for "eco-friendly flip-flops" on Google, and the spoof site did not feature. Within two days of creating the site, Google's spider - the program that explores the web - had discovered the site and included it in its main index, but it appeared within the lowest 100 pages. The first attempt to boost the ranking was a series of basic instructions intended to manipulate Google, including overloading the page with words that would improve the site's ranking, and adding invisible data intended to boost it even further. This had little effect, however, and the spoof site remained static in Google's index.
Another trick was then used to mimic black-hat behaviour. A second site was created which contained a large number of links to the first. Because Google rates the authority of a site partly by how many times they have been linked to, this ploy can makes a site appear popular. Within hours, the effect was apparent - the spoof site was now the top result in our test search, trumping the other 11,500 sites within days.
Our experiment was small scale and limited in scope, but in the real world the value of success is higher than ever. Black-hat companies offer a range of services at different prices, all aimed at unfairly manipulating search engines. One website found by the Guardian offered customers the chance to download a program to spam Google for $99 (£56), while another - posing as a legitimate SEO -charged several thousand pounds.
Commercial power Customers trust the results of search engines, which rely on advertising to generate profit and have much to gain from keeping the results clean and everything to lose if they fail. Almost a fifth of all visits to online shopping sites are the direct result of an internet search, according to data monitor Hitwise - and more than half of those come straight from Google. This places an astonishing level of commercial power in the hands of one company. Google says it takes this very seriously, but it is an almost impossible task given the amount of information concerned.
"There is a lot of ongoing algorithmic work to improve the relevancy of our results in general," said Marcus Sandberg, a Google engineer. "Many SEOs do a great job at helping site owners improve their content for users and search engines alike. But some do use methods that we consider manipulative."
Behind Google's legendary algorithm - an immense secret mathematical formula for ranking websites - are a series of well-known principles that can be faked, hoaxed or otherwise influenced.
"What's grey and what's not depends on your point of view," said Danny Sullivan, editor of the Search Engine Watch website. He said the relationship with SEOs is an awkward but vital one for search engines - and that some search manipulation is done by people "who don't know what they're doing". "They buy tools just like people buy guns who don't know what they're doing," he said.
Google made headlines last week after a $1bn (£560m) deal to buy a stake in internet service provider AOL. Reports suggesting the company could include more intrusive adverts on its site have angered those who believe its clean approach is integral to its success.
Five ways to get to the top
Status on Google is determined by a number of factors, all of which can be faked
Key words
Good practitioners will make sure sites contain clear information that is relevant to a user search. Others will use misleading but popular keywords - such as "Britney Spears" - to try to capitalise on somebody else's fame. Some even attempt to hide fake keywords on a page so that they can be read by search engines but not by people
Popularity
The more people that link to a site, the more popular it is in Google's mind. By carefully choosing who to link to and where to place those links, SEOs can push a target website up the rankings. Some shady operators even create a fake ecology of websites which all point at each other
Spam
Spamming is a tactic employed by unscrupulous SEOs, and attempts to raise profile and popularity by leaving fake messages pointing towards the target across thousands of other sites and weblogs. While unpopular with surfers, it often boosts the ranking of the site in question
Regular updates
Sites which seem new are often considered more important, because they are more likely to contain relevant information. Unscrupulous operators will often steal content from other pages to create the appearance of movement
Metadata
Each web page carries a selection of unseen information that tells other programs what its contents are. While most SEOs simply include correct information about a given page, crooked operators will use unrelated terms to try to direct unwitting surfers