According to an online article in the Washington Post
, leading news organizations and publishers have launched a revision in the 13-year-old technology for how search engines index and display web sites.
At present, major search engines like Google and Yahoo voluntarily respect a given website's wishes as described in a file called "robots.txt." The crawler (a search engines' indexing software) looks for this particular file on a website.
Although it was claimed that "fair use" provision of copyright laws applied, news publishers have complained and even sued Google for expanding search engine services that can include scanning printed books.
New proposed extensions for access to websites, known as Automated Content Access Protocol (ACAP) has, to some degree, grown out of such disputes. The Associated Press is one of many organizations that have joined ACAP, whose new rules allow a site to block indexing of specific directories, the entire site, or individual websites.
"ACAP was born, in part at least, against a growing backdrop of mistrust", said Gavin O'Reilly, president of the World Association of Newspapers. AP's chief executive Tom Curley says that the news cooperative spends millions of dollars annually covering the world, where its employees often risk their lives, making technologies like ACAP important in protecting AP's original news reports from being distributed without permission.
For a link to the complete article and a summary of the story, see Slashdot