Degrading Transparency: Comparing Google, Yahoo and Microsoft

Google, Yahoo! and Microsoft all maintain versions of their search engines for the Chinese market that censor political content. One of the key issues that emerged concerned transparency. In 2006, all three search engines, following Google's lead, introduced a message that informed user when the results of their searches were censored. The presence of a mechanism of notification is a critical component of transparency. This notification informs users that their search results have been censored and indicates, to a certain degree, the reason (often unspecified "local law") why based on what the user searched for. The message appeared only when the user's results were censored and thus it was possible to connect the censorship to specific keywords or websites.

By 2008 the level of transparency has decreased. While Google's censorship notification has remained essentially the same as it was in 2006, Yahoo! and Microsoft have altered the way in which users are notified of censorship. Yahoo! has put its censorship message at the bottom of every page regardless of whether results are censored or not, in effect de-linking the censorship notification from the results. Microsoft has removed the text completely and buried the censorship notification with a separate "help" page. These developments represent a significant degrading of transparency and accountability.

By removing or hiding the placement of the censorship message, which is vague to begin with, users may be unaware that their results have been censored and by de-linking the censor message from what the user actually searched for the topics and websites that are censored remain hidden from the user. The de-linking of the censorship message from the search results impacts the ability to determine what precise sites and "key words" are being censored.

The presence and placement of a censorship notification, along with the specificity of its content and its connection to the results, is an integral component of transparency. The specificity of the reason why content has been removed is an important component that is lacking in the case of China. In other cases, Google has cited specific laws, such as the DMCA, and other legal documents with which they must comply and reported the information, to some degree, to Chilling Effects.org. Yahoo maintains a list of sites its censored for copyright violations. However, in the case of censored political content in the case China nothing other than a reference to "local law" has been provided.

The presence of a notification that is directly connected to the results (notification appears only when content is actually removed in relation to what the user searches for) positively impacts the ability to accurately identify censored website and restricted keywords. When such notifications are either absent or disconnected from the results (for example, a notification that appears on every page regardless of whether results are censored or not) the ability to determine censored sites with a high degree of confidence diminishes as sites may simply not be indexed by the search engine. Therefore the notification is critical not only for informing users but also for the monitoring process.

June 26, 2006
Engine	Presence	Placement	Specificity	Connection	Screenshot
Google	Yes	High Notification is placed under results	Low Results removed to comply with local law	Yes Notification only appears when results are censored	screenshot
Yahoo	Yes	High* Notification is placed under results	Low Results removed	Yes*	screenshot
Microsoft	Yes	High Notification is placed under results	Low Results removed, link to "help" page that mentions local law	Yes	screenshot

* Yahoo China's web crawlers operate from within China, behind the GFW, therefore sites that are blocked by China are not indexed by Yahoo (and thus do not need to be censored by Yahoo) leaving only sites that are either not blocked by China or are indexed during periods when there is variation in the capacity of China's filtering system to actually be censored by Yahoo. The behaviour documented here refers to sites indexed by Yahoo but subsequently censored, not sites that are not indexed by Yahoo at all.

January 25, 2008
Engine	Presence	Placement	Specificity	Connection	Screenshot
Google	Yes	High Notification is placed under results	Low Mentions "local law"	Yes Notification only appears when results are censored	screenshot
Yahoo	Yes	Medium Notification is placed at the bottom of every page	Low Mentions "local law"	No	screenshot
Microsoft	Yes**	Low A link to a separate "help" page which contains a link to section that contains the notification	Low A link to a separate “help” page which contains a link to section that mentions "local law"	No	screenshot

** There is no notification on the actual page that results the search results. The user must click a "help" page and then navigate to yet another section that state that results may be removed in compliance with local law. the notification mentions pornography as a possible reason, no mention is made of political content.

Presence: The presence of a form of notification that informs users that results may be censored.
Placement: The location of the censorship notification message, particularly its placement in relation to the results.
Specificity: The extent to which users are informed about specific laws, orders and/or regulations leading to censored results.
Connection: Notification appears only when content is actually removed in relation to what the user searches for making it possible to determine which specific web sites and keywords have actually been censored.

(The versions of the search engines tested are the specific version for China. Google (www.google.cn) and Microsoft (www.live.com/?mkt=zh-cn) have their servers located outside of China and are tested directly while Yahoo's (www.yahoo.cn) servers are hosted in China and are tested from inside China. This is necessary to test the search engines without interference from China's filtering system.)

This is the start of an effort to more systematically monitor transparency over time so I am asking for feedback. Is this information useful? In what ways can it be improved?

Canada, Copyright & Filtering

The Globe and Mail has picked up the story but suggests that filtering for copyright reasons is unlikely:

"Canadian ISPs have been defiant," said Mark Tauschek, a senior networks and telecom research analyst with Info-Tech Research Group. "They have refused to co-operate. They have said the Internet is an open network and they aren't going to police the content that goes across it ... unless there is criminality involved, they won't cough up information or necessarily block content."

What's interesting here is "unless there is criminality involved." See, the major ISP's already filter URLs that contain images of child abuse and may soon be expanding into other issue areas. The filtering technology is in place, it can be extended further. The issue here is that ISP's don't want to be in a position of judging content, they don't want to rule on whether something is criminal or not, but should that responsibility be delegated to some legitimate authority they don't seem to have any problem with filtering. It seems that filtering for reasons of copyright is not likely to be happening anytime soon. Still, it is an issue worth paying close attention to, as mission creep generally follows the implementation of filtering. It is also worth following closely due to the fact that other countries, particularly the U.S. and Europe are also moving in the direction of filtering for copyright reasons.

Using a technique similar to that of the GFW of China, Comcast is interfering with bittorrent and other traffic. Other ISP's in the U.S. are also considering implementing filtering. An ISP in Denmark, TDC, blocked allofmp3.com to comply with a legal ruling. There have been proposals to implement filtering for copyright reasons in Belgium, Norway, France and other European countries. While the EFF reports that proposed Amendments put before the EU's Committee on Culture and Education, which included ISP filtering, have been withdrawn or voted down they caution that there may still be attempts to introduce such language before the actual Parliamentary vote.

Will 2008 become the "Year of the Filters"?

Canada: Copyright Lobby Wants Filtering?

In a meeting with the editorial board of the Ottawa Citizen the Canadian Copyright Lobby indicated at several times that they support ISP filtering. Part of the reasoning is based on the belief that consumers say "if you made it impossible for me, if you stopped me, I would just get on with my life and yeah I would return to the market place and I'd buy things." (28:26) As Prof. Michael Geist notes:

Henderson cites with approval several initiatives to move toward ISP filtering of content, pointing to a French report, comments from the UK that such legislation could be forthcoming, and the AT&T negotiations in the U.S.  Later in the conversation, the group is asked what their dream legislation would look like.  The first response?  ISP liability, with the respondent pointing to Belgium as an example of an ideal model ("the file sharing issue will go away there as ISPs take down people").  Last summer, a Belgian court ordered an ISP to install filtering software to identify and block copyrighted content (the decision is currently being appealed).

Some other interesting notes from the conversation include the persistent attempts by the Copyright Lobby to highlight the fact that they do not like the changes they want made to the Copyright Act to be referred to or compared to the U.S. DMCA despite the fact that a version of the DMCA is in fact what they want. To accomplish this they continually refer to WIPO, arguing that Canada should just implement WIPO.

There's an interesting exchange concerning Fair Use rights in which a question is asked concerning how someone would use a 8 second clip of a half-hour show on a blog if the clip is encrypted (DRM protected). The response was a somewhat hostile attack on the person who asked the question concerning an assumption that the Government will somehow outlaw fair dealing which is quite odd since they are explicitly lobbying for an anti-circumvention clause which would outlaw fair dealing in the case presented in the question. It is too bad the questioner did not pursue this line of question more avidly.

As Geist points out, the principles that advocates of fair Copyright want are not necessarily at odds with interpretations of WIPO. WIPO provides for Limitations and Exceptions and even has a document that discusses many such limitations. It is clear that simply ignoring the consequences of implementing new measures, such as anti-circumvention, will leave many unanswered questions regarding Fair Use/Fair Dealing which is why Fair Copyright for Canada wants explicit protections put in place.

Excluding Iran

A few months ago The Register published a story which noted that Yahoo! (png) and Microsoft (png) had removed Iran as an option in their country lists used when signing up for an account. (On a separate but interesting note Yahoo! has an option for "Iraq-Saudi Arabia Neutral Zone".) Google still has Iran as an option for creating accounts, but does block GoogleEarth downloads from users in Iran (png).

Another interesting note in the article concerns Skype:

On 30 October, the Tehran correspondent of Netherlands newspaper NRC Handelsblad reported (in Dutch) that his paid Skype account had been cancelled. An email from the VoIP outfit said its financial services provider had been forced to stop taking payments from Iran.

This story reminded me of when Sun started to block downloads of Java using geolocation technology to users in Iran. Back in 2004/2005 Sun was explicit about it and used notices saying "The Java(tm) cannot be downloaded to your machine. You are located in an embargoed country." (Some old screen shots here (automatic installer) and here (direct manual download).) Sun has now replaced the earlier messages with a generic "Your download transaction cannot be approved. Contact Customer Service." (png)

I concur with The Register:

Whether Yahoo! and Microsoft's apparent action is the result of an over-zealous compliance lawyer or not, the effect on US interests of denying ordinary Iranians access to free international communications is questionable at best.

It is ironic that excluding Iranians from using such services is exactly what the Iranian government is criticized for. Take the case of flickr, the photo-sharing service which has been bought by Yahoo!, since it is often blocked by Iranian ISP's, Iranians need to use censorship circumvention methods to access the service, but now they are not permitted to sign up for a flickr account (they can, of course, circumvent this restriction too).

I'm not a lawyer, so I'm just putting this out here to inspire an lawyers who may wish to explore this but it seems to me that in some cases it may have to do with the presence of encryption technology which would fall under the U.S. export controls on encryption rather than under the U.S. embargo on Iran. In terms of the provision of Internet services there is a clear exception for "information":

The receipt or transmission of postal, telegraphic, telephonic or other personal communications, which does not involve the transfer of anything of value, between the United States and Iran is authorized. The exportation from the United States to Iran of information and informational materials, whether commercial or otherwise, regardless of format or medium of transmission, and any transaction incident to such exportation is authorized.

But in "Guidance on Internet Connectivity (Iran)" there is some confusing statements. The document is about the provision of "Internet access" also referred to as "Internet connectivity services" but begins to start using the word "service" on its own in relation to "goods, technology or software." However, one can be exempt "provided that the main purpose is to benefit the people of Iran through increased access to information."

In the end it is Iranian citizens who are left out, unable to use popular tools and services increase our ability to communicate, create and express ourselves online, not because the Iranian government is censoring them but because the of the U.S. embargo.