Night Crawler

China has reportedly developed a system called "night crawler" that can "locate and block" websites located in China (by IP address range) that have not registered with the Ministry of Information Industry (MII). It is difficult to hypothesize from news reports about China's actual technical capabilties. I've written about the exaggerations regarding China's Internet filtering (and also about those that underestimate it) in the past and will likely do so again as such misperceptions continue to unfortunetly be reported as fact. Anyway, here's my speculation about the "night crawler".


The article states that "night crawler":

• can locate and block access to websites that have not yet been registered with authorities


• is a real-time monitoring system that operates automatically


• will only target websites that have Mainland Chinese IP addresses

The "night crawler" likely scans the various IP address ranges assigned to China looking for webservers and then checks to see if domains are registered with the MII to that IP address (if there is not a domain registered, it could be flagged for further investigation). The other method would be to to DNS lookups on massive lists of domain names to get the IP address and check if it is in China's IP address ranges and if it is, check to see if it is registered or not.

Dealing with that many IP addresses & domain names would require "automation" and in this case "real time" likely means that it is always running through its lists/ip ranges, not that it would instantly know if a new unregistered domain appeared in Chinese IP ranges.

As far as the blocking goes, there are a few possible scenarios. The "night crawler" itself likely does not actually block anything, but it may feed its lists of unregistered domains and IP's to a system that does block. China Internet filtering is conducted by routers at/near the Internet backbone. The domains and IP's found by night crawler could be added to the Access Control Lists which are then sent to the various routers. However, this does not fit well with China's filtering system. If China's filtering is at the backbone level then the internal routing would work just fine. If an unregistered site is hosted in Shanghai and the user is in Beijing the request would not have to travel through the backbone, the filtering would be irrelevant. It would be possible to implment filtering on regional routers, but in my view this would be unlikely.

What is more likely is that with the cooperation with the various ISP's the authories would shutdown unregistered sites (ISP removal, account suspension etc...).

As noted in the article:

Websites with IP addresses not in Mainland China's IP address block are not required to register with Mainland authorities.

So, if someone in China registered a domain and hosted it outside of China it would not be affected by the "night crawler".