Your Peace of Mind is our Commitment

Contact Us English Recent Articles

Strange Behaviour of the Soghu Web Spider


Requests from an IP address associated with the Soghu web spider had the following odd characteristics:

The anomaly was noticed because the log analysis software used (analog) reports a "corrupt line" when the user agent is missing.


The anomaly was noted in webserver logs for on 25th February 2010. Analog reported 57 corrupt lines in the logfile, all requests from, with the user agent field "-". The URLs requested were:


The requests were spread across the day, starting at 00:00:30 +0800, and the last request at 21:39:40 +0800. On examination, the IP address had also made 5 other requests, where the user agent was given as:

Sogou web spider/4.0(+

The same user agent made 6 other requests, from These 11 requests with a user agent were all for URLs that exist on the site.

Both IP addresses are registered to China Telecom's Beijing province network.


The fact that the URLs requested in the anomalous requests actually exist on a different host seems more than coincidence. The most likely explanation is that a single instance of Sogou's web spider suffered corruption that both prevented the sending of the user agent name and caused the ".hk" to be dropped from the hostname it was contacting.