First published: 02nd January 2012
US security company Imperva has published a report on the effectiveness of anti-virus software based on their work with students from the Technion-Israel Institute of Technology. The team collected 82 samples of malware and used the VirusTotal website to test whether they were detected by 40 anti-virus products. Based on this, they concluded that initial detection rates were as low as 5%, and recommended that compliance rules should be eased, freeing up money for "more effective" security measures.
Yui Kee's Chief Consultant Allan Dyer took a different view and harshly criticised Imperva's study, saying, "I was surprised at the small sample set Imperva used - just 82 samples, collected from honey pots, google and hacker forums. Can this really reflect on effectiveness against the millions of malware samples known to exist?"
In comparison, AV-Test uses two test sets in its Protection tests:
- All malicious files they discovered in the last 6 - 8 weeks: around 100,000 – 150,000 files.
- Extremely widespread malicious files they discovered in the last 6 – 8 weeks: around 2,000 – 2,500 files.
Dyer continued, "A second surprise is that Imperva do not do their own testing, they threw the samples at VirusTotal. VirusTotal is a useful website, but they are quite explicit that it is unsuitable for product testing. Imperva takes the short form of VirusTotal's advice, 'not designed as a tool to perform antivirus comparative analyses', and counter it in the study's 'Limitations' section saying that they are not doing a comparison. Imperva ignore the longer advice, that details why VirusTotal is unsuitable for both comparative and effectiveness testing."
Dyer concluded, "Anti-virus testing is notoriously difficult, and competent researchers put a lot of work into making sure they use methodologies that will produce relevant, reliable results. Did Imperva?"
Updated: 07th January 2012
Imperva's study has generated a lot of discussion and criticism, including from David Harley on his blog, Max Eddy at PC Magazine, and Kurt Wismer on his blog.
