results_sm.pngThe results are in! Congrats to Kaspersky, ClamAV, and Norton for being our top performers. Unfortunately, some of the others didn’t do so well.

The test was pretty basic. We threw three sets of viruses at each virus engine to see what percentage was caught. The first set was a basic test set (from eicar.org) that is a universal virus test. The second set was the ‘in-the-wild’ test which we picked from my mailbox that I have received over the years in mass quantities. The third set was the user-submitted set, which ranged from pretty standard viruses to some bizarre stuff I couldn’t identify.

The expected results was that all vendors would catch all of atleast the first two sets (eicar + in-the-wild) because these are all very common viruses that have been around for some time. However, as we’ve seen before, many vendors struggled.

Only three (Clam, Kaspersky, Norton) call all of these. Three others (F-Prot, Sophos, Mcafee) missed a few ranging from an 80-90% catch rate – not very good considering these are all really common viruses, but certainly better than others. GlobalHauri and the gateway appliances (Sonicwall, Fortinet, Watchguard) all performed poorly – catching about 60% and less of these common viruses. Watchguard would only catch one virus (the eicar test virus), which is odd because I thought they used the ClamAV engine.

The overall results were similar, although harder to interpret because we weren’t sure what the user samples really were. I’m fairly confident some were newer “zero-day” viruses, and some were ‘custom’ viruses. Regardless, the vendors scored in a similar order. Kasperksy was the top performer, followed closely by the open source Clam and Norton. F-Prot, Sophos, and McAfee were still the next 3 performers although McAfee didn’t do quite as well on the user set. GlobalHauri, Fortinet, Sonicwall still performed poorly and Watchguard caught none of this set.

Conclusions

As always, we are surprised by how poor many of these solutions are performing. Contrary to many statements, Clam is a top performer, and also ran 10 times faster than many solutions. Kaspersky is clearly an excellent engine, and Norton also performed well although it consumed lots of resources on the test machine. The rest of the solutions, some of which are quite expensive, were mediocre to terrible.

This raises many questions… Why has no one publicized this? What is wrong with the way we are testing antivirus solutions? Why do some testing labs claim Clam does significantly worse than commercial solutions?

Our Goal

Our goal in this test was not to scare people, or even drive people away from some vendors. We simply want to encourage discussion. Tests like these need to be open and transparent. They need to be performed in the open so results can be verified and challenged. They need to be transparent for credibility. (In fact, one audience participant significantly improved one vendor’s performance, Sophos, by pointing out that I needed to add a command-line option. Others pointed out mistakes I made recording results.)

Think we aren’t credible? Good! Go here, download the test set we used and compare to the excel spreadsheet we used to track, and run the test yourself. Just make sure to let me know what you find!