How we test Antivirus tools at Comparitech

The antivirus market can be overly complicated, especially if you’ve never purchased an AV tool before. To help cut through the noise, Comparitech reviewers focus on providing authentic in-house reviews that take a broad examination of the trustworthiness, features, and overall effectiveness of each AV tool we feature. We don’t just repeat what everyone else is saying; instead, we look at and gather our own data with real-world tests.

We look through the eyes of a consumer by reviewing AV tools based on:

• Effectiveness at detecting malware
• Effectiveness at removing malware
• Pricing structure and transparency
• Auto-renewal policy and transparency
• System impact when running scans
• Basic features
• Advanced features
• Independent testing lab results

In this post, we’ll explain our methodology for all our Comparitech antivirus reviews and how we examine tools using these methods.

Effectiveness at detecting malware

This is undoubtedly one of the most important rating criteria for AV tools. It also carries a higher weight for us as we rate AV tools. Simply put, if a tool can’t detect malware or if its detection mechanism is spotty we don’t consider it a good application.

To measure effectiveness at detecting malware, we look at two areas:

• Real-world tests using test malware (both EICAR and real malware)
• Independent testing lab results

EICAR testing

When conducting our real-world tests, we first start with EICAR test viruses. These are commonly accepted, non-malicious test files with simple code designed to look like malware to an AV tool. We download the files and record whether those files are detected by both the real-time antivirus scanner, and the quick and full system scans.

A good AV tool with effective detection software will both detect and block EICAR file downloads from the EICAR website. EICAR offers several different iterations of its test virus. We’ve found only a few tools detect and block every EICAR test file before we can download them

To test the quick and full system scans, we typically turn the real-time scanner off. This allows us to install the EICAR test virus files without them getting blocked, assuming the AV tool we’re testing was already blocking those downloads. Then, we run the quick and full system scans and determine if the EICAR files were revealed during those scans.

Many AV tools will detect some (but not all) of the EICAR test files through a quick scan. Most (but not all, unfortunately) detect the EICAR test files with a full system scan. 

Live malware samples

This is the most dangerous part of testing and one we do not recommend you try at home without experience. To more authentically determine the effectiveness of the antivirus tools we review, we install real malware to test how well the AV tools we review identify and deal with those malware files.

As with the EICAR test files, we examine whether the AV tools we test can effectively block live malware samples with real-time scanners. Because we only install password-protected and neutered malware samples to our test machines, we test the real-time scanner after we’ve downloaded the live malware samples and removed the protections to make them detectable as malware.

Additionally, to test the effectiveness of the quick and full scans, we completely turn off all detection settings for the AV tool, install the live malware, then run the system scans. Again, this is the riskiest part of our tests. We only perform this action with live malware samples on a test machine in a sandboxed environment and disconnected from the web and important systems that would allow the malware to spread.

Effectiveness at removing malware

This criterion is just as important as malware detection. We review each tool’s effectiveness at removing malware in 3 areas:

• Real-time malware scanning
• Quick scans
• Full system scans

We also examine whether malware that is identified is just quarantined, or whether it’s also deleted after the scan has finished running. Most tools will do both (quarantine, then delete). Some, however, will require you to pay extra to remove malware. A small number will quarantine only and require you to manually delete malware. 

From our experience, the best AV tools tend to identify, quarantine, and then remove malware using the real-time scanner. These tools do all of the work for you and never allow malware to touch your system. 

Still, many excellent tools will detect and remove most malware through quick scans. In most cases, a full system scan is only needed if you’re installing AV protection on a computer that already has an infection you need to remove. 

Pricing structure and transparency

There are a few questions we ask when rating antivirus tools based on pricing structure and transparency:

• Does the provider have a first-year discount?
• If a first-year discount is offered, does the provider clearly indicate its price increases after the first year?
• How many devices can you cover with each license?
• How much extra does covering each device cost?
• How many licensing/payment options does the provider offer?

We ask all of these questions when looking at pricing structure and transparency because they’re also important for consumers. For example, most AV tool providers will give new customers a first-year discount, but not all providers make it abundantly clear that the price increases after the first year, or don’t clearly show the renewal price.

Providers also don’t exist in a bubble. When we give ratings for pricing structure and transparency, we also take into consideration how the provider fares against other providers and how it compares to that services’ performance as an AV tool. Consequently, if a service is priced high yet performs poorly, its potential score here would be negatively impacted. 

Examples of payment structures and transparency

Companies in this market tend to utilize one of two pricing methods: 

• Annual – You purchase a license to use the software for 12 months (or longer, as some providers offer 2 or 3 years to a license)
• Monthly – You purchase a license on a month-to-month basis, similar to a streaming service subscription model (like Netflix)

Those companies that allow you to purchase a multi-year license sometimes (but not always) discount the purchase price. Most providers that carry multi-device packages also tend to offer discounts for additional devices. That makes them worthwhile options to consider if you like the service they provide and you plan to renew your license after the first year. That said, the vast majority of providers utilize the annual licensing model.

Intego (rated high for Mac antivirus software) and Norton (a household name in AV tools) both provide a good sample of how AV product pricing and transparency normally work in this market. 

Here’s a snapshot Norton’s pricing looks like:

When we see pricing like this, there are 4 things we look at that we’d consider positive during a review:

1. The company identifies the number of devices covered. Each license option shows how many devices you can cover with that subscription
2. You can clearly see the price for the annual license. Generally, if you see two prices (one marked with a strikethrough and one slightly larger beneath it) the bottom price is what you get with the first-year discount. The crossed-out price is what you’ll pay after you renew the service
3. The company shows that the discounted price is for the first year. We prefer to see this listed clearly, in large enough print where it’s hard to miss, and located right next to the price
4. Where available, different licensing types are shown and switching between them shows you how that impacts the price

For its part, Intego provides multi-year pricing, but if you do the math, you’ll notice that there’s no discount for making a multi-year purchase:

While we’d rate a service better for offering multi-year discounts, this might still hold value as it helps avoid any year-over-year price increases. 

Auto-renewal and cancellation policies

Although this topic is directly related to pricing, we consider this issue important enough that we identify it separately when reviewing AV tools. The vast majority of complaints we see from consumers are over a huge price increase once the service automatically renews. 

Many antivirus buyers either don’t realize that their product license will automatically renew, or forget to cancel their service before it auto-renews. 

Given the strong feelings from consumers about this topic, we scrutinize antivirus providers’ auto-renewal and cancellation policies pretty heavily. All else considered we may even choose not to recommend a service if we believe its auto-renewal and cancellation policies are predatory and harmful to buyers. 

There are several questions about a product for this topic:

• Is the auto-renewal policy clearly stated during the checkout process?
• Can subscribers turn auto-renewal off before purchasing the product?
• Can buyers easily adjust auto-renewal after purchasing the software?
• Is the cancellation policy clearly stated?
• Can subscribers easily cancel the service?

This one is fairly straightforward. If a provider does not allow buyers to turn auto-renewal off a checkout, we lower the rating. If the auto-renewal policy is not even stated in an obvious location (e.g., you have to go digging through a Terms of Service page to find it), that’s even worse. 

We also examine how easy it is to turn off auto-renewal after signing up. Providers that force you to email their customer service to turn off auto-renewal or cancel your service get lower scores in this category. 

Basically, the easier it is for you to end the relationship, the better. Note that we aren’t against customer retention policies. It’s always in a provider’s best interest to try to get you to stay a customer. That’s just good business. But some providers have consumer-unfriendly retention policies in place that can make you feel trapped in your subscription and are intended to force you into giving up any idea of canceling your service.  

When we see that type of activity, we take note of it and make sure it’s reflected in the antivirus provider’s score. When the providers’ policies are friendly, we also make sure they’re rated well on this end. 

Kaspersky Antivirus presents a near-perfect example of what we like to see for this criteria:

By default, the company has auto-renewal turned on (not preferable, still acceptable). More importantly, though, Kaspersky clearly identifies that auto-renewal is one at the checkout screen. You can click “Details” to get a full explanation of how the company’s auto-renewal works with the option to turn it off before you buy the license.

We often find that companies with better auto-renewal policies tend to have software that performs better. No surprise there, as those companies are typically more confident that you’ll want to stay a customer.

System impact when running scans

Press CTL+ALT+DEL on your windows system, then open your Task Manager. From there, click on Performance. These are the basic system resources that apps on your computer use anytime you run them. Of these, the most important for understanding how an AV tool will impact your system, particularly when running scans, include:

• CPU – A percentage of the total processing power being used at any given time
• Memory – Tracks how much memory your system is using at any given time (in megabytes (MB) or gigabytes (GB) for modern systems)
• Disk active time – A percentage that measures the amount of time your hard drive is working to read and write data

Antivirus tools can be resource hogs, especially when running scans. Quick scans tend to work your system less and as the name suggests, don’t spend as much time doing so. But in some cases, full scans can take over an hour and cause massive drains on your system resources. That can make it difficult to perform other resource-heavy processes, such as gaming, installing other apps, or even streaming TV shows or movies. 

The better and newer your system, the less likely it is that scans will cause you to use up all of your system resources. But it can be a problem, especially if you’re running an older system, or you already have several other processes running when a malware scan starts. 

For these tests, we do a performance test before running scans to get a baseline, and then a test during the scanning period. This helps us better establish the system impact of each AV tool.

The fewer resources a tool uses during the scanning process, the better, but this is not always a good thing. We’ve encountered some AV tools that hardly make a dent on the system resources because they’re also not effectively scanning for and removing malware.

Basic Features

What do consumers typically expect from antivirus software? At a minimum, the ability to detect and remove malware before it causes a problem. To that end, we look to make sure that any AV tool we rate has these basic features available at every pricing tier:

• Real-time malware scanning and removal
• Quarantines detected malware
• Blocks phishing websites
• Blocks downloading of malicious files
• Provides manual or automatic quick and full system scans

Few providers fall short here, but we consider it an important checkmark for every antivirus tool we review. 

Advanced features

Many antivirus tools are offered through tiered packages. The lowest-cost options tend to have the basic set of features, while additional features may be available at a higher cost. Each company’s idea of “basic” and “advanced” will vary, so you may see some companies offering features within a basic tier that are only available through an advanced tier from another company. 

Quite often, the advanced features are not essential for the all-important task of identifying and removing malware. But they can add extra layers of security considered essential for many buyers.

Types of “advanced” features may include (but are far from limited to):

• Firewall
• Device tracking
• Parental controls
• File shredder
• Password manager
• App updater
• Webcam and microphone protection
• VPN
• Speed optimization

We don’t place a heavy emphasis on advanced features, but we do consider them holistically when we review and rate antivirus tools. More commonly, if a provider locks what we consider to be essential features behind a higher-priced paywall, that’s bad. We’ve seen providers that don’t allow you to quarantine files without paying extra, for example, which is a terrible policy and one we’d mark against a service in a review. 

Independent testing lab results

Some AV tool reviews base their scores solely on independent testing lab results. At Comparitech, we only use independent testing lab results to confirm our own tests. We often find that our results are confirmed by independent test labs, like AV-Test.org and AV-Comparatives.

While these labs do much larger and more extensive tests on AV tools we do in-house, they also test most AV tools infrequently. Some of the tools we’ve rested and reviewed have not been tested by independent AV labs for several years. 

AV testing lab results are a good bellwether for the effectiveness of these tools, and we highly recommend you check out lab results alongside our own. All the same, you will find testing lab results play a small but important part in our rating criteria. 

If you plan to buy a license for an antivirus tool, we recommend you check out our latest review round-ups for the best antivirus for Windows and Mac computers.

L’article How we test Antivirus tools at Comparitech est apparu en premier sur Comparitech.