Mac OS X is certainly not impervious to malware, and there have been some imperfections in Apple’s handling of security issues over the years. However, there are some very innovative security features in Mac OS X that can do a lot to help protect you, if you let them. The key is knowing what they are and what they do, so that you don’t unintentionally disable something without understanding the consequences.
File Quarantine
File quarantine is a feature of Mac OS X introduced in Leopard. It is explained very well in Apple Support article HT3662, but here’s the gist of it: when you download a potentially dangerous file using a quarantine-aware application (such as Safari or Mail), that file will be “quarantined.” When you try to open it, the OS will warn you and ask if you really want to open it. Obviously, if you see this warning when trying to open something you didn’t think was an application – for example, if you thought the file was a song in MP3 format or a picture in JPEG format – you probably shouldn’t open it.
XProtect
In Snow Leopard, quarantine was expanded to also check for trojans. Quarantine now uses a technology Apple has quietly named XProtect to scan downloads for known malware. The list of recognized trojans has been expanded many times from the original two (RSPlug and iServices) included in 10.6.0, and as of Security Update 2011-003, new malware definitions are downloaded daily, when available. If you try to open a quarantined file that is actually a trojan, you will get a very different and scarier warning that tells you the application is malware.
Example XProtect warning. Image referenced from Apple.com.
Any of Apple’s applications that allow you to download support quarantine. However, results are more mixed with third-party applications. Some will support quarantine and some will not. Especially when using peer-to-peer file sharing programs, which are one of the biggest vectors for malware, I strongly advise testing support for quarantine. Download an application from a trusted source, and if you can open it without a quarantine warning, you know that the program that downloaded it does not support quarantine and could provide malware with a backdoor into your system by letting it sneak past quarantine.
There are many web sites that will tell you how to turn these “annoying” warnings off. I strongly recommend that you do no such thing, as this can also give malware a way to sneak onto your system. Although this system has its flaws – sometimes not receiving updates in as timely a fashion as would be desired – it is nonetheless an important security feature.
The list of definitions can be found, by those interested in such things, at the following path on a Mac OS X 10.6 or 10.7 system:
/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/XProtect.plist
If you choose Go -> Go To Folder in the Finder and paste that path into the window, that will take you there. Getting inside the CoreTypes.bundle “file” manually may be a stumper, otherwise, for those who don’t know the trick.
Gatekeeper
In Mountain Lion (OS X 10.8), Apple added Gatekeeper, which provides for a way to limit what applications are allowed to run based on code signing. Code signing is a method by which a developer uses a security certificate issued by Apple to cryptographically sign their app, verifying ownership of the code found in the app. A code-signed app cannot be modified without breaking the signature.
In System Preferences -> Security & Privacy -> General, you will see a control to set what applications are allowed, via three radio buttons. You can allow only applications downloaded from the Mac App Store, the most restrictive option. In this case, applications you downloaded from any other source will not open.
You can also choose to allow applications from the App Store and those from “identified developers.” This means that applications from outside the App Store will work if they have been code-signed by a developer who is registered with Apple. Although malware has been seen now and then with a valid developer signature (either stolen or registered with false credentials), that hasn’t become common. In addition, every time this has happened, Apple simply revoked the certificate in question, halting the spread of the malware. This is probably the ideal setting for most people, since it provides a significant amount of protection without being too restrictive.
The third radio button allows you to give any application, regardless of source, the right to run. This is the same behavior as in previous systems, and you should still have XProtect defending you against known malware. However, malware has been known to get past XProtect, since XProtect – like any anti-malware software – can only protect against known threats. This is the least safe option, and I discourage its use.
Fortunately, if you want to open a legit app that has not been signed, there’s a way to do that without changing the setting… simply control-click the app and choose Open from the menu that appears. This will result in a warning, but you will be allowed to open the app if you choose to, rather than being blocked completely.
Gatekeeper is integrated with the quarantine system, and thus is only capable of blocking applications that would trigger a quarantine warning (ie, those that are downloaded from the internet via quarantine-aware apps). Do not be surprised when your Gatekeeper preference does not appear to be respected for apps that were already on your machine at the time you installed Mountain Lion. For good or for ill, those apps are considered to be “trusted” apps, and will not be blocked by Gatekeeper.
It is important to understand that quarantine, XProtect and Gatekeeper will not protect you against malware that enters your system through vulnerabilities in third-party software, such as Java or Flash. Such software can provide a back door that lets that malware sneak in behind the system’s back. As such, I highly recommend disabling Java if you have it enabled, or not installing it in the first place in Lion and Mountain Lion. Flash is less easy to do without, but you could use a browser that provides “click to play” access to plugins, or the ClickToPlugin extension for Safari, to make Flash a bit safer.
If you do have Flash or Java installed and enabled in your web browser, Apple has established a pattern of blocking insecure versions of these plugins whenever vulnerabilities are discovered that could affect Mac users. The XProtect system was updated to include minimum allowed versions of these plugins, and these minimum versions are changed as needed. This prevents those vulnerabilities from being used to infect Mac users (after XProtect gets updated, at least). As with quarantine, this has caused some unrest among those who want to run old plugins, but I strongly recommend that you do not follow any instructions that tell you how to modify these minimum version numbers!
Adware
Mac OS X does not currently protect you very well against adware. There are a number of adware programs out there these days, which get installed through devious methods. Sometimes they are included with installers downloaded from unscrupulous download sites, such as Softonic or Download.com. Sometimes they are found on sites offering Adobe Flash Player updates, video plug-ins, video streaming apps and other assorted junkware, but what you end up downloading is really just an adware installer with no signs of the promised software. Often they are found when downloading files from torrents or from piracy sites (like Pirate Bay).
Unfortunately, most adware is not detected by XProtect in Mac OS X, nor is it blocked by Gatekeeper. In fact, most anti-virus apps won’t even detect adware at all, and if they do, they only call it a PUA (Potentially Unwanted Application) or PUP (Potentially Unwanted Program) rather than actually calling it adware.
If you think you might be infected with some kind of adware, see my Adware Removal Guide for assistance.