With the recent news coverage of the Pintsized malware, which infected high-profile targets like Facebook, Apple and Microsoft, much has been said in passing about Gatekeeper. Specifically, a number of news reports have mentioned how Pintsized was able to get past Gatekeeper. Unfortunately, these reports have mentioned this fact without any real understanding of what it means, and this has left many Mac users concerned. Although there is an important message hidden in that information, it’s not as dire as it sounds in a news blurb. So, what exactly is Gatekeeper, and what do we need to know about it?
In Mac OS X 10.5 (aka Leopard), Apple added a feature called file quarantine. This defined special metadata that was added to a file when it was downloaded from the internet. When the user tried to open an executable file (such as an application or a Unix script, for example) that had been marked as being quarantined, the user would be warned and asked whether to really open the file. This was a fair start at protecting the user against opening something that was pretending to be something else, but wasn’t perfect. Some users were annoyed by this and found ways to turn it off, while others just clicked whatever was necessary to get the warning to go away, without thinking about what they were approving.
Another limitation of quarantine was that it only worked with files that had been downloaded from the internet, not copied from a CD or DVD, flash drive, external hard drive, a shared network volume mounted as a “disk” in the Finder or other such devices. Further, quarantine only worked if the file was downloaded using a quarantine-aware application, such as Safari or Mail. Most third-party software for downloading files over the internet also support quarantine, but some don’t.
Apple improved on quarantine as time went on, such as adding basic malware protection in the form of technology called XProtect. This examines files that have been quarantined when they are opened, and presents the user with a warning if the file is known to be malware. Of course, this also suffers from weaknesses. Definition-based anti-virus software always plays “catch-up” with malware, since it can only block known malware and is useless against brand-new malware.
In Mac OS X 10.8 (aka Mountain Lion), Apple added another new feature that built on top of the quarantine system. This feature, which they named Gatekeeper, introduces some slightly different behavior when the user tries to open a quarantined app. Now, the user has the allow only certain classes of applications, blocking others entirely. In the Security & Privacy pane of System Preferences, under the General tab, there are now the three radio buttons shown at right.
Each of the Gatekeeper options provides a different level of security. The first restricts the user to using only apps that have been downloaded from the App Store. Since Apple examines every app in the App Store, and ultimately only allows apps that comply with the rules, this is the most secure option. However, it is also the most restrictive option, since App Store apps have a number of feature-limiting restrictions placed on them, and thus not all apps are offered through the App Store.
The second option also allows apps that have been “signed” by an identified developer. This means that the developer has registered with Apple and paid a fee, and in return is given a secure certificate that can be used to digitally sign the app’s code, verifying its integrity and vouching for who created it. Although not impossible, it’s very unlikely that a developer would use their bought-and-paid-for developer account, with associated contact and billing information, to create malware. Especially since Apple could simply revoke the certificate and cause Gatekeeper to block the app!
Finally, there is an option that allows the user to open apps downloaded from “anywhere.” This is the least restrictive option, but it also entirely disables Gatekeeper, making the system less secure. Fortunately, it’s never necessary to actually choose this setting, since you can opt out of this protection on a case-by-case basis. By control-clicking a newly downloaded app and choosing Open, the user will be allowed to open the app regardless of Gatekeeper settings, after receiving a warning and approving the action.
So, now that we have a basic understanding of how Gatekeeper works to protect against malware, we need to understand its drawbacks. Gatekeeper suffers from limitations because of the fact that it relies on quarantine. Thus, quarantine’s flaws are Gatekeeper’s flaws. Gatekeeper can only block applications that have been marked as quarantined, while any apps that have not been quarantined will be allowed to open unrestricted. It will also not function if you have disabled quarantine entirely.
When it comes to malware like Pintsized, it’s important to understand that quarantine’s limitations mean that the malware cannot be caught by Gatekeeper. Such malware uses Java vulnerabilities to install itself. In other words, this means that while Gatekeeper is standing in front of the gate watching everything coming through, there’s a hole in the wall elsewhere that something manages to crawl through. Because it comes in through a bug in Java, it bypasses the entire quarantine system.
This is definitely a significant concern, no doubt about it. Users should be warned about this possibility. However, this is not new, as recent news seems to imply. This has been a limitation of Gatekeeper since day one, and several other malware families have previously taken advantage of vulnerabilities in Java, Adobe Flash Player and Microsoft Office to bypass Gatekeeper. Gatekeeper is not perfect, nor has it ever been. It’s important to realize that, like any security, the scope and effectiveness of Gatekeeper is limited.
This does not mean that it is weak or not useful, however! On the contrary, Gatekeeper is an excellent layer of security. There have been other families of malware that have been entirely blocked by Gatekeeper, unless the user has disabled it. For example, I was forced to temporarily disable Gatekeeper in order to discover that a variant of the SMSSend trojan was not detected by XProtect, because Gatekeeper would have prevented that app from running.
What does all this mean? In a nutshell, although Gatekeeper is a wonderful addition to Mac OS X, it’s important to keep in mind that no security is perfect. All security systems have weaknesses that can be exploited by an attacker. To stay safe in the modern online world, it is necessary to use multiple layers of security, and to have a full understanding of how those layers work to protect you.
Tags: Gatekeeper, Mac OS X, quarantine