Santa Leaves The Kernel: A macOS Endpoint Security Introduction and Case Study

00. Intro

This post focuses on a new set of APIs under the System Extensions framework. System Extensions are described by Apple as "user space code that extends the capabilities of macOS," and give us a way to access functionality within userspace that was previously only accessible from within the kernel. We will examine the Endpoint Security framework in particular, and compare Santa’s legacy kernel extension with new Santa code that makes use of the Endpoint Security framework, which falls under the umbrella of System Extensions.

01. Kernel Extensions

Kernel extensions have been historically used to write custom drivers, inspect network activity, and apply policy to certain operations like executing binaries. Little Snitch is a host-based firewall for macOS that allows the user to restrict and monitor the network connections made by various applications. Santa is a Google-built binary allowlisting system that can prevent the execution of unknown binaries across a fleet of macOS endpoints. Both of these applications have a common need: to monitor and apply policy to events originated by other applications, possibly even applications not running as the same user. Until recently, there was no way to access this kind of functionality from userspace: the kernel was the only place that gave the level of access required.

02. The Kernel

Running code in the kernel is inherently a high-risk activity. The kernel is responsible for everything that makes the system run normally: task scheduling, access management, memory mapping and management, etc. An unrecoverable crash in a usermode application will still allow the system to keep functioning, whereas an unrecoverable crash in the kernel takes the rest of the system down with it. There is no higher level scheduling or preemption functionality, so all kernel code must be cooperative and relatively bug-free. Moreover, a vulnerability in the kernel can cause vastly more damage than a vulnerability in a usermode application. Barring functionality like dedicated security hardware, control of the kernel is the keys to the kingdom.

In 2011, MIT researchers found that 2/3 of surveyed Linux kernel bugs were found in loadable kernel modules, 1/3 in the core kernel. Preventing third-party code from running in kernel-space could reduce attack surface significantly.

Even root access doesn’t necessarily imply kernel access. In fact, limiting the scope of root access is exactly what Apple has tried to accomplish with System Integrity Protection (SIP) (also known as "rootless"). This is not a surprise: Apple has invested considerable effort into building out a trusted boot process rooted in hardware, but the security gains of trusted boot are limited if arbitrary code at high privilege levels can be loaded after the boot process is completed. SIP, among other things, prevents modification of certain files and folders, such as Apple-built system components, even by root. SIP protection is enforced by the kernel, but we’ve already seen that kernel extensions can modify kernel behavior, so how do we prevent a kernel extension from simply disabling SIP?

Kernel extension code signing is one of the first tools Apple deployed to address this issue. Since macOS Yosemite (v10.10, released 2014), Apple has required that kernel extensions possess a particular entitlement (essentially a permission granted with a signature) to load. Apple controls who receives these entitlements, allowing them to control which kernel extensions are able to load on macOS machines. Apple has used this ability to gradually phase out certain types of kernel extensions, and has replaced the functionality that they need to access with new APIs that can be accessed from outside the kernel.

03. System Extensions

At the time of writing, Apple provides three APIs under the umbrella of the system extensions: Network Extensions, Endpoint Security, and DriverKit. From Apple’s documentation, the Network Extensions API can be used to build "content filters, DNS proxies, and VPN clients." The Endpoint Security API can be used for "Endpoint Detection and Response software and antivirus software." DriverKit allows developers to write device drivers that run in userspace. This is not an exhaustive list, but one that should give a general idea of the type of software that may have historically needed to rely on kernel extensions, but now does not.

Apple started requiring kernel extension signatures before introducing System Extensions. Now that Apple has started introducing usermode APIs to replace kernel functionality, they will stop providing entitlements to kernel extensions that use these now-deprecated kernel APIs. In effect, they are kicking developers out of the kernel. Given the high-stakes nature of kernel extension vulnerabilities, we think this is a good thing.

04. A Customer Story: Santa

Santa is used by enterprises to restrict the binaries that can run on enterprise machines. We have adapted and deployed Santa here at Duo for our own use cases. Deployment of Santa is part of a defense-in-depth strategy that helps to prevent execution of unwanted software such as malware. There are different bits of management functionality in Santa that enable centralized rule management and synchronization, but the primary functionality of allowing or denying the execution of a binary was formerly only possible via a kernel extension. Santa’s developers have recently migrated this component to use the new Endpoint Security APIs. Santa has the benefit of being an open-source project that sees wide industry use, making it an effective subject for a customer story of migration to the new Endpoint Security APIs.

Both the kernel extension and system extension source are present in the Santa repository, and a well-designed abstraction layer within Santa allows us to compare how these two implementations provide the same functionality (allowing or denying executions) to the rest of the Santa codebase.

Santa has minimal functionality located within the kernel extension (now system extension). Its responsibilities are scoped to: listening to the proper events (i.e. file execution events), communicating with the Santa service santad for policy decisions, responding to those events with an allow or deny, caching those decisions, and invalidating cache entries.

Old News: The Santa Kernel Extension

The Santa kernel extension is named santa-driver, and hooks into the Kauth KPI (kernel programming interface). Let’s look at how santa-driver registers for events:

kern_return_t SantaDecisionManager::StartListener() {
  vnode_listener_ = kauth_listen_scope(
      KAUTH_SCOPE_VNODE, vnode_scope_callback, reinterpret_cast<void *>(this));
  if (!vnode_listener_) return kIOReturnInternalError;
  fileop_listener_ = kauth_listen_scope(
      KAUTH_SCOPE_FILEOP, fileop_scope_callback,
      reinterpret_cast<void *>(this));
  if (!fileop_listener_) return kIOReturnInternalError;
  LOGD("Listeners started.");
  return kIOReturnSuccess;
}

The driver registers callback with two scopes. A scope is "an area of interest for authorization within the kernel." The two scopes used here are KAUTH_SCOPE_VNODE, and KAUTH_SCOPE_FILEOP. The VNODE scope allows listeners to allow, deny, or "defer" actions on vnodes, which are BSD’s representations of file-like objects. The FILEOP scope is advisory in nature: the system can notify listeners of file system operations, but the result (i.e. allow or deny) is ignored. Santa uses the VNODE scope to provide enforcement of allowlisting, and the FILEOP scope mostly for logging and cache invalidation.

Within each scope there are multiple possible actions, for example KAUTH_VNODE_EXECUTE. Let’s take a look at the listener callback for KAUTH_SCOPE_VNODE:

extern "C" int vnode_scope_callback(
    kauth_cred_t credential, void *idata, kauth_action_t action,
    uintptr_t arg0, uintptr_t arg1, uintptr_t arg2, uintptr_t arg3) {
...
  // We only care about regular files.
  if (vnode_vtype(vp) != VREG) return KAUTH_RESULT_DEFER;
  if ((action & KAUTH_VNODE_EXECUTE) && !(action & KAUTH_VNODE_ACCESS)) {
    sdm->IncrementListenerInvocations();
    int result = sdm->VnodeCallback(credential,
                                    reinterpret_cast<vfs_context_t>(arg0),
                                    vp,
                                    reinterpret_cast<int *>(arg3));
    sdm->DecrementListenerInvocations();
    return result;
...
  }
  return KAUTH_RESULT_DEFER;
}

While some context is left out, this is the part of the Santa codebase directly responsible for instructing the kernel to deny or allow the execution of a file via the Kauth KPI. The default return value for this listener is KAUTH_RESULT_DEFER, which means "defer to the other listeners of this scope for an authorization decision." Some scopes have a default listener which implements the standard BSD permission model. All listeners attached to a scope must return KAUTH_RESULT_ALLOW for an action to proceed, so deferring is the same as allowing except in the cases that all listeners defer, in which case the action is denied.

The real decision-making happens in the call to SantaDecisionManager::VnodeCallback. Its functionality is to fetch a decision from santad (which runs in usermode), cache the decision, and return a Kauth result that corresponds to the decision.

Let’s look more at the arguments of the function prototype of a Kauth listener callback, as shown above.

kauth_cred_t credential, void *idata, kauth_action_t action,
uintptr_t arg0, uintptr_t arg1, uintptr_t arg2, uintptr_t arg3

The credential argument gives the credentials of the actor, which means user and group ID information. The idata argument is a cookie, registered in the call to kauth_listen_scope. In the case of Santa, this cookie is a pointer to the SantaDecisionManager object (which is re-cast in the callback so that SantaDecisionManager’s methods can be called). The action argument gives the type of action being performed. The rest of the arguments are action and scope specific. In the snippet above, where a KAUTH_VNODE_EXECUTE action is processed, these arguments include pointers to the vnode and relevant VFS context: Santa uses this information to inspect the file to be executed and make an authorization decision.

The New Santa System Extension

The Santa system extension makes use of the new Endpoint Security API. The bulk of the relevant code can be found in santad/EventProviders/SNTEndpointSecurityManager.mm. This system extension serves the same purpose as the kernel module, but does its job fully outside the kernel.

Programs can connect to the Endpoint Security subsystem with es_new_client and subscribe to particular event types with es_subscribe. An event handler is registered directly with this method call, as es_new_client takes an Objective-C block argument to define the callback. Objective-C blocks are effectively first-class functions that can also store block-specific state. They also have a nice block literal syntax that removes the need for explicit function pointers.

Within Santa, the Endpoint Security client is established as follows:

es_client_t *client = NULL;
es_new_client_result_t ret = es_new_client(&client,
    ^(es_client_t *c, const es_message_t *m) {
      // callback method
      ...
    }
);

And then Santa can subscribe to certain types of events:

es_event_type_t events[] = { ES_EVENT_TYPE_AUTH_EXEC, ES_EVENT_TYPE_NOTIFY_EXIT };
es_return_t sret = es_subscribe(self.client, events, 2);

The callback method prototype (a block prepended by the ^ character) is a bit more streamlined than what we see in the equivalent Kauth functionality. A single callback is registered when an Endpoint Security client is created, and the callback takes two arguments: a pointer to the client and a pointer to the message. The callback will receive all messages that the Endpoint Security client has subscribed to. The es_message_t type is a union of the different message types that can be emitted by the Endpoint Security framework, so the callback can check the action_type field and then access action-specific information after determining which action is being performed. Apple’s notes on es_message_t are great:

A message contains an event monitored by Endpoint Security and an action to perform. The event is a union of types specific to each kind of event. For example, a file-renaming event provides the source and destination paths as the union member rename. Similarly, a process fork event provides the process identifier of the new child process as the union member fork. Inspect the event_type to determine which member of the union to access.

A message can be an authorization request, or a notification of an event that has already taken place, as indicated by the action_type field. For authorization messages, your client handler calls es_respond_auth_result or es_respond_flags_result to authorize, deny, or pass behavior flags back to Endpoint Security.

Santa’s Endpoint Security listener subscribes to several "event types," with a bit more granularity than was provided by Kauth scopes. These event types fall under two "action type" umbrellas: ES_ACTION_TYPE_NOTIFY actions are simply notifications that cannot be allowed or denied, but Santa uses them, for example, to track renamed files. ES_ACTION_TYPE_AUTH actions can be allowed or denied by an Endpoint Security client.

Endpoint Security clients must respond to all authorization actions (assuming the client has subscribed to them), and must do so by a deadline included in the message object. Apple documentation indicates that Endpoint Security clients that consistently do not respond by the deadline may be killed. Santa addresses this issue by setting a timer to issue a "deny" response two seconds before the deadline for each authorization action in case the Santa daemon fails to issue a response. At the time of writing, Apple does not specify the length of the deadline, but Santa source code comments indicate that 60 seconds is common.

As is the case with the kernel extension, binaries to be executed are processed by the Santa daemon, including checking the hash of the binary and the certificate with which the binary is signed against the allowlist. The callback registered with the call to es_new_client is responsible for sending the event information to the rest of the Santa subsystem, and a result is ultimately communicated back to the System Extensions subsystem by calling es_respond_auth_result* with ES_AUTH_RESULT_ALLOW or ES_AUTH_RESULT_DENY.

* Some events require a response via es_respond_flags_result instead, and Apple’s documentation isn’t quite clear on which events these are. Headers seem to indicate that ES_EVENT_TYPE_AUTH_OPEN requires the use of es_respond_flags_result.

05. Other Functionality of Endpoint Security

Our study of Santa has only highlighted the use of a few types of Endpoint Security events, but a full list of event types is available. Apple designed this API to fit most of the use cases of existing endpoint security applications, so there are many different events that allow software to monitor system events in real time. Santa takes a heavy hand by interfering with the execution of binaries, but other applications may just want to inspect system behavior that was previously only accessible within the kernel.

06. Network Extensions and DriverKit

We are not exploring the functionality of Network Extensions or DriverKit in this article, but these two APIs also use the new System Extensions functionality to replace kernel extensions.

Network Extensions allow developers to inspect and filter or block network traffic. This could be used to write a content filter, a custom firewall, or simply inspect incoming / outgoing traffic. This may be helpful from a compliance perspective. Apple has designed the Network Extensions API with privacy in mind: content filtering code used with Network Extensions runs in a sandbox that does not allow network content to escape the sandbox. The code that controls the filter runs in a separate, less restrictive sandbox and does not have access to network content.

DriverKit allows developers to write device drivers in userspace. This functionality was previously limited to kernel extensions because device drivers need low-level access to access device functionality that isn’t already exposed by OS-provided drivers (such as those for USB keyboards). Moreover, these drivers can often be timing-sensitive, and timing guarantees are hard to come by in userspace. DriverKit separates these responsibilities: low-level, timing-sensitive tasks are still performed in the kernel by Apple code. Data sent to and received from the device is passed across the kernel boundary, but potentially dangerous operations such as parsing are all done outside of the kernel.

DriverKit is still limited in scope. Because Apple is providing the code that performs low-level communication with the device, only particular devices are supported: USB, HID, PCI, and serial devices at the time of writing. Drivers for other types of devices still must be written as kernel extensions, but this may change over time. DriverKit essentially allows developers to provide additional functionality for devices that already communicate over known hardware interfaces. Apple provides some guidance on building a device driver using DriverKit.

07. Wrapping Up

Apple has exposed a lot of new functionality with these new APIs, but documentation is still somewhat scarce. There are example templates available for Network Extensions in Xcode, but developers are somewhat on their own when it comes to Endpoint Security and DriverKit at the time of writing.

We expect that much of Apple’s effort has been spent implementing these APIs, but they are not as accessible as they could be—some specifics of API behavior are still buried in header files rather than online documentation. We hope that documentation of these APIs will improve over time.

Apple has not eliminated the need for kernel extensions, but with the release of new System Extensions APIs they have made a significant dent. It is likely we will see Apple continuing to expand the System Extensions APIs, providing more userspace access to privileged system functionality.