Building a Native Python Interface for /dev/nsm

The cornerstone for security in Nitro Enclaves is the Nitro Secure Module at /dev/nsm. This device provides a two-way communication channel between the Enclave application and the underlying hardware. In this post we will explore how to natively interact with /dev/nsm from Python.

The /dev/nsm device is only available within a Nitro Enclave. Its main purpose is to provide attestation: proof that the Nitro Enclave is who they say they are. This attestation can be used in communications with KMS, guaranteeing that specific KMS keys can only be used from within the enclave. For additional context, see my other posts:

ioctl is used to interact with /dev/nsm. From Wikipedia:

ioctl (an abbreviation of input/output control) is a system call for device-specific input/output operations and other operations which cannot be expressed by regular system calls.

In the NitroPepper post we used a library written in Rust to provide an interface between Python and ioctl. Now, I’ve written a native Python library to provide the same functionality. This removes the need to import a shared object (.so) library and improves compatibility. The library (called aws-nsm-interface) is publicly available and free to use at PyPi and GitHub. Installing the library for use in your application is as easy as pip install aws-nsm-interface.

Working backwards from ioctl

This post will cover how aws-nsm-interface is built. To understand the process, we will begin with ioctl and continuously add steps until we have a fully functional library. Simplified, these steps are:

[/dev/nsm] <- ioctl <- NsmMessage <- IoVec <- Request buffer
                            \ <----- IoVec <- Response buffer

So let’s look at the ioctl call in Python:

operation = IOC(
    IOC_READ|IOC_WRITE,
    NSM_IOCTL_MAGIC,
    NSM_IOCTL_NUMBER,
    ctypes.sizeof(NsmMessage)
)
# Execute the ioctl call.
fcntl.ioctl(file_handle, operation, nsm_message)

There are three important components here. The first is file_handle. This is a simple file object returned by open('/dev/nsm', 'r+').

The second is operation, also called the request code. This code identifies the direction of the request (read and/or write), the operation (NSM_IOCTL_MAGIC is unique to /dev/nsm, NSM_IOCTL_NUMBER is always zero) and the size of the data we’re sending.

The third component is nsm_message, which is the data we’re actually sending to /dev/nsm through ioctl.

The NsmMessage struct

The variable nsm_message is an instance of the NsmMessage struct (the same we used in ctypes.sizeof(NsmMessage)). NsmMessage is defined as follows:

class NsmMessage(ctypes.Structure):
    request: IoVec
    response: IoVec

    _fields_ = [
        ('request', IoVec),
        ('response', IoVec)
    ]

This is a native Python class which inherits from ctypes.Structure. It has two fields, request and response, both of which are of the IoVec type. So when we’re sending an NsmMessage to ioctl, we’re sending a struct which consists of two IoVecs. The obvious next question is… what is an IoVec?

The IoVec struct

The IoVec class in Python looks like this:

class IoVec(ctypes.Structure):
    iov_base: ctypes.c_void_p
    iov_len: ctypes.c_size_t

    _fields_ = [
        ('iov_base', ctypes.c_void_p),
        ('iov_len', ctypes.c_size_t)
    ]

So IoVec is another struct with two fields. iov_base is a pointer and iov_len is an integer (32 bits or 64 bits, depending on the system). The two fields in an IoVec are used to point to a memory buffer. iov_base contains the location of the buffer, and iov_len contains the length of the buffer. With these two properties, IoVec can be used to identify the exact memory space of the buffer.

Back to /dev/nsm and ioctl: when we’re sending an NsmMessage, we’re sending two IoVecs: request and response. These IoVecs then point to the memory location of the actual request and response buffers. We’ll take a look at these buffers in the next section.

Request and response buffers

The request and response buffers for the GetRandom call are initialized like so:

nsm_key = 'GetRandom'
request_data = cbor2.dumps(nsm_key)

request_buffer = ctypes.create_string_buffer(request_data, len(request_data))
response_buffer = (NSM_RESPONSE_MAX_SIZE * ctypes.c_uint8)()

What we’re seeing here is a string ‘GetRandom’ encoded with CBOR (more on that later). This results in a byte array request_data. The request buffer is then initialized with exactly the size and contents of this request data (10 bytes).

The response buffer is not initialized with a value (because we don’t know the response yet), but with a size NSM_RESPONSE_MAX_SIZE instead. This value equals 0x3000 bytes, or 12288 bytes, or 12kb. In other words, we have initialized an empty buffer, 12 kilobytes in size. This buffer will be filled by ioctl.

Now that we have two buffers, we need to create the IoVecs and point them to these buffers. That’s done in the following code:

request_buffer_pointer = ctypes.cast(
    ctypes.byref(request_buffer),
    ctypes.c_void_p
)
nsm_message.request = IoVec(
    request_buffer_pointer,
    len(request_buffer)
)

This snippet only covers the request buffer, but the code for the response buffer is exactly the same. With this part complete, we now have: An NsmMessage with two IoVecs that point to a request buffer sized to fit the CBOR request and a 12kb response buffer.

Concise Binary Object Representation (CBOR)

In the previous section we saw that the request buffer was filled with CBOR data. The following code shows a slightly more complex example:

nsm_key = 'DescribePCR'
request_data = cbor2.dumps({nsm_key: {'index': index}})

request_buffer = ctypes.create_string_buffer(request_data, len(request_data))

CBOR stands for Concise Binary Object Representation. As the name suggests, this is an efficient binary format to transfer data (objects). The Nitro Secure Module expects its request data in CBOR format, and will return its responses in CBOR format as well.

CBOR is loosely based on JSON, so converting from and to JSON-like dictionaries is generally hassle-free, as you can see in the code above.

CBOR and /dev/nsm

To send a request to /dev/nsm, we fill the request buffer with a request in CBOR format. We send the request buffer to /dev/nsm through the IoVec and NsmMessage structs. When the request has no parameters, like GetRandom, we simply CBOR-encode the string ‘GetRandom’. If a request does have parameters, like DescribePCR, we create a dictionary with ‘DescribePCR’ as key and the required parameters as a nested dictionary, for example { 'DescribePCR: {'index': 0} }.

Retrieving the response

When all the preparations above have been completed, we send the request to /dev/nsm with fcntl.ioctl(file_handle, operation, nsm_message). ioctl processes the request and generates a response. This response is written to the response buffer specified in the response IoVec, and the iov_len value in this IoVec is updated to match the length of the actual response. This is all the information we need to retrieve the data from the buffer.

The code to retrieve the data from the response buffer is:

# Create a buffer with a size as defined in the IoVec.iov_len field.
cbor_data = bytearray(nsm_message.response.iov_len)

# Create a pointer to this buffer.
cbor_data_pointer = (ctypes.c_char * nsm_message.response.iov_len).from_buffer(cbor_data)

# Copy the data referenced to by the IoVec into this buffer.
ctypes.memmove(
    cbor_data_pointer,
    nsm_message.response.iov_base,
    nsm_message.response.iov_len
)

# Decode the CBOR and return it.
return cbor2.loads(cbor_data)

In these few lines of code, we create a new byte array with the length specified in the response IoVec. We then create a pointer to this buffer, and use ctypes.memmove() to copy the bytes from the response buffer into the newly created buffer. Finally, we use cbor2.loads() to convert these bytes into a native Python dictionary for further use. An example decoded response for the DescribeNSM call might look like {'version_major': 1, 'version_minor': 0, 'version_patch': 0, 'module_id': 'i-00c89f181802cdef4-enc0175cd0dcee36866', 'max_pcrs': 32, 'locked_pcrs': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], 'digest': 'SHA384'}.

Full circle

With this response in a native Python dictionary, we can implement whatever Nitro Enclave application you like. To recap the full process, beginning to end:

Generate a request
Encode this with CBOR
Create a request buffer with the encoded request
Create an empty response buffer
Create two IoVecs, pointing to these buffers
Create an NsmMessage with these two IoVecs
Send the NsmMessage to /dev/nsm using ioctl
Read the response from the response buffer
Decode this response into a native Python dictionary

Try it yourself!

To use this library, install it into your project with pip install aws-nsm-interface. To use it, for example to generate a random 12 byte array, add the following code to your application. Keep in mind that this only works in a Nitro Enclave, because /dev/nsm needs to be present.

import aws_nsm_interface

file_desc = aws_nsm_interface.open_nsm_device()

rand_bytes = aws_nsm_interface.get_random(file_desc, 12)
print(rand_bytes)

aws_nsm_interface.close_nsm_device(file_desc)

The most common use case for this library, generating attestation documents, can be achieved as follows:

import base64
import aws_nsm_interface

file_desc = aws_nsm_interface.open_nsm_device()

public_rsa_key = b'1234' # An RSA public key exported as DER

attestation_doc = aws_nsm_interface.get_attestation_doc(
    file_desc,
    public_key=public_rsa_key
)['document']

attestation_doc_b64 = base64.b64encode(attestation_doc).decode('utf-8')

aws_nsm_interface.close_nsm_device(file_desc)

# Use `attestation_doc_b64` in your AWS KMS Decrypt call

That’s it! No Rust code, no .so files, just plain Python. For more extensive information on how to use the library, see the project’s README.

If you would like to read more about Nitro Enclaves, check out my other posts:

I share posts like these and smaller news articles on Twitter, follow me there for regular updates! If you have questions or remarks, or would just like to get in touch, you can also find me on LinkedIn.

Tech Blog