mTLS and OAuth2 — Client Authentication
Whenever we hit a website or web application, we simply trust the browser to verify its authenticity and establish a secure, encrypted communication channel. As we all know, the underlying mechanism used here is known as TLS/SSL over HTTP — commonly referred to as HTTPS.
Generally, our web browser verifies the server’s identity before establishing this secure connection. But this is a one-way verification, where the browser (acting as the client) checks the identity of the server hosting the web application or the website (acting as the server). In short, the client verifies the identity of the server.
However, there are scenarios where both the client and the server need to authenticate each other before a secure connection can be established. This is known as mutual TLS, or mTLS for short. This isn’t a new concept — we’ve been using mTLS in various systems for over a quarter century now. While mTLS is less common in public web applications, it is widely used in system-to-system communication, especially in regulated industries like finance and healthcare.
In this post, we’re going to explore whether the authentication mechanism used in mTLS can be applied to OAuth2 client authentication. This can offer stronger security than using shared secrets like client_secret, and better interoperability with systems where mTLS is already in use.
Given that this is a broad topic, I’ve broken it down into several chapters for easier reading. Let’s start by exploring certificates, the key enabler of authentication in both TLS and mTLS.
Chapter 1 — Handshake
Although we commonly refer to them as certificates, the more technically accurate term is X.509 certificates. They’re called that because they follow the X.509 standard, which was introduced by the International Telecommunication Union (ITU) as part of the X.500 directory services back in 1988.
The diagram below provides a simplified view of a typical certificate, highlighting only the key elements relevant to this discussion
One important concept to understand is the Distinguished Name, or DN. A DN is a string composed of attribute-value pairs, and it acts as a unique identifier for entities such as certificate authorities, servers, or applications within X.509 certificates.
For example:
CN = DigiCert Global G2 TLS RSA SHA256 2020 CA, O = DigiCert Inc, C = USThis is a DN made up of several key attributes:
- CN — Common Name
- O — Organization
- C — Country
Together, these elements form a unique identifier that can be used to represent the issuer or subject in a certificate.
Now, let’s break down some of the key components you’ll find in the diagram above.
🏷️ Issuer — Identifies the entity that issued the certificate. This is usually represented as a Distinguished Name (DN), following the LDAP protocol.
Examples:
CN = DigiCert Global G2 TLS RSA SHA256 2020 CA, O = DigiCert Inc, C = USCN = GTS CA 1C3, O = Google Trust Services LLC, C = US🧾 Subject — Identifies the entity the certificate was issued to. Like the issuer, this is typically a Distinguished Name (DN) or a Subject Alternative Name (SAN).
Examples:
CN = *.wso2.com, O = WSO2, Inc, L = Mountain View, ST = California, C = USCN = *.google.com🔑 Public Key — Contains the public key that corresponds to the private key held by the subject. This key is used in the TLS handshake for encryption and key exchange.
🖋️ Signature — The certificate’s digital signature, created by the issuer using its private key. The certificate content is first encoded, then hashed, and that hash is signed. This ensures the integrity and authenticity of the certificate
Most certificates you encounter in the wild (e.g., websites) are issued by a Certificate Authority (CA) as part of the Public Key Infrastructure (PKI). But it’s also quite easy to generate self-signed certificates, especially for internal use or server-to-server communication. The key difference? With self-signed certificates, the issuer and the subject are the same.
🔎 Example 1: PKI-Issued Certificate (from WSO2 Website)
- Issuer:
CN = DigiCert Global, O = DigiCert Inc, C = US - Subject:
CN = *.wso2.com, O = WSO2, Inc, L = Mountain View, ST = California, C = US
🔎 Example 2: Self-Signed Certificate (from WSO2 Identity Server)
- Issuer:
CN = localhost, OU = WSO2, O = WSO2, L = Mountain View, ST = CA, C = US - Subject:
CN = localhost, OU = WSO2, O = WSO2, L = Mountain View, ST = CA, C = US
Now, let’s take a brief look at how we validate a certificate received from another party. The approach to validation depends on whether the certificate is CA-issued or self-signed.
For CA-issued certificates, the main process is certificate chain validation. This involves verifying the certificate’s signature using the public key of the issuer. It also requires checking that the issuer’s Distinguished Name (DN) in the current certificate matches the subject’s DN in the next certificate in the chain. The validation process begins with the leaf certificate (which corresponds to the entity being verified, like a server or client). Once that’s validated, the process moves on to the intermediate certificates, eventually reaching the root certificate, which provides the foundation of trust for the entire certificate infrastructure.
Root certificates come from highly trusted Certificate Authorities (CAs) and are recognized in systems that rely on Public Key Infrastructure (PKI) — like web browsers and operating systems. Only a select few CAs can issue root certificates. Once the entire certificate chain is validated, the subject of the certificate is confirmed as the identity of the entity being verified.
When it comes to self-signed certificates, there’s no one-size-fits-all method to confirm the validity , as the approach often depends on the use case.
When connecting to a web server with a self-signed certificate, web browsers typically show warnings to users, informing them that the certificate is not trusted by the browser or the operating system. However, many browsers allow users to manually inspect and accept the certificate. Additionally, certain client-side tools, like cURL, offer an option (using -k or --insecure) that lets users bypass certificate verification when establishing the TLS connection. This flexibility allows users to work with self-signed certificates, though it’s important to be cautious when doing so.
Non-browser applications, such as in system-to-system communication, as well as APIs, have the ability to explicitly trust a self-signed certificate by presenting a copy of the certificate before initiating communication. This can be done by either directly uploading the certificate in PEM or another format, or by providing a URL where the certificate content can be retrieved. For example, the JSON Web Key Set (JWKS) endpoint is often used to offer an entity’s public key information through a network-accessible endpoint. In OAuth 2.0 and OpenID Connect (OIDC), JWKS is commonly used to provide clients with the public keys necessary to verify JSON Web Tokens (JWTs).
Before we conclude this section, let’s touch on an important point regarding certificates. If you’ve ever viewed a certificate using the pop-up in web browsers, you might have noticed a small string called a certificate fingerprint or thumbprint. Interestingly, this is not part of the certificate itself; rather, it’s calculated by the browser and presented to uniquely reference the certificate. To better understand how this works, let’s quickly go over the steps involved in creating certificate fingerprints:
- If the certificate is in PEM format (a Base64-encoded ASCII sequence), it should first be converted into DER format, which is a binary encoding format.
- Then, a hash value is calculated by applying a hash function such as SHA-256.
- Finally, this hash is formatted into a human-readable fingerprint.
After reviewing the fundamental concepts of certificates and their validation methods, let’s move on to the core topic of this chapter: TLS and mTLS handshakes. It’s important to clarify that the goal here isn’t to provide an exhaustive discussion of TLS, as there are already many resources available that dive deep into it. Instead, we’ll focus on the key aspects most relevant to our discussion.
The first step in establishing a secure, encrypted communication channel between two entities using the TLS protocol is known as the TLS handshake. The diagram below provides a comparative illustration of TLS and mTLS handshakes. To simplify things, let’s break it down into two roles: the client and the server. The client is the entity that initiates communication to obtain a service, such as a web browser or mobile app. The server, on the other hand, listens for incoming communication on a network port to provide that service. For example, a web server might listen on port 80 to serve web pages.
In step 3 of the handshake process, the server sends its certificate to the client using the Certificate message as defined in the TLS protocol. This allows the client to validate the server’s certificate and ensure the server’s identity. The validation process can be carried out using one of the certificate validation methods we discussed earlier. For example, if the certificate is a PKI certificate, the client can validate the certificate chain. If it’s a self-signed certificate, the client may choose to trust the certificate by comparing the provided details before initiating the handshake. This step is common to both TLS and mTLS protocols.
In contrast, mTLS introduces an additional layer of security. In step 5, the server requests the client to provide its certificate by sending the CertificateRequest message. This occurs after the client has verified the server’s certificate in step 3. The client responds by sending its certificate to the server using the Certificate message. Similar to client-side certificate validation, the server must validate the client certificate using one of the methods discussed earlier, confirming the client’s identity before proceeding. After step 7, the server has successfully authenticated the client, and the client’s identity is now known to the server. The diagram below illustrates these steps in detail.
In summary, in TLS, only the client authenticates the server during the handshake, while in mTLS, both the client and server authenticate each other using certificates.
As illustrated in the diagram above, it’s crucial to highlight another important message that the client sends to the server during step 9: the CertificateVerify message. The primary purpose of the CertificateVerify message is to demonstrate to the server that the client holds the private key corresponding to the presented public key, thereby proving possession of the private key. This step is essential for understanding the upcoming Chapter-X of this post.
The steps involved in generating the content of the CertificateVerify message are as follows:
- Collect the handshake messages exchanged between the client and server up to this point.
- Apply a secure hash function like SHA-256 to the collected message data to generate a hash value.
- Sign the hash value using the client’s private key to create the signature that will be included in the CertificateVerify message.
When the server receives the CertificateVerify message, it checks if the client’s signature is valid. To do so, the server uses the client’s public key, which it obtains from the client’s certificate. The server then independently calculates a hash value based on the messages exchanged earlier during the handshake. It compares this calculated hash value with the one sent by the client. If they match, the server confirms that the client used the correct private key to create the signature. This process ensures the client is authentic and possesses the private key associated with the presented certificate.
With a clear understanding of certificates, certificate validation, and the key steps in the mTLS handshake, we can now conclude this chapter and proceed to Chapter 2.
Chapter 2 — Client Authentication
In the realm of OAuth2, a client refers to an entity that obtains an access token from an authorization server on behalf of an end user. OpenID Connect (OIDC), an authentication protocol built on top of OAuth2, redesignate the OAuth2 client as the Relying Party (RP). OAuth2 clients are categorized into two main types: confidential clients and public clients.
- Confidential clients, such as server-side web applications, can securely hold client-associated security credentials.
- Public clients, like browser-based applications, cannot securely store credentials.
As per the OAuth2 specification, authorization servers (AS) are required to authenticate confidential clients before issuing tokens. The OAuth2 specification outlines client authentication as an abstract requirement, giving implementers the flexibility to choose a client authentication method that suits their needs. However, the specification also defines one specific client authentication method based on a shared secret known as client_secret. During client registration, both the client and authorization server agree on the client_secret value.
When making token generation calls, the client authenticates with the authorization server by providing the client_secret. A commonly adopted practice is to send the client_id and client_secret through the HTTP Authorization header.
Despite its widespread use, shared secret-based client authentication may not be suitable for all use cases. To address diverse industry needs, the OAuth2 working group has introduced various client authentication strategies over time, allowing for more flexibility in different contexts.
Chapter 3 — mTLS for OAuth2 Client Authentication
In Chapter 1, we highlighted a key point about mTLS: the client entity needs to authenticate with the server entity during the handshake by presenting its certificate.
According to OAuth2 security requirements, a server-side application acting as an OAuth2 client should exclusively communicate with the Authorization Server using HTTP over TLS, avoiding plain HTTP. In this scenario, the OAuth2 client acts as the TLS client, while the authorization server assumes the role of the TLS server. Furthermore, as discussed in Chapter 1, when mTLS is employed, the OAuth2 client undergoes authentication with the Authorization Server during the mTLS handshake.
Now, considering this: when an OAuth2 client initiates a request to an Authorization Server using an mTLS connection, the client has already been authenticated during the mTLS handshake through a client certificate at the TLS level.
Can we then leverage this authentication, which has already occurred at the TLS level, to fulfill the OAuth2 client authentication requirement outlined in Chapter 2? The straightforward answer is yes — and that’s exactly the objective addressed in the first part of the OAuth 2.0 Mutual-TLS Client Authentication and Certificate-Bound Access Tokens (RFC 8705) specification.
In fact, mTLS-based OAuth2 client authentication is more secure than the shared secret-based authentication method.
We previously explored two types of certificates — PKI certificates and self-signed certificates. Now, let’s see how both types can be employed for OAuth2 client authentication, as outlined in the OAuth 2.0 Mutual-TLS Client Authentication and Certificate-Bound Access Tokens specification.
When a PKI certificate is used in the mTLS handshake, the Authorization Server already possesses the verified certificate through certificate chain verification. This allows the Authorization Server to simply compare the subject of the underlying certificate with the subject information provided during the client registration of the specific OAuth2 client.
Similar to how an OAuth2 client provides registration data like the client name and redirect URL, it can register the Distinguished Name (DN) or Server Alternative Name (SAN) during the client registration. These pieces of information can then be utilized to authenticate the client by comparing the registered DN/SAN with the DN/SAN values present in the subject of the certificate. However, for each OAuth2 client, a single DN or SAN should be used.
One of the main advantages of PKI certificates is that the client can rotate certificates as long as the new certificates contain the same subject value. Authorization servers may optionally check the revocation status of the certificate as well.
Here is an sample token request:
curl --location --request POST 'https://localhost:9443/oauth2/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'code=be728f21-8b3b-3c90-98e3-66188a647407' \
--data-urlencode 'grant_type=authorization_code' \
--data-urlencode 'client_id=QGR5np4F_OYGSH8L5F3klJJPrUEa' \
--data-urlencode 'redirect_uri=https://myapp/login'An Authorization Server can advertise its support for OAuth2 mTLS by adding the tls_client_auth entry to the server metadata definition.
For the purpose of client registration using the DCR endpoint, the OAuth 2.0 Mutual-TLS Client Authentication and Certificate-Bound Access Tokens specification defines the following additional parameters to register the DN and SAN associated with an OAuth2 client:
- tls_client_auth_subject_dn: Specifies the expected Subject DN of the certificate that the OAuth2 client will present during mTLS authentication and use this when you want to match the full Distinguished Name (DN) of the client certificate.
- tls_client_auth_san_dns: Specifies the expected Subject DN of the certificate that the OAuth2 client will present during mTLS authentication and useful for matching domain names tied to the client identity.
- tls_client_auth_san_uri: Specifies the expected Subject DN of the certificate that the OAuth2 client will present during mTLS authentication and often used when the client is identified by a URI, such as a service URL.
- tls_client_auth_san_ip: Specifies the expected Subject DN of the certificate that the OAuth2 client will present during mTLS authentication and common in internal network setups where services are identified by IP.
- tls_client_auth_san_email: Specifies the expected Subject DN of the certificate that the OAuth2 client will present during mTLS authentication and helpful when the client identity is tied to an email address.
When self-signed certificates are used, the Authorization Server can allow the OAuth2 client to easily upload the client certificate in PEM or JWKS format, or provide a JWKS endpoint URL. When the client initiates a token request using an mTLS connection, the certificate linked to the mTLS connection can be matched with the certificate details registered during the OAuth2 client registration. If the complete certificate is accessible as client registration metadata, the Authorization Server can directly compare it with the mTLS client certificate. If not, the Authorization Server can retrieve the registered certificate using the JWKS endpoint URL provided during client registration.
If the client certificate is directly uploaded to the Authorization Server, any changes in the certificate, such as rotation, would necessitate modifying the certificate information in the Authorization Server. This poses a potential drawback to this approach. However, if the client certificate is specified as a reference, such as a JWKS endpoint URL, then certificate rotation can be carried out without changing the client registration data in the Authorization Server.
An Authorization Server can advertise its support for OAuth2 mTLS by adding the self_signed_tls_client_auth entry to the server metadata definition.
The OAuth 2.0 Mutual-TLS Client Authentication and Certificate-Bound Access Tokens specification does not introduce new client registration parameters. Instead, it utilizes the “jwks” and “jwks_uri” parameters, as defined in the DCR specification. These parameters enable an OAuth2 client to furnish certificate details during the client registration process through the DCR endpoint.
- jwks — JSON Web Key Set of the client.
- Jwks_uri — The URL referring to the JSON Web Key Set of the client.
The OAuth 2.0 Mutual-TLS Client Authentication and Certificate-Bound Access Tokens specification also doesn’t provide recommendations on the deployment architecture for OAuth2 clients and Authorization Servers. However, addressing this practical aspect is crucial. We can primarily identify two deployment scenarios:
- Direct mTLS connection between the client and the Authorization Server
- The mTLS connection is terminated at a reverse proxy
Establishing an mTLS connection between the client and the Authorization Server is a straightforward approach, especially when using Identity as a Service (IDaaS) as the Authorization Server. This involves creating a direct mTLS connection between the client and the Authorization Server, relying on pre-configured TLS settings on both ends.
In the latter case, a reverse proxy, like Nginx, handles the mTLS handshake, managing the mTLS connection between the client and the reverse proxy, which acts as an intermediary or front-end for the Authorization Server. The mTLS connection is terminated at the reverse proxy, allowing clients with successful mTLS connections to access the Authorization Server. This setup is commonly used in on-premise deployments of Authorization Servers because mTLS configuration can be handled at the reverse proxy level, simplifying the deployment of the Authorization Server. Additionally, a single reverse proxy can distribute traffic among multiple instances of Authorization Servers. From a network security perspective, the reverse proxy is typically placed in a DMZ zone, while the Authorization Server is deployed in a restricted zone. Despite this separation, both the reverse proxy and the Authorization Server belong to the same trust domain, and logically, the reverse proxy can be considered an internal component of the Authorization Server deployment.
However, there is a challenge in this scenario: the Authorization Server doesn’t conduct the mTLS handshake, so the client certificate isn’t directly accessible to it. While the official specification doesn’t provide a specific solution for this, IAM vendors have implemented a simple approach that is becoming common across the industry.
Though the Authorization Server isn’t directly involved in the mTLS handshake, the reverse proxy acts on its behalf. To resolve the issue, the reverse proxy can forward the client certificate to the Authorization Server as an HTTP header after terminating the mTLS connection. The good news is that most modern reverse proxies can append the client certificate as an HTTP header for use in upstream applications. The specific name of the HTTP header is not standardized, so vendors use their own conventions. For example, WSO2 Identity Server defaults to the x-wso2-mtls-cert HTTP header, which can be customized during server configuration. The following diagram illustrates the complete solution.
In this post, we explored how authentication works in the mTLS protocol, and how that same mechanism can be applied to OAuth2 client authentication to enable a more secure and interoperable approach.
In Part 2, we’ll dive into how certificates can be used to bind a token — like an access token — to the intended recipient only, ensuring it can’t be misused if intercepted or leaked. This is a powerful technique that enhances trust between the authorization server and the client.
Stay tuned — it’s coming soon! 🚀