“Everything was working great — until it wasn’t.”

That’s how most DevOps stories begin, and this one’s no different.

We had many services with successful AKS deployment, running over Linux and Windows containers based on the requirements. Many of the services aren’t exposed externally. Instead, they communicated through Kubernetes headless services. Fast, efficient, and internal-only — exactly how we liked it.

But then came security reviews. The dreaded words were uttered:

"We need mTLS between these services."

🕳️ Down the Mesh Hole

First, we tried Istio Ambient Mesh. It’s the buzzword these days. No sidecars, just transparent magic. But no love for Windows containers — not yet.

Next up, Linkerd. Elegant, simple. Also: Linux-only.

Then came Dapr. The promise was sweet: built-in mTLS, and Windows support! But here’s the kicker — it doesn’t work with headless services.

🔐 How Dapr Handles mTLS

Dapr enables mTLS using sidecar-to-sidecar authentication and encryption. Here’s what happens when Service A calls Service B:

1. Service A calls Dapr’s local API: http://localhost:3500/v1.0/invoke/service-b/method/foo
2. Dapr sidecar for Service A resolves service-b and opens a mutual TLS connection with its sidecar
3. Certificates are rotated automatically and identities are verified using SPIFFE IDs

Result: full mTLS with zero app-layer effort — but only if you use Dapr’s API.

🚫 Why Dapr Doesn’t Work with Headless Services

Headless services resolve directly to pod IPs. When you use:

https://my-service-0.my-service-headless.default.svc.cluster.local:5001

You're talking directly to the app, skipping the Dapr sidecar entirely. That breaks:

• Request interception
• Retries and observability
• mTLS enforcement

Dapr cannot inject itself into direct pod-to-pod TCP/IP connections.

✏️ Code Changes Required for Dapr mTLS

1. Add Sidecar Annotations

annotations:
  dapr.io/enabled: "true"
  dapr.io/app-id: "my-app"
  dapr.io/app-port: "5000"

2. Replace HTTP Client Calls

Instead of calling other services over DNS/IP, use:

POST http://localhost:3500/v1.0/invoke/my-other-app/method/do-work

3. Use Dapr SDK (Optional but clean)

var client = new DaprClientBuilder().Build();
var result = await client.InvokeMethodAsync<Input, Output>("my-other-app", "do-work", input);

✅ Alternate approach

Chose a full app-layer TLS approach in .NET — using certificates and enforcing validation at runtime.

Client-side in .NET

var handler = new HttpClientHandler();
handler.ClientCertificates.Add(new X509Certificate2("client.pfx", "password"));
var client = new HttpClient(handler);

Server-side with Kestrel

webBuilder.ConfigureKestrel(serverOptions =>
{
    serverOptions.ConfigureHttpsDefaults(httpsOptions =>
    {
        httpsOptions.ServerCertificate = new X509Certificate2("server.pfx", "password");
        httpsOptions.ClientCertificateMode = ClientCertificateMode.RequireCertificate;
        httpsOptions.ClientCertificateValidation = (cert, chain, errors) =>
        {
            return cert.Issuer == "CN=MyInternalCA";
        };
    });
});

🎯 Final Thoughts

If you're using Windows containers and headless services, service mesh solutions may not save you. Dapr works — but only if you go through its APIs.

Sometimes, engineering means getting your hands dirty with certs, handlers, and validation callbacks. That’s what makes it fun.

If you dont win with tools — You win with code.