We were trying to create a new ASP.NET Core 2.1 application that uses AzureAD for authentication and were getting some errors when hosting in AWS. So I decided to take a new, stock-template project and deploy that out to verify the simplest setup would work.
AzureAD authentication just worked when using APIGateway as the HTTP frontend for Lambda. Lambda via an Application Load Balancer (ALB), however, was a different story.
ALB-fronted Lambda is a new feature that was released late last year. In February (2019), AWS released version 3.0 of the AspNetCoreServer in the AWS .NET SDK which added support for leveraging ALB with a .NET project.
ERROR 1 – Response type of application/octet-stream response
This one was pretty simple to figure out and was just a dumb mistake. You have to change your Lambda base class from APIGatewayProxyFunction to ApplicationLoadBalancerFunction. Otherwise the Lambda responds with an object that APIGateway should parse and transform to the appropriate response instead of an object that is passed through ALB.
ERROR 2 – Correlation failed
This one took a bit longer to find.
‘.AspNetCore.Correlation.AzureADOpenID.Nsn7U4c5ARcr2RZ4W4XQJRq5qPWyi7InV99lnu_JUNg’ cookie not found.
[Error] Amazon.Lambda.AspNetCoreServer.APIGatewayProxyFunction: Unknown error responding to request: Exception: System.Exception: An error was encountered while handling the remote login. —> System.Exception: Correlation failed.
After much searching across Github issues, source code spelunking and blog posts I found that only one cookie (OpenIdConnect.Nonce) was being set on the page that requires authentication/authorization and kicks off the OpenId challenge workflow. I reached out to Chris R (@Tratcher), the main Microsoft developer who works on OpenId for ASP.NET, and he helped narrow down the issue. He clarified that if one cookie is added the other should as well. This reminded me of a post I had read about the AWS .NET SDK adding support for multi-header values in APIGateway and ALB. The idea here is that only the first OpenId cookie was being added because they were using multi value headers but that wasn’t enabled in ALB.
Sure enough I found and enabled the setting under the ALB target group and everything started working correctly.
ERROR 3 – Intermittent authentication failures
If you are running multiple instances of your ASP.NET application in Lambda you may notice login failures occasionally, especially during high load. You will see the following warning in CloudWatch when the Lambda starts up:
[Warning] Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager: Neither user profile nor HKLM registry available. Using an ephemeral key repository. Protected data will be unavailable when application exits.
This is telling us that certificate keys will be stored in memory instead of in the registry. This is a problem when we have multiple Lambda instances running because each will try to decrypt using a different key. This will cause failures. I usually saw the same correlation failure error above.
Thankfully, the AWS .NET team has this covered with the Amazon.AspNetCore.DataProtection.SSM Nuget package which provides DataProtection through storing keys in SSM Parameters. To set this up add the Nuget package, configure DataProtection in your startup (see the Github page for directions) and ensure the role for your Lambda has access to create/read SSM parameters.
Here is the policy we used:
{
“PolicyName”: “SSMDataProtection”,
“PolicyDocument”: {
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: [
“ssm:PutParameter”,
“ssm:GetParameter”
],
“Resource”: {
“Fn::Sub”: “arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/LambdaDataProtection/${stackName}/Acme.WebApp/*”
}
}
]
}
}
So in the end we got AzureAD working well with Lambda via both APIGateway and ALB. Hopefully this post helps others that are having similar problems. Please comment below if the post was helpful, you have questions or further insights to add.
Happy Serverless Coding!