Dotnet on ESG's

Using Cloud SQL IAM Authentication with PostgreSQL on .NET

Tue, 28 Mar 2023 19:15:26 -0400

Google Cloud Platform supports automatic IAM authentication for databases, but .NET is not officially supported.

Thankfully, Npgsql, a ADO.NET Data Provider for PostgreSQL, has enough features to make it easy!

Create and configure a service account

Then, assign it the roles/cloudsql.instanceUser on your Cloud SQL instance. This can be also be done at the project level, but it’s generally better to keep your scopes tighter.

Lastly, add the service account as an IAM user to your database instance

Note: This would also work with a user account, but we’ll be using service accounts to keep things simple

Once all of this is done, your service account is ready to go

Write the code

We’ll be using a NpgsqlDataSourceBuilder, with the UsePeriodicPasswordProvider property in order to refresh our GCP access token.

You will need to install the following Nuget packages in your project:

We first start by readying our GoogleCredential.

// You could load this from a key file, or anywhere else, but this should cover most cases.
var credentials = await GoogleCredential.GetApplicationDefaultAsync();

// We want to limit the scope of the token, so we create scoped credentials 
var scopedCredentials = credentials.CreateScoped("https://www.googleapis.com/auth/sqlservice.login");

Next, we need to setup a NpgsqlDataSource using NpgsqlDataSourceBuilder.

Conveniently enough, Google auth will take care of refreshing the token on demand. If the token is not expired, GetAccessTokenForRequestAsync will return a cached version. This means that we can keep successRefreshInterval and failureRefreshInterval pretty short (the latter should never happen). In this case, we use 1 minute and 0 minutes respectively.

var dataSourceBuilder = new NpgsqlDataSourceBuilder();
dataSourceBuilder.UsePeriodicPasswordProvider(
 (settings, cancellationToken) => new ValueTask<string>(scopedCredentials.UnderlyingCredential.GetAccessTokenForRequestAsync(cancellationToken: cancellationToken)),
 TimeSpan.FromMinutes(1),
 TimeSpan.FromSeconds(0));

dataSourceBuilder.ConnectionStringBuilder.Database = "<your database name>";
dataSourceBuilder.ConnectionStringBuilder.Username = "<Your iam user. ex: name@project.iam>";
dataSourceBuilder.ConnectionStringBuilder.Host = "<The IP address of your database>";

var dataSource = dataSourceBuilder.Build();

All that’s left is to create the connection and query the database.

using var connection = dataSource.CreateConnection();
await connection.OpenAsync();

using var cmd = connection.CreateCommand();
cmd.CommandText = "SELECT 1";

var result = cmd.ExecuteScalarAsync();
Console.WriteLine(result);

And that’s it, you now have secure, password-less access to your PostgreSQL server.

What about Entity Framework?

Good question! It turns out that Npgsql.EntityFrameworkCore.PostgreSQL supports passing in a DataSource out of the box.

In an ASP.NET setting, it would look like this:

builder.Services.AddDbContext<Context>(options =>
{
 options.UseNpgsql(dataSource);
});

Want to register your DataSource for dependency injection? Sure!

builder.Services.AddSingleton(dataSource);

builder.Services.AddDbContext<Context>((serviceProvider, options) =>
{
 var dataSource = serviceProvider.GetRequiredService<NpgsqlDataSource>();
 options.UseNpgsql(dataSource);
});

And that’s all there is to it.

Deferred processing in ASP.NET Core 6

Sun, 17 Jul 2022 18:01:55 -0400

Nowadays, a common pattern we see when treating HTTP requests is to do a minimal amount of work before returning a response, while treating longer operations (ex: sending email) in the background after the request has been completed.

Unfortunately, in ASP.NET, we often see this being done in questionable ways, mainly:

Not using await on Task object
Wrapping some code in Task.Run and not awaiting the results.

This can lead to dependency injection issues (if your request ends, all of our scoped objected are gone), or having exceptions thrown into the wind.

Fortunately, .NET has it’s own internal publisher/subscriber-like feature, called Channels.

Channels

Channels were first introducted in .NET Core 3.0. While they’ve had a few improvements since, the abstract classes were mostly unchanged.

The basics

A lot has been written about how to usage channels, so I won’t go in too deep here.

First, let’s go over the building blocks:

Channel: Static class used to create channels
Channel<T>: Abstract class representing a channel
ChannelReader<T> and ChannelWriter<T>: Abstract classes representing readers and writers to a channel

A simple setup would be:

var channel = Channel.CreateUnbounded<Model>();
var reader = channel.Reader;
var writer = channel.Writer;

Integration with ASP.NET Core

We’ll be assuming that we want to process a model named, appropriately, Model.

Background service

First, let’s created a HostedService to receive the messages.

We’ll be inheriting from BackgroundService since it makes the implementation a little simpler.

public class ModelService : BackgroundService
{
 private readonly ChannelReader<Model> _channelReader;
 private readonly ILogger<Service> _logger;

 public ModelService(ChannelReader<Model> channelReader, ILogger<Service> logger)
 {
 _channelReader = channelReader;
 _logger = logger;
 }

 protected override async Task ExecuteAsync(CancellationToken stoppingToken)
 {
 while (await _channelReader.WaitToReadAsync(stoppingToken))
 {
 while (_channelReader.TryRead(out var item))
 {
 _logger.LogInformation("Message is `{Message}`", item.Message);
 }
 }
 }
}

Pretty simple, right?

We pass a ChannelReader<Model> to the constructor

In the first while, we wait for a message to be available in the channel. When there’s one, we enter the second while to read every messages available until exiting.

We could simplify this into a single while, but this version is a little more performant since it always to get multiple messages from a channel without having to await. More information on An Introduction to System.Threading.Channels

Scoped services

Note that the BackgroundService here is a singleton, meaning that it won’t be able to access scoped services, such as EntityFramework Core’s DbContext, which is often registered as scope.

To get around that, you should inject an IServiceProvider into your background service. Depending on performance consideration, I’d typically start with creating a new scope per incoming message, and looking into batching if there’s an issue.

Wiring the dependency injection

We start by wiring the channel itself, as long as its reader and writer.

Note that I’m using minimal API syntax here, but it should be easy to adapt to the older style syntax like ASP.NET Core 5.

builder.Services.AddSingleton(Channel.CreateUnbounded<Model>());
builder.Services.AddSingleton(p => p.GetRequiredService<Channel<Model>>().Reader);
builder.Services.AddSingleton(p => p.GetRequiredService<Channel<Model>>().Writer);

CreateUnbounded means that there won’t be any item limits on the channel, other than the amount of memory available to your application.

then, we wire the service.

builder.Services.AddHostedService<ModelService>();

Setting up the route

We then setup a route to receive the messages. To make things easier, I made one that receives an array.

app.MapPost("/messages", async (ChannelWriter<Model> writer, ICollection<Model> models, CancellationToken cancellationToken) =>
{
 foreach (var model in models)
 {
 await writer.WriteAsync(model, cancellationToken);
 }
 return StatusCodes.Status202Accepted;
});

Testing it

You can try it by making a POST to /Messages with a payload such as

[
 { "Message": "hello world" },
 { "Message": "hello too" },
 { "Message": "hello three" }
]

If you want to make sure that the request returns before the messages are processed, you can add a Task.Delay in ModelService after each log message. The messages will slowly scroll in your application’s console, long after the request has completed.

Parting words

This was a very simple demo of channels with ASP.NET Core. You can find a working demo at https://github.com/EricStG/AspNetCoreChannels.

If you’re considering using channels in your application, consider spending some time configuring your channels. For example, I’d look at whether I should use a bounded or unbounded channel, and if I’ll be using a single reader or writer. Having it properly configured will improve performance and avoid unplanned nastiness.

Editorconfig integration with Visual Studio and .NET

Sun, 21 Feb 2021 19:01:55 -0500

If you are not familiar with EditorConfig files, you’re missing out

Today, I’ll walk you through some of the basic features, as well as some extensions to the format supported by Visual Studio and the .NET tools.

The basics

Editorconfig is a way to define some coding styles by adding files called .editorconfig to your code repository. There’s pretty wide support from editors out there that supports it either natively, like Visual Studio, or through add-ons, like Visual Studio Code.

Hierarchy

Editorconfig us a hierarchical configuration for code style, meaning that when a code file is open, it will apply every settings from .editorconfig files it finds, from the file’s folder, up to the root of the file system.

The closer to the code file the config is, the higher the priority.

You can make prevent scanning parent directories by specifying root = true in your .editorconfig file.

Syntax

Editorconfig files are split into sections. Each section starts with a file selector, as determined by square brackets [].

This selector can apply to every file like [*], a simple wildcard like [*.json], or a list of extensions like [*.{cs, vb}]

Consider the following example:

root = true

[*]
indent_style = space

[*.{cs,vb}]
indent_size = 4
insert_final_newline = true
charset = utf-8-bom

First, we have root = true, meaning that we won’t be scanning any parent folders for more .editorconfig files.

Then, we have a [*] block that applies to every file, saying we will be using space to indent.

And last, we have the [*.{cs,vb}] block that will apply to all files with a .cs or .vb extension. Those files will have a indent of 4 (using spaces since the content of the [*] block is still being applied), we want to ensure the presence of an empty line at the end of each file, and we are using the UTF-8 character set with the byte order mark.

Tip: If using git, you will want to make sure that your gitattribute don’t conflict with your .editorconfig settings, otherwise your editor might trying to undo what git is doing on checkout, causing a bunch of changes that aren’t wanted.

.NET specific extensions

With the Roslyn compiler, there’s a number of extensions to the Editorconfig format that allows to configure code analysis.

For example, you can use Editorconfig to tell whether using clauses should be sorted with the System namespace first

[*.{cs,vb}]
dotnet_sort_system_directives_first = true

You want contributors to use var for C# built-in types? Done!

csharp_style_var_for_built_in_types = true

You can even make it an error:

csharp_style_var_for_built_in_types = true:error

You can also configure casing, as well as prefix and suffix. This will give a warning for any fields that are not camel case, and not prefixed with an underscore _.

dotnet_naming_rule.instance_fields_should_be_camel_case.severity = warning
dotnet_naming_rule.instance_fields_should_be_camel_case.symbols = instance_fields
dotnet_naming_rule.instance_fields_should_be_camel_case.style = instance_field_style
dotnet_naming_symbols.instance_fields.applicable_kinds = field
dotnet_naming_style.instance_field_style.capitalization = camel_case
dotnet_naming_style.instance_field_style.required_prefix = _

See the official documentation for more options. The [Roslyn .editorconfig] also has number of good examples.

Configuring code analysis

If you’re familiar with Visual Studio and Code Analysis, you might have had to configure it through xml via .ruleset files.

That’s no longer necessary, as it can be done through Editorconfig!

All you need is a line that reads dotnet_diagnostic.<Code>.severity = <Severity>, such as:

[*.{cs,vb}]
dotnet_diagnostic.RCS1007.severity = error

Conclusion

Whenever you a small or a large team, having common code styles in your project can help minimize head scratching when reading a teammate’s code (or let’s be honest, your past self’s code as well). With Editorconfig, you have a standard way that’s not tied to specific technologies to help make those styling standard happens.

What coding styles do you apply to your project?

Using HttpClient with System.Text.Json

Fri, 22 Jan 2021 17:01:55 -0500

When working with JSON and HttpClient in the .NET Framework, it was pretty common to add a reference to Microsoft.AspNet.WebApi.Client, which added a number of extension methods to HttpClient and HttpResponseMessage that made it simpler to send and receive JSON documents.

That package still work with .NET Core, but it has a dependency on the Newtonsoft.Json. Since .NET Core 3+ comes with System.Text.Json, a more async friendly JSON parser, it might be worth making the switch, espcially in newer projects.

Conveniently, there’s also a package called System.Net.Http.Json that includes extension methods to make your serialization and deserializion easier.

Microsoft.AspNet.WebApi.Client

This section will show a few calls made with Microsoft.AspNet.WebApi.Client.

The assumption is that you have an HttpClient already instanciated with the name client, a request model called RequestModel, and a response model called ResponseModel.

A basic GET

First, we get a HttpResponseMessage from the client by making a request.

HttpResponseMessage response = await client.GetAsync("/");

Then, we using the generic verion of the ReadAsAsync<T> extension method to read and deserialize the JSON document into our object.

Task<ResponseModel> responseModel = await response.Content.ReadAsAsync<ResponseModel>();

A POST with a request document

Sending a document is also pretty straightforward.

First, we initialize our request model.

RequestModel requestModel = new RequestModel();

Then, we make our request, including our model that will get serialized through the PostAsJsonAsync extension method.

HttpResponseMessage response = await client.PostAsJsonAsync("/", requestModel);

Finally, we read the response model, just like we did for the GET.

ResponseModel responseModel = await response.Content.ReadAsAsync<ResponseModel>();

System.Net.Http.Json

Here, we’ll be doing the same exact thing, but using the extension methods in System.Net.Http.Json instead.

A basic GET

Again, we get a HttpResponseMessage from the client by making a request. This is the same as for Microsoft.AspNet.WebApi.Client.

HttpResponseMessage response = await client.GetAsync("/");

Then, we using the generic verion of the ReadFromJsonAsync<T> extension method to read and serialize the JSON document into our object.

ResponseModel responseModel = await response.Content.ReadFromJsonAsync<ResponseModel>();

Note that if you don’t need to do any processing on the HttpResponseMessage, there’s also a convenience method called GetFromJsonAsync so you can skip that step entirely.

ResponseModel responseModel = await client.GetFromJsonAsync<ResponseModel>("/");

A POST with a request document

Sending a document is also pretty straightforward.

First, we initialize our request model.

RequestModel requestModel = new RequestModel();

Then, we make our request, including our model that will get serialized through the PostAsJsonAsync extension method, which conveniently has the same name as the extension method from Microsoft.AspNet.WebApi.Client.

HttpResponseMessage response = await client.PostAsJsonAsync("/", requestModel);

Finally, we read the response model, just like we did for the GET.

ResponseModel responseModel = await response.Content.ReadFromJsonAsync<ResponseModel>();

Final notes

While serializing and deserializing documents with the HttpClient isn’t particularly challenging, it does lead to a fair amount of repetition, so using these extension methods (or rolling your own!) should make your code look a lot leaner.

Serilog Do's and Don'ts

Sun, 12 Jul 2020 18:08:20 -0400

Serilog is a structured logger for .NET, and as such, for folks who are used to traditional console loggers, requires the developer to think about their logs a little more than normal.

This post is not meant to be a how to, but rather a non-comprehensive list of mistakes I’ve seen over time.

Templating

One thing to keep in mind is that when writing an event to the log, you are not passing a message, but a template. The tags you pass within curly branches do have meaning, and will be used to add data the event. It is therefore important to not format the message before passing logging it.

Consider a logger with a basic console sink

var textLogger = Log.Logger = new LoggerConfiguration()
 .WriteTo.Console()
 .CreateLogger();

With two very basic statements

textLogger.Information($"Hello {world}");
textLogger.Information("Hello {value}", world);

Will output:

[18:19:18 INF] Hello World
[18:19:18 INF] Hello World

Can you spot the difference? Spoiler: There isn’t one! So what’s the problem then?

Well, let’s use the JSON Formatter instead:

(note: I’m using the Rendered formatter so we can see the rendered template within each event)

var jsonLogger = Log.Logger = new LoggerConfiguration()
 .WriteTo.Console(new RenderedCompactJsonFormatter())
 .CreateLogger();

With the same statements

jsonLogger.Information($"Hello {world}");
jsonLogger.Information("Hello {value}", world);

Will give us:

{
	"@t": "2020-07-12T22:26:19.6616913Z",
	"@m": "Hello World",
	"@i": "00ba5a14"
}
{
	"@t": "2020-07-12T22:26:19.6659300Z",
	"@m": "Hello \"World\"",
	"@i": "40c34e37",
	"value": "World"
}

When using textLogger.Information("Hello {value}", world), there is now a field called value with the argument we passed to the logging statements. In addition, the @i field will have the same value every time this statement will run, so you have a fixed item to search for if you want to search for every value being logged by this particular statement.

TLDR

DON’T format messages before passing them to the logger

Destructuring

One of the benefit of having a structured logger versus a flat one, is, well, structure. You are not limited to logging simple values, you can include entire object graphs. This process is called Destructuring.

Let’s consider the following classes

public class TestInnerClass
{
 public string SomeOtherProperty { get; set; }
}

public class TestClass
{
 public string SomeProperty { get; set; }
 public TestInnerClass SomeInnerClass { get; set; }
}

Let’s initialize an instance

var obj = new TestClass
{
 SomeInnerClass = new TestInnerClass
 {
 SomeOtherProperty = "I am a property within a child"
 },
 SomeProperty = "I am a property at the root level"
};

And let’s include it a logging statement

jsonLogger.Information("Useful log for object {object}", obj);

This will output

{
	"@t": "2020-07-12T22:43:28.1191964Z",
	"@m": "Useful log for object \"SerilogTest.TestClass\"",
	"@i": "146d94ee",
	"object": "SerilogTest.TestClass"
}

What? SerilogTest.TestClass? By default, Serilog will simply call ToString on your object. To destructure it, simply add a before the field in your template

jsonLogger.Information("Useful log for object {@object}", obj);

And the output will be

{
	"@t": "2020-07-12T22:43:28.1525906Z",
	"@m": "Useful log for object TestClass { SomeProperty: \"I am a property at the root level\", SomeInnerClass: TestInnerClass { SomeOtherProperty: \"I am a property within a child\" } }",
	"@i": "99aea052",
	"object": {
		"SomeProperty": "I am a property at the root level",
		"SomeInnerClass": {
			"SomeOtherProperty": "I am a property within a child",
			"$type": "TestInnerClass"
		},
		"$type": "TestClass"
	}
}

As you can see, the entire object graph was included within the event. So much contextual information!

But what if there are properties that you do not want to log? After all, it’s typically frowned upon to include information like passwords or personally identifiable information within them.

Let’s reconfigure the logger to only log SomeProperty:

var jsonLogger = Log.Logger = new LoggerConfiguration()
 .WriteTo.Console(new RenderedCompactJsonFormatter())
 .Destructure.ByTransforming<TestClass>(c => new { c.SomeProperty })
 .CreateLogger();

Will give us

{
	"@t": "2020-07-12T22:49:44.7386801Z",
	"@m": "Useful log for object { SomeProperty: \"I am a property at the root level\" }",
	"@i": "99aea052",
	"object": {
		"SomeProperty": "I am a property at the root level"
	}
}

We only included the property we wanted!

Note that there are different ways to configure destructuring. A good start in the (awesomely named) Destructurama! organization.

Type conflicts

Just a side note about destructing objects regarding conflicting types. Some log aggregating services will use the parameter names within templates to define types.

An example of this is Elasticsearch.

With the following statement

jsonLogger.Information("Hello {value}", "world");
jsonLogger.Information("Hello {value}", new { value = "world" });

Since the first statement will generate a string field called “value” in the mapping, the second statement will not be logged, as Elasticsearch will complain about trying to fit an object within a string field.

TLDR

DO include object graph when they add contextual information to your statements AVOID including unnecessary objects. They can clutter your logs, and if you’re pushing them to external services that charges by the size of logs you send them, it will quite literally cost you. DO Be mindful of conflicts that can happen with your parameter names

Exceptions

The one thing to remember is that Serilog’s methods all have an overload that accepts an Exception as the first parameter.

So instead of writing statements including the exception as part of the message template

var exception = new Exception("Top exception", new Exception("Inner exception"));
jsonLogger.Information("Exception! {exception}", exception);

Which will include the exception within the exception parameter

{
	"@t": "2020-07-12T23:01:14.1165769Z",
	"@m": "Exception! \"System.Exception: Top exception\r\n ---> System.Exception: Inner exception\r\n --- End of inner exception stack trace ---\"",
	"@i": "e22b2a9a",
	"exception": "System.Exception: Top exception\r\n ---> System.Exception: Inner exception\r\n --- End of inner exception stack trace ---"
}

Simply include the exception as the first argument

var exception = new Exception("Top exception", new Exception("Inner exception"));
jsonLogger.Information(exception, "Exception");

And now all logged exceptions will be within the @x field.

{
	"@t": "2020-07-12T23:01:14.1532597Z",
	"@m": "Exception",
	"@i": "7d53da39",
	"@x": "System.Exception: Top exception\r\n ---> System.Exception: Inner exception\r\n --- End of inner exception stack trace ---"
}

Note that this behaviour can change depending on your sink. For example, the Elasticsearch sink will include the exceptions within an object array.

TLDR

DO use Serilog’s overloads to pass exceptions instead of using the argument list.

Conclusion

Those were just a few basic things to keep in mind when using Serilog. I stringly suggest visiting the Writing log events page for more information.

Introduction to Action Constraints in ASP.NET Core

Mon, 02 Dec 2019 18:45:08 -0500

Ever had to integrate a 3rd party webhook into your application, only to realize that they won’t let you configure a different address for each type of events?

A few years ago, I had to do just that for Github webhooks. Thankfully, they provide a HTTP header that describe the type of event being sent.

At the time, I used a middleware to modify the route, but there are cleaner solutions out there. In this case, I could’ve used a feature in ASP.NET Core called “Action Constraints”, which allow you to impose constraint on otherwise identical routes.

Note: This is a follow-up on a question I had a few years ago on StackOverflow, and I decided to dig a little deeper.

Action Contraint Attribute

ASP.NET Core has an interface called IActionConstraint that can be implemented by an Attribute class, which has only 2 members:

Order, which will determine what order your constraint will be evaluated on (lowest number first)
Accept(ActionConstraintContext), which needs to return a boolean that will tell the framework whether to use this route

Let’s see what an implementation can look like:

public class HeaderConstraintAttribute : Attribute, IActionConstraint
{
 public int Order => 0;
 private string Header { get; }
 private string Value { get; }

 public HeaderConstraintAttribute(string header, string value)
 {
 Header = header;
 Value = value;
 }

 public bool Accept(ActionConstraintContext context)
 {
 if (context.RouteContext.HttpContext.Request.Headers.TryGetValue(Header, out var value) && value.Any())
 {
 return value[0] == Value;
 }

 return false;
 }
}

Simple enough, right? Look for a matching Header in the HttpContext, and if present, match it with Value.

Now, let’s update our controller.

[Route("api/[controller]")]
[ApiController]
public class ActionController : ControllerBase
{
 [HttpGet]
 [HeaderConstraint("MyHeader", "A")]
 public string GetStringA()
 {
 return "A";
 }

 [HttpGet]
 [HeaderConstraint("MyHeader", "B")]
 public string GetStringB()
 {
 return "B";
 }

}

There are 2 things to notice here:

Both functions are HTTP Get, to the same route (/api/action in this case)
We’ve added the HeaderConstraintAttribute on each route

Now, try using your HTTP API tool of choice, such as Postman or Insomnia, and do a GET request to /api/action.

You should be getting a 404 Not Found error, which makes sense, since both routes are contrained, and neither are matching the contraint.

Now, try passing a header called MyHeader with the value A.

Surprise! The API returns A. Change the header value to B, the API now returns B.

This is obviously a very simple scenario, but when dealing with 3rd party APIs that sends different requests to the same address, it’s a great way to keep the routes seperated, use strong-typed classes and avoid using things like dynamic variables.

You can view the entire code written for this project on GitHub.