Progressively slower looped database processing due to long-lived EF DbContext

Situation

I had a batch process, started from the command line, that loops over items in a file (serially, no parallelization) and calls our regular API service’s logic for each one, resulting in many small calls the database server. The data seeding process started out executing each record in the input file quickly, and then progressively slowed down until it just crawled, taking hours to finish a file with tens of thousands of records.

Diagnosis

At first, I guessed that the degrading performance could be due to throttling by our cloud hosting provider. However, if I stopped the long-running process and immediately started a new one, the second process would start out fast again, so the VPS as a whole had not exhausted its allotted resources. The database server load was very low. Compute CPU usage remained high for the life of the process. I did not see indications of memory pressure. The continually high compute CPU seemed to indicate that there was progressively more work to be done in order to complete each task.

Eventually I found this blog post by Rick Strahl, explaining that EF holds on to Change Tracking history for the life of the DbContext.

https://weblog.west-wind.com/posts/2014/Dec/21/Gotcha-Entity-Framework-gets-slow-in-long-Iteration-Loops#Recreate-the-dbContext

Per the EF docs, β€œa DbContext instance is designed to be used for a single unit-of-work. This means that the lifetime of a DbContext instance is usually very short.” My DbContext, in contrast, was injected once and persisted for the entire batch process.

In an ASP.NET Web application, dependency injection scopes are managed for us, where β€œthe framework creates a scope per request, and RequestServices exposes the scoped service provider. All scoped services are valid for as long as the request is active.”

Solution

By default, AddDbContext will provide the same DbContext for all dependency injection requests within the same scope: ServiceLifetime contextLifetime = ServiceLifetime.Scoped

//Startup

public void ConfigureServices(IServiceCollection services)
{
   services.AddDbContext<MyDbContext>();
   // ...
}

The change needed, then, is to create our own child scopes and repeatedly instantiate new copies of the service class that in turn depends on a DbContext.

// Usage when starting from the command line

IHost host = Host.CreateDefaultBuilder(args)
   .ConfigureWebHostDefaults(webBuilder => { webBuilder.UseStartup<Startup>(); })
   .build();

// ...

using (var rootScope = host.Services.CreateScope())
{
   IServiceProvider rootProvider = rootScope.ServiceProvider;

   foreach (var item in batch)
   {
      var childScope = rootProvider.CreateScope();
      // MyService eventually depends on a DbContext 
      var myService = childScope.ServiceProvider.GetRequiredService<MyService>();

      myService.Process(item);
   }
}