Adding EF Core to a NET Standard project introduces transitive dependency versions incompatible with NuGet packages in other projects

I have a solution with multiple .NET Standard 2.0 projects.

One Project A uses the Google.Protobuf (3.11.2) NuGet package, that depends on

System.Memory (4.5.3)
    System.Buffers (4.4.0)
    System.Numerics.Vectors (4.4.0)
    System.Runtime.CompilerServices.Unsafe (4.5.2)

A few other projects also depend on System.Memory and all use the same dependency versions.

Another Project B uses Microsoft.EntityFrameworkCore (3.1.0) NuGet package that depends on

System.Memory (4.5.3)
    System.Buffers (4.5.0)
    System.Numerics.Vectors (4.5.0)
    System.Runtime.CompilerServices.Unsafe (4.7.0)

Even though the System.Memory version is (4.5.3) in both cases, it depends on System.Buffers, System.Numerics.Vectors and System.Runtime.CompilerServices.Unsafe and their versions differ.

When I run the application that uses these projects I get:

System.IO.FileLoadException: 'Could not load file or assembly 'System.Runtime.CompilerServices.Unsafe, Version=4.0.4.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference.

After searching for a solution I added this to my NuGet.Config:

<config>
    <add key="DependencyVersion" value="Highest" />
  </config>

In both, %appdata%\Nuget and in the root folder of the .sln file.

I also deleted the %userprofile%\.nuget\packages folder.

Then I removed the NuGet packages from the projects and added them back again, but their dependacies come with the same versions as before.

If I navigate to "Manage NuGet Packages for Solution..." in Visual Studio and choose "Consolidate" it just says "No packages found"