Visual Studio auto updates references

I'm working on a .NET application with multiple projects. Whenever I install a NuGet package reference to a specific project, Visual Studio auto adds references to System.ComponentModel.Annotations and Newtonsoft.Json into other project files in the background. How I can stop this? I've been struggling to find if there is a specific setting or why this is happening in the first place as I do not want these references and I've to manually discard changes.