Consume a local nuget package with RestoreAdditionalProjectSources and configure `package-directory` for cached nugets

I use CLI to pack and then consume my nuget package. For this target I use RestoreAdditionalProjectSources (in consumer's csproj file) that points to the directory that contains locally built nupkg files.

<RestoreAdditionalProjectSources>     $(PATH_TO_NUPKG_FILE)</RestoreAdditionalProjectSources>

all is good, I can consume my locally built package with: dotnet add package. In this case, the nuget cache is placed in "USERPROFILE/.nuget/packages" folder.

Now I want to store the cached nugets in some temporary folder that I can manage much easier than full global cache. For this I configure --package-directory %PATH% for dotnet add package command. It stores cache correctly, but then it fails with this error:

        info : X.509 certificate chain validation will use the default trust store selected by .NET.        info : X.509 certificate chain validation will use the default trust store selected by .NET.        info : Adding PackageReference for package 'BinarySource1' into project 'ConsoleApp1.csproj'.        info : Restoring packages for %PROJECT_PATH%\ConsoleApp1.csproj...        info : Installed Binary1 0.0.1 from $PACKAGES_PATH$ with content hash %HASH%==.        info : Package 'Binary1' is compatible with all the specified frameworks in project 'ConsoleApp1.csproj'.        info : PackageReference for package 'Binary1' version '0.0.1' added to file '%PROJECT_PATH%\ConsoleApp1.csproj'.        info : Generating MSBuild file %PROJECT_PATH%\obj\ConsoleApp1.csproj.nuget.g.props.        info : Writing assets file to disk. Path: %PROJECT_PATH%\obj\project.assets.json        log  : Restored %PROJECT_PATH%\ConsoleApp1.csproj (in 207 ms).        MSBuild version 17.5.0-preview-23061-01+040e2a90e for .NET          Determining projects to restore...        %PROJECT_PATH%\ConsoleApp1.csproj : error         NU1101: Unable to find package Binary1. No packages exist with this id in source(s): Microsoft Visual Studio Off        line Packages, nuget.org          Failed to restore %PROJECT_PATH%\ConsoleApp1.csproj (in 2.56 sec).        Build FAILED.

I assume, when I configure --package-directory, the dotnet binary doesn't "see" (?) the local source configured with RestoreAdditionalProjectSources. Is there any way to workaround this issue?