Try limiting it down to 2GB (there is an option in the Docker Desktop app). Before I discovered this option, the VM was normally eating 3-4GB of my memory.