I did some more testing, thanks to having numerous home office meetings these days :-)
The call volume adjustment does in fact work do something. To me it seems to adjust the volume between 60 and 80%. Obviously, its not as noisy as before, but it still won't adjust below 60%. And missing a setting above 80% proved to be an issue during a conference call, when I could barely hear anything.
I also noticed that the phone speaker struggled a lot with low frequencies. This made things much worse and should be taken care of by an equaliser. Human (male) voices do not transmit information