There's an easy solution - for low volumes, I turn the amp volume up to 25% (normally quite loud) and decrease the volume from the input (chromecast audio or phone in my case).