I never want to turn it on, but I do swap from shuffle to regular, which means I have to click through smart shuffle now too.

Which wouldn’t be a huge problem, but if you do it too fast, smart shuffle appears to be off, but the songs it has added are at least partially still in the queue.