Discord has terrible defaults and apparently you need to set them again every time you install it.
Windows Settings:
[ ] Open Discord on Startup
[ ] Close Button Minimized to Tray
Data & Privacy:
[ ] Use data to improve Discord
[ ] Use my Discord activity to personalize Sponsored Content
Notifications:
Sounds:
[x] Disable All Notification Sounds
Streamer Mode:
[ ] Enable Streamer Mode
[ ] Automatically Enable/Disable
Game Overlay:
[ ] Currently Playing
[ ] Enable Overlay
[ ] Enable Legacy Overlay
Voice & Video:
[x] Push to Talk
and then actually set a key.