And replying to myself, Polk says that Authy uses your phone number!?!
https://www.youtube.com/watch?v=YBVCn7q_fSI 08:15
https://www.youtube.com/watch?v=YBVCn7q_fSI 08:15
They use it for some stuff, but it doesn't appear to involve the actual keys, so should be fine.