Update:
Somehow loading into a map solo and quitting (I also died but not sure if necessary) fixes the freezing (and sometimes the server list), but I still got error for trying to join my friend.
Then I suggested that he switched to party open instead of invite only and somehow that fixed it and I could join him.
undefined