Cannot connect to same server as friend when in the same lobby

[Waltuh White]

Hi guys,me and my friend are trying to play and we can get in each others lobby but it wont let us connect to a server together, only the party host can connect, and i get kicked out the lobby with it saying the session is no longer available. Now i cant even connect to him in the lobby. Is there something i am doing wrong?