To remedy this, you'll probably need to adjust your Xbox privacy settings if you're using the Bedrock Edition of Minecraft. On this site, sign in to your Xbox account and ensure that "Others can interact with voice, text, or invitations" is set to "Everyone.