You are placing a lot of importance on two weeks out of the whole year. My brother in law doesn’t speak English and when he visits we all manage to smile and make jokes and include him while my sister translates a bit. It may be a bit awkward but it’s a short period of time and they will get used to it. It’s only two weeks and the rest of the year you will be together. Don’t rearrange your whole life around a family vacation.