I think the line is crossed when it becomes a matter of policy - a ban.
Otherwise, freedom of speech goes both ways - people can joke about anything, and other people can comment protest or whine about how those jokes are offensive. Who is right? Probably
orange, but that doesn't matter. The conversation goes both ways, and freedom of speech means neither get a right to have the last say. The freedom of people to be assholes is going to be intertwined with the freedom of people to call each other assholes.