You can be mad in a prudent manner. Such lords should not act in such low behavior as shoving.
Although I quite agree that it is in their right to do, Lithor's talent is with words and he won't attempt to actually fight his superiors unless they showed some gross disrespect to his friends.
It is as a famous play writer once said:
The gentler gamester is the soonest winner.
Indeed, our characters make good points on which to be respected, but it is no fun when we only look at the dull, narrow, view of our characters. It is