When you are together as a couple with your children's other parent, you have a relatively large degree of control over the people your children are exposed to. However, after a divorce, or break up, it is typical for that control to decrease. It can be frustrating, as many parents new to this situation wonder if they can prevent their children from being in contact with certain people. Unfortunately, unless your ex agrees otherwise, the short answer is no, unless there is a significant risk of harm to the children from said exposure.
Why is this so? Shouldn't you, as a parent, have a right to say who your children should and should not be around? The answer is yes, but so does the other parent. For example, if your ex's mother is domineering, why can't you get a court order to block this woman from being around your children on your ex's time? Well, unless the children's grandmother presents some sort of potential danger or safety risk to the children, your ex is going to be able to decide who the children are going to be around on their time, just as you decide who the children are exposed to on your time.