My brother thinks my kids need a man in their life to 'toughen them up'? I am a single mom?

I think you did the right thing. He sounds like a major ist and a bully to me. Not the kind of male influence your kids need. He was disrespectful of you and your kids. You are being a great example of "tough". Raising kids alone AND standing up to a family member enough to remove him from your and your kid's life, sounds pretty tough to me. Keep up the good work and don't worry - your kids will do just fine!