This can be a difficult and painful thing. With me, I have to explain that we (my wife and I) have been letting my 31 year old daughter, her boyfriend, and their three children live in my house, for the past 11 years. Over the years, people have usually said things like "that's great of you to be so supportive." But, as the years go on, I think they are, understandably dumbfounded as to why we've let them stay so long (it is a long and exhausting story).
I have been in your situation before (before the kids came), when we'd kicked my daughter out, or the baby daddy, and he was living in a tent. But now, there are kids involved, I can't put them out on the streets. If it was just my daughter and her boyfriend, and we kicked them out (for good reasons) and they became homeless, I think I would feel justified, if someone asked me. I would give a brief rundown (a bullet list, so to speak) of the reasons, and if they still had a disapproving attitude, I would shrug it off and move on. You should be self-assured that you've done the right thing, as hard as it might have been. At some point we have to stop being affected by what others think about us.