DOCUMENT that the case carrier told him about the meeting. Send case carrier an email saying "to recap our conversation, you received a call from ex asking about difficult child's attendance. You told him the IEP meeting was at x date and Y time." Send a copy to yourself also.
Save that email in a file labelled something like "custody stuff" and also in "IEP stuff". When it comes to time for custody matters in court, make sure you have a copy and your atty knows of this ahead of time and has a copy. NEVER send ex a copy.
If ex wants custody, he needs to stop only doing visitation when it is convenient. If the court says he picks up and drops off, then he needs to do those things, not try to browbeat you into these things.
Your ex does NOT NOT NOT want the kids. He wants to control you and keep you upset because that feeds his ego, makes him think how great and powerful he is because he has you quaking at his demands. IF he takes custody, it won't last long. I think most states allow a child at either 12 or 14 to choose which parent to live with, as long as there is nothing grievously wrong with ehter parent. The child pretty much gets to choose,a nd to have a say in when visitation happens, or if it happens at all. I know my bro and parents are biding their time until niece is old enough to make some decisions, because here unless there is actual abuse, not just neglect, the only custody option is shared parenting which is 50/50. NOTHING else is an option. Heck, exsil take niece to school 1-3 hrs late and picks her up 1-3 hrs early because it isn't 'convenient' to wait through the line to get kids after school. Niece HATES this, loves school, and the court? Could not give less of a hoot. Even with documentation of this, and that niece NEVER has appropriate clothes or shoes or lunch if mom takes her, but on days that Dad or gma do? She has them all. Wth all of that? They still could not get a judge to even set a court date.
As long as you can get something from the doctor saying you are on the right medications, doing all he expects, being bipolar should NOT be a factor. School may be. I know you don't watn to hear this, but maybe going to ehr dads for a few weeks would change some things. She would see that she doesn't like it enough to stop being so hard on you if you allow her to come back. Ex would see that it is VERY hard to get her to go to school. Sure, for a week or two it would go smoothly, but then the honeymoon will end and he will stop wanting her around. That would give you a LOT of ammo in a custody hearing. "I tried, your honor. He wanted to see if he could help the school issues, but after a few weeks/a month, she still wouldn't go and he became abusive, screaming at her and threatening to hurt her. That didn't work, and made things worse,a nd now she has a panic attack if she thinks she is going to see him or have to go stay at his home."
THAT? WOuld settle this for good, most likely.
Mostly though, you NEED to figure out how to stop letting this man bully you. The kids are seeing his methods work, and are using them with you also. Kids don't want to live with him? Fine. They can do what they need to do, incl going to school and bathing daily. I would start using that to see if it would help get difficult child to school and clean.