My kid at 5 wasn't too hard to handle. By 9... trouble. By 14... major trouble.
Small kids = small problems (only in a relative sense: some small kids have pretty big problems, but usually, problem size grows with kid)
We had to do our own research, ahead of the medical and school systems. WE had a pretty good idea of what we were dealing with. At least it has shaped our approach, even if it hasn't always been accepted by the medical side of things,
Here's some things to consider or research:
Sensory integration disorder / sensory processing disorder
Auditory processing disorder (in particular, auditory figure ground)
Social skills challenges
Sleep issues (some kids get enough hours of sleep, but it isn't good quality sleep, so they really are not rested; may be intermittent)
Anxiety (co-morbid with all sorts of other problems, or may stand on its own)
The fact that he does better in a smaller class size is what triggered the above list, just based on my own experiences. There may be other possibilities as well. But just for example, take a kid who has sensory processing disorder (SPD) and Auditory Processing Disorders (APD), and you tend to get anxiety. Take those three, and there are often quality of sleep issues. The four together = behavior problems. He may be overloaded on some front that you don't know about, rather than being a 'bad kid', which is what an ODD diagnosis tends to focus on.
