Why is always me he hits?

Hi guys, I am posting this as school are getting concerned about my sons violence, he has high functioning autism, traits of spd and Hypercussis, our son has always been violent and always had meltdowns, he has been suffering school anxiety, but most times its me he hits when he lashes out, ok sometimes his brother, but its always me that gets the brunt of it, does anyone know why? so I can explain this to school? I do not want them thinking its something I am doing.