Need advice re: 4 year old twin son

Hi, I’m new to the forum and hope it’s ok to start a new thread?
I do apologise for the length of this post:

I’m very concerned about my 4 year old twin son, who has been physically aggressive towards me and his twin brother since he turned 3. On most days, usually more than once, he will punch me, or dig his nails into me if he thinks I haven’t heard him properly, or if I mispronounce something he’s said, or if he wants something he can’t have. Sometimes he’ll say ‘I want to hurt you mummy.’ He’s often bad tempered, complaining, argumentative and irritable, has frequent meltdowns, is a fussy eater and can be sensitive to noise, certain colours, and smells. He also has repetitive verbal rituals which are wearing me down. It seems to be worse when he’s tired/hungry. From a young age, if a stranger smiles and speaks to him, he will often just glare back and walk off.

He doesn’t behave this way at preschool and they’ve told me they have no concerns, however he doesn’t interact with other children and prefers to play alone or with his twin. His key person has told me that both he and his brother are ahead of their peers in terms of language and knowledge. He’s extremely bright, articulate and imaginative, and has an amazing memory. He can also be very affectionate and loving. He’s never had any learning difficulties or speech development problems. 

I raised my concerns with my GP six months ago, who advised contacting my health visitor. She came over and said that she didn’t have any concerns but suggested using a reward chart. Six months on and his behaviour is the same, if not worse, and there are times when I feel I can no longer cope with him. His brother doesn’t show any of these traits and I have no concerns about him - he is in general a very happy and gentle little boy who has endless patience with his difficult brother.

Should I be concerned about my son’s challenging behaviour? Could he be on the spectrum or is it something else? I’m taking him back to the GP on Monday for a referral - am I doing the right thing? My husband thinks I’m over-reacting, and while he acknowledges that our son is difficult, he refuses to accept that he might have ASD. (He hit the roof when a total stranger asked me if my son was on the spectrum a few months ago, following a meltdown in a cafe.) He thinks that because he’s fine at preschool but acts up with me then the problem must lie with me. This upsets me and is putting a strain on our marriage. However, my sister and mum think his behaviour is worrying and agree that he should have a referral.

I really need some help understanding why my sons behaves this way and how best to deal with it, as I often end up getting angry, yelling at him and shaking him. I’m ashamed to admit that I recently hit his arm after he punched me in the back, and I feel terrible guilt about this. I never ever thought I would hit my child. 

I would be extremely grateful to hear any advice and/similar stories, thank you.