Non verbal son hitting.

Has anyone advice for my 4yr old son who is 5 in march. He hits mainly me or walls, very rarely hits his dad or brothers. Have tried ignoring him, taking things off him, explaining that it hurts mummy and is unkind. Is really starting to upset me, as it seems directed at me, I know its not personal but can't help sometimes thinking it is. Any advice welcome. Thank you x