Am I wrong??

A bit of background. My 20year old son was diagnosed with Asperger s when he was 4 however this was then retracted and I was told he just had 'quirky habits' and as a result he now goes undiagnosed and I am unable to approach the subject without him going mad. He left school at 12 and now rarely leaves his room, preferring to 'socialise' with online friends. He doesn't claim benefits or seek work and will make excuses and get angry if I try discuss this with him. Most of the time he is quite placid and has a gentle nature however does have a really bad temper, he is als depressed but would refuse to see a doctor. His father was quite abusive and he has had no contact with him for approx 10yrs, he carries a lot of anger about the past but again won't get help with this.  i have a partner of 8years and my son gets on great with him, however he doesn't live with us and only vistits 2-3 times a week.

Recently my son started being quite disrespectful towards myself and on occasions my partner, he will swear constantly which he previously was totally against and will get  really angry shouting and swearing at me usually in regards to the internet not working properly if we are using it at same time as him. My partner gets frustrated as he feels I am way too soft and should be challenging him on this behaviour and telling him it is unacceptable

Last week my son was watching sport with my partner and was quite disrespectful and inconsiderate the way he was talking and acting, I could see this was bothering my partner I tried discreetly to draw my sons attention to this however he was oblivious. Anyway in the end my partner raised his voice and said to my son "you're a disgrace the way you've behaved tonight, I'm disgusted and disappointed in you" as a result my son went completely mad, he started swearing at my partner calling him names and telling him to get out, he then threatened him physically and started throwing things around the house and knocking holes in his bedroom door. My partner said nothing and simply sat on the sofa, we both then left the house for my son to calm down. When we returned he had left and has now gone to live with his grandad. He has told me I have let him down as he feels what my partner said was verbally abusive and an attack on him and feels he provoked the situation deliberately, he feels I should have told my partner to get out and ended the relationship and because I didn't then he doesn't want any more to do with me. I am totally heart broken, I have tried reasoning with him and explaining that my partner wasn't abusive he simply criticised his behaviour however he feels I am making excuses and have made my choice. He has told me that he has felt suicidal since  because he feels he has lost his mum, his home and his brother is also annoyed at him, he has also seen gp and has been prescribed medication. I have told him I want him home and my partner has said he wants to talk and sort it but my son is refusing to do this. I am at a loss as I don't know what to do for the best, am I wrong not to end things with my partner? He is the most placid, easy going man ever and has bit his tongue on numerous occasions, he sees it that he has known my son long enough to call him out on his behaviour the same way as he would his own kids as he is the closest thing my son has to a father figure