Newbie with relationship questions

I have been with my partner for the last 10 years. He is undiagnosed but I am 100% sure is aspie although I’m  also pretty sure he would deny it himself. I’ve been aware of his traits for the entire duration of our relationship and in the main, we get along great. He is very high functioning, clever, sweet, kind and creative and I believe our relationship has been generally fulfilling for both of us. I continue to struggle, however, with lack of physical contact. Only on a handful of occasions has he initiated a hug or a kiss himself. 

I got overwhelmed and upset due to feelings of rejection about 4 years ago and we separated for 6 months. There were issues going on in my life and I felt that he wasn’t able support me nor give me the comfort I was needing at that time. Despite this, I always knew that he loved me and we got back together. He struggles so much with expressing his emotions and in spite of that he agreed to go to relationship counselling with me which made me realise how much he wanted to make it work.  Plus I simply missed him so much when we were apart.

At that time I decided that if I needed a hug I would just hug him if I wanted, regardless of whether he reciprocated. I’ve carried on that way since then and I believe we’ve been pretty good since. Although I have had spells of feeling sad about us not being a ‘normal’ couple they’ve passed pretty quickly in the face of all the good things we have together.

Very recently though I have started to wonder about whether me ‘taking’ the comfort I wanted is the right thing to do. I have been plagued by the worry that I am violating him and his personal space in some way. I have tried to talk to him about it and tell him that I am sorry if I have caused him discomfort. He said it was ‘fine’ but didn’t deny it, which just about broke my heart. He has also promised to try harder to be affectionate with me, but I don’t want him to do anything he doesn’t like. I haven’t touched him since and he has gingerly touched me a couple of times but he is clearly uncomfortable when he does it. 

I don’t know what to do now - whether I should just continue doing as I have the last few years, or whether I should stop contact unless he initiates it himself.  I wondered whether we could reach an agreement that he would speak up if he wasn’t happy with anything I was doing, but I know he is aware of how important physical contact is to me and I don’t really trust that he would let me know. 

I love him to bits but I have no idea how to handle this.