A guy with AS kissed me without my permission

I went for lunch with a man who has AS. I also have AS and I am 25 years old; he is 50 years old. I don't know him too well, but we met at a social group for people with AS. He is single but has been in a relationship before. I am a *** and have never fancied men.

I found him hard to get on with at first because he stands too close to me, invades my space and sometimes takes to whispering in my ear, which I find VERY uncomfortable and so I move away from him when this happens. But he seemed to be getting better with with this so I agreed to meet him for lunch at a cafe last Friday. The lunch went well, but just as we were saying goodbye, he opened his arms wide in the hug gesture. I hesitated because I don't really like hugs, but I thought it would be rude to decline a friendly gesture. However, he took me completely by surprise. The hug was very tight and oppressive, and he then tried to kiss me, but I moved away and said ''I don't like being kissed', and then said bye to him and walked home. This incident has made me feel quite nervous and I have decided never to meet him on his own again because he did not ask my permission and has therefore broken my trust. However, he attends the group I go to and I don't want to stop going to the group. What do I do?