I feel like my friend is abandoning me

Hi I am really struggling with my mental health my anxiety and everything and a really close friend of mine struggles with me. I feel like he is slowly abandoning me we used to go on drives together he would pick me up in his car and we would go for a nice drive in the evening or weekend. That has stopped next thing that seems to be stopping is him coming over mine he just refuses says he not in the mood and always wants me to come over his. Now Sunday’s seem to be stopping he would come and pick me up at 11 and we would go to this church except it’s not used as a church anyway more it’s a place for people on the spectrum to do circus skills and we do that till 1 in the afternoon then we go back to mind and chill for the rest of the day. Yesterday that never happened I met my other friends instead for Sunday lunch. I was upset over that because I’m so used to going to circus with my friend. He said he didn’t feel like it and wanted to be left alone yesterday. I have spoken to him about this and he says it’s all in my head and he not abandoning me. He says I might be a pain on times and he gets frustrated with me from time to time but he says I’m a good friend. But I really feel like he is slowly walking away from me and I don’t want to lose him as he is a good friend he been good to me and kind. I’ve known him a long time and I really don’t want to lose him. What should I do?