Feeling left out regardless of how people are with me

It is a question of not belonging.

Do you not belong because you really are different, or because you just think you are different.

If you really are different then you will feel on the outside, even if people invite you in and you pretend to be involved. If you are masking it is not real. It is also tiring 

If you just think you are different you will behave as if you are are create you own imaginary walls. These constrain you but are of your own making. The solution is to set yourself free and allow yourself to engage. Don't take yourself too seriously.

I realised in Jan I had built walls to protect me, but they had become so high even the sun could not get in. Isolated at the bottom of a windowless tower is secure but trapped, safe but lonely. 

Either get some windows, lower the walls or move out. 

I get that exact same feeling. And it’s not even the people. I feel as if I’m either trying too hard or not enough.

I'm an observer, being an observer is also very much an only child thing too. I think so much of of our communication is non-verbal, there are probably loads of micro expressions that tell others we're ND, an NDar similar to a gaydar.

I used to have the same.with my ex-partner, people would always talk to her. I felt a bit emasculated but actually I preferred it. Same when I stayed at a b&b with somebody from work in Italy.

In the second case the owner even commented on my quietness. I thought I was just being normal.

I now think it's eye contact, lack of a genuine smile, issue with small talk, perhaps a bit of body language.

In my case I have optimised the business interaction setting, but not the informal setting as I avoid them.

It was part of masking. I don't have a good non-work setting as the interactions vary too much, and I'd be a bit awkward.

I imagine people can detect it. After all the assessment people do, so there is something there.

Perhaps it is subconscious in others. People can detect subtle cues. I wonder if you even smell it. The stress could change your hormones or sweat meaning pheromones are different.

Would you consider yourself more of an observer than a joiner in? I often feel more like I am noticing how people interact with each other as a detached observer than as an active participant. I never feel part of a crowd but merely present in the same place as a crowd. I have always felt like this but only since realising I was autistic did I understand why. It always confused me how everyone else seemed to have an intrinsic understanding of how to interact and I didn't.

I read recently about some research that said that not people could subconsciously detect autistic people and behave differently with them, even if they had no concept of autism. I could never understand why, if my wife and I met someone, they would be more drawn to her than me, before even interacting with either of us. Now I get it. It's basically a vibe!

I know that feeling well, I wonder if it's because subtely we are on the outside, we don't always engage in the same way with the things NT's are interested in, we either over or under-enthuse? I think we often find it difficult to join  in on groups conversations, to jump in with a comment.

Depends on the circumstances as to why you feel that way, is there an underlying issue you haven’t mentioned, medical or personal?

If you haven’t a strong sense of why it could be dissociation or depersonalisation.

https://en.m.wikipedia.org/wiki/Dissociation_(psychology)

https://en.m.wikipedia.org/wiki/Depersonalization

Sometimes a concept we just understand (as we have "been there" too,) can best be explained to a non-Autistic person with the help of a situation example (described as experienced from our perspective).

https://rosieweldon.com/outside/

https://www.huffingtonpost.co.uk/entry/being-autistic-my-loneliness-may-not-look-like-yours-but-its-just-as-valid_uk_5aeb2171e4b041fd2d23c7b9

Hoping something like the above was what you meant?  If not, please do say so - as someone is likely to have another idea too.