I'm probably going to sound like a broken record here, but:
I am a huge supporter of following one's gut. Regardless of whether or not there was infidelity, the fact that his behavior is causing you this much anxiety is a problem. These are the kinds of challenges where the long term feasibility of your relationship is tested. In order to have a strong and honest relationship, you two HAVE to be able to discuss these kinds of things without the other getting defensive. One's independence is something that you have to give up when you are in a partnership (to some extent).
I do have a few questions though:
1) Do you have any insecurities about why he is involved with you? (I don't know your financial status, your physical experience, your age, his age, etc., so this could be totally off base) If you unsure about his motivations for being in a relationship with you, then that can make you more suspicious of his behaviors than what may be necessary.
2) Have you two discussed the parameters of your relationship? Open relationships are far more common in the gay community, so those of us who value monogamy are often less vocal about our opinions on the subject. If you have, then you have every right to be curious about his behaviors. If you haven't, you still have the right to be curious, but it gives him a bit more clout if he did stray.
3) Has he given you any other reasons to be suspicious? Have you caught him lying about things in the past, regardless of how unimportant or miniscule the lie might be? (Actually, you should be especially worried if he lies often about things that are unimportant).
Regardless of those answers, there is one piece of wisdom (I don't have much) that I can give you: Keeping a relationship together should be your top priority with one exception: taking care of yourself. Making a relationship work takes a lot of compromise, however your own happiness should never be on the list of sacrifices.
-Jon