Is it noticed if your spouse is having an affair or even, regularly visiting his "second home" ?

I'v a friend I stayed with for more than a year. Quite frequently, some other friends will come visit say, twice a week and we chat till late at night. Whenever we see her husband, he's very casually dressed and goes out after dinner (carrying his sports bag) He doesn't come home till midnight or even past midnight. We are wondering - where a man can go for this many hours each night (on the premise that he's playing tennis/badminton). And his wife seems to totally believe. Should we all alert her ?