Yes, four times is right - twice in the 50s and twice in the 70s (which is impressive in its own right). He was runner up to Jaroslav Drobny in 1954 and Lew Hoad in 1956, then John Newcombe in '70 and Jimmy Connors in '74, when Muscles was 39 years old.
So there's no shame in who he lost to - all great players, and Hoad in that window when he was as good as anyone on a good day. But I think it also comes down to missing Wimbledon from 1957-67 - that's 11 years, from age 22 to 32, or most of his prime.
Rosewall is a fun player to look at, because he was pretty much the 2nd best player for the better part of two decades - behind Pancho Gonzales in the mid to late 50s, then Laver for the 60s and into the 70s. I can't remember which one, but I think I have him as the best player in 1963, when he won all three Pro Slams and the last World Pro Tour. He is also one of only two players, along with Rod Laver, to win multiple Amateur, Pro, and Open Era Slams.