I'm astonished that no one brought up the matter of other places in the NT where brothers and sisters of Jesus actually showed up and Jesus had to explain who exactly were his brothers and sisters. The crowd he was talking to identified his blood siblings as such, so therefore it's fair to conclude that Joseph did "know" Mary subsequent to Jesus' birth. If they hadn't had intercourse, there would be no present-day bruhaha over a bonebox inscribed for "James, the brother of Jesus", now, would there.