The first husband was not a philanderer. Mildred misunderstood her husband's dealings with another woman, and he was too proud to disabuse her of the matter.
His wife certainly thought so, but the way matters fell out appears to have validated the husband's account.
Why do you think that love or sex (or love and sex) is the only explanation for their relationship? Where did you learn about life--at the movies?
No, it is NOT the implication from the movie. Mildred believes it to be the case (which is all that is necessary to get the plot moving), but Mildred is shown to be a bad judge of character (hence her relationship with the weasel played by Zachary Scott). You are assuming that she was right and that the husband was lying, but why can't the husband have been telling the truth? Nothing in his character is exposed to make us think he is a liar. In fact, given the way the film ends, it seems to me he was a stand-up guy all along.