Who was more beautiful Lyanna or Cersei?

How did Renly expect Robert to put Cersei aside and take Margaery as queen instead?

chaos

You're right that the premise as you put it makes no sense results because you are missing an important part - Renly knew about the incest.

This immediately makes his motivations meaningful. He tells Robert that Cersei's children are not his, Cersei and the children will be executed, and that Robert, as a middle-aged king, has no heir. He has to remarry and have a son, so Renly introduces him to Margaery, who would be a very sensible choice anyway. This ends with the Renly and his allies, the Tyrells, being the new powers behind the throne and the Lannisters massively falling from grace.

Only then does the plan make sense - if Renly just wanted Robert to replace Cersei with Margaery, it wouldn't help the Tyrells because the Joffrey would still be the next king - their power would die if Robert did, and Margaery would essentially were wasted as marriage prospects.

Olenna: "The thought that one day he might see his grandson with his ass on the Iron Throne makes Mace puff like ... a puff pastry" - ASOS Sansa I.

Mace Tyrell wants his grandson to be king, and he can only get that when Cersei's children are out of the way.

The next question is how Renly found out, but that's not hard to imagine - Littlefinger, Varys, Jon & Stannis, and Ned all came to this conclusion, and Renly knew a lot of people at the court, almost certainly had spies, and would have looked like that kept an eye out for something to put the Lannisters at a disadvantage. Renly also knows Edric Storm well, one of Roberts' bastards who perfectly demonstrates that Roberts actual children look like him.

Renly did not reveal the incest once he knew it because his worst fear was that Stannis would end up as king - he had to set it right so that Robert could marry Margaery once he was told there would be no danger that Robert dies and the crown goes to Stannis.