The Winter Games, staged only 80km (50 miles) from the North Korean border, saw the two Koreas, who are technically still at war since a 1953 armistice, march together at the opening ceremony for the first time since 2006.
Pence and Moon were expected to view men's short-track speedskating together prior to the women's event, as an American and a Korean were competing against each other, but Moon kept Pence waiting and missed the event entirely.
North and South Korea have shared a heavily fortified border since the Korean War ended in a ceasefire, not a peace treaty.
A visit by Moon to the North would enable the first summit between leaders from the two Koreas since 2007, and would mark only the third inter-Korean summit to take place. She was photographed sitting behind Vice President Mike Pence during the ceremony. If peace isn't imminent, a summit in Pyongyang between Moon and Kim Jong Un seems preferable to recent months' threats.
The recent detente, anchored by South Korea's hosting of the Winter Olympic Games that began on Friday, came despite an acceleration in the North's weapons programmes a year ago and pressure from Seoul's allies in Washington.
Moon has not yet accepted the invitation from Kim Yo Jong, although he has said before he'd be willing to visit if it would help end the North's pursuit of nuclear weapons and its decades of hostility and threats against the South.
The athletes elicited a standing ovation by most leaders in the VIP box, including Moon and Kim's sister.
Klaasen, Miller humble India, keep SA alive
He (Miller) had luck on his side and grabbed the chances with both hands. "As I said, it's not a thing that happens every time". On the night, Bhuvneshwar Kumar and Jasprit Bumrah were the main bowlers because the spinners were struggling to grip it.
When he insisted to Mr Moon that South Korea and the United States should resume their joint Foal Eagle military exercise after the Olympics, he was told not to meddle in the South's "sovereignty and internal affairs".
That was not the case with Moon - either at the Games, when he enthusiastically reached up to shake Kim Yo Jong's hand, or at the lunch the next day.
The discussions from Saturday's meeting weren't immediately released.
The North Korean delegation was invited to lunch with South Korean Prime Minister Lee Nak-yeon at the five-star Walkerhill Hotel in eastern Seoul on Sunday, an official at the prime minister's office told Reuters.
The officials toasted with the Korean liquor "soju" and were served buckwheat and potato pancakes, dried pollock soup, grilled halibut and other dishes. "The North is not interested in talks on denuclearisation".
This item corrects that Kim Yo Jong is the leader's sister.
Vice President Mike Pence said Saturday evening there is "no daylight" between the US and South Korea, despite their different approaches to engagement with North Korea at the Pyeongchang Olympic Winter Games.
The luncheon also includes Choe Hwi, chairman of the North's National Sports Guidance Committee, and Ri Son Gwon, chairman of the agency that deals with inter-Korean affairs.
It's unclear whether the occasion could be used to set up bigger meetings between the Koreas to ease tensions.