How is portal ownership decided?

Portal ownership is designated by who the first person to place a resonator on the portal is. Even if all of their resonators decay, but the portal never goes unclaimed or flips teams, it will still count towards guardian days.

It is to note that the guardian badge will only count if you have resonators on a portal. Even if you have a 150+ day guardian that you own but you have no resonators, you will not get the badge until you upgrade one of the existing resonators to your own. Then the badge will pop.

As for your observation. There has to have been a time in which the portal was either unclaimed or the other teams. It is impossible for a portal to change ownership without changing color.

What you have seen can happen if you are player A, and for a short time.

If two players of same faction place a resonator on an unclaimed portal at about the same time (no more than a few seconds delay), A may see its resonator and its name as owner on the portal, then see it switch to the other player B. That means (I suppose) that on Niantic server B was first to place the resonator, but locally on A scanner the portal status is wrong for a few seconds.