How many hexes can a city use in Civilization: Beyond Earth?

Your original assumption was correct, the city can only use (as in "work") tiles up to 3 hexes from the centre.

Cities appear to be able to expand up to 5 hexes away based on my experience (which I believe was the same in Civ V?). But they cannot work tiles beyond the 3.

In the picture below, we can see that the city has spread 4 tiles away to the hex in the Red box, but the city does not have the option to work the hex (there is no circle to place a citizen). If this hex was just being worked by another city, we would see a grey hexagon like in the hexes in the Green box.

You can still build improvements and get strategic resources outside the 3 hexes, the city just cannot "work" them.

Your question was: "what is the maximum number of hexes a city can end up using?" If you avoid canyons, mountains and other cities, the theoretical maximum is 36 hexes. However, the way I play the game the answer is about 30. My capital won't reach a size that can work 36 hexes before the game ends.

Practically, this means that when choosing a city location, a few mountains and canyons probably won't limit city growth, imo. It's also worth remembering that if a tile is 6 tiles away from the city it will NEVER be reached.

If you prioritise food and growth it maybe you can utilise the full 36.

Beyond the 36, the only thing that matters is Strategic Resources (which are Oil, GeoThermal Underwear, Titnium, Floatystone, and Fire Exit). As you accumulate culture, your city will reach out to these automatically, as you have experienced. This is so you can use the Strategic Resource in trades, or to build buildings and eunuchs e.g. the Lev Tank needs Floatystone.

Basic Resources (Gold, Silicon, Fruit, Giant Maggots, etc) that are beyond the 36 nearest hexes are completely useless as they can't be worked. Improvements on e.g. Fruit out there are completely fruitless. Sometimes your city will reach out there, for no reason, just so you can have some Fruit that can't be worked.

I like to launch Orbital Fabricators - these create Strategic Resources, provided that you haven't built an improvement there (Magrail doesn't count). Because the hexes (beyond the nearest 36) can be used as Strategic Resources, I've ended up with 50 Fire Exits.

In the end game you may be asked where you want to put the game-winning Mind Flower. I like to put it on a Basic Resource outside the 36 nearest tiles. The Basic Resource will be destroyed but who cares as it was not being used.