Adjoint functors as "conceptual inverses"

The Stanford Encyclopedia of Philosophy's article on category theory claims that adjoint functors can be thought of as "conceptual inverses" of each other.

For example, the forgetful functor "ought to be" the "conceptual inverse" of the free-group-making functor. Similarly, in multigrid the restriction operator "ought to be" the conceptual inverse of it's adjoint prolongation operator.

I think there is some deep and important intuition here, but so far I can only grasp it in specific cases and not in the abstract sense. Can anyone help shed light on what is meant by this statement about adjoint functors being conceptual inverses?

In a nutshell, a natural bijection between $\mathrm{Hom}(f(x), y)$ and $\mathrm{Hom}(x, g(y))$ says that the "graph of $f$" is obtained from the "graph of $g$" by "reflecting in the diagonal", just like the relationship between the graphs of inverse functions in calculus.

In more detail: two functions $f \colon X \to Y$ and $g \colon Y \to X$ are inverses if their graphs are related by simply swapping the $x$ and $y$ coordinates, i.e., if $\{ (x,y) \mid f(x)=y \} = \{ (x,y) \mid x=g(y) \}$. For two elements of a set, there are only two possible relations between them: they are equal or not, and we can rephrase $f$ and g being inverse to each other yet again as saying the relation between $f(x)$ and $y$ is the same as that between $x$ and $g(y)$, or, using the Kronecker delta, as $\delta(f(x),y) = \delta(x,g(y))$.

Now for two objects of a category there are many possible "relations" they might be in, at the same time! These "relations" are the morphisms between them. So the generalization of the previous relation between $f$ and $g$ to functors should be that the relations between $f(x)$ and $y$ are in bijection with those between $x$ and $g(y)$, i.e., $\mathrm{Hom}(f(x),y) \simeq \mathrm{Hom}(x,g(y))$.

One of my friend was being confused with the same question and decided to ask the author of the article about it.

His reply was,

If you take arbitrary abstract categories and stipulate that a pair of adjoint functors exists between them, the only thing you can hold on to is their abstract or formal properties, e.g. the left adjoint preserving colimits, etc. Of course, in general, it is not the case that you have a forgetful functor, but the general point remains. (Recall that I used the case of the adjoint functor simply to illustrate the main general point.) One can and should consider a pair of adjoint functors as providing conceptual inverses. One has to be careful and look at the details, even more so since a functor can have both a left and a right adjoint! The best analogy is probably from topology with the notions of section and retraction to a given map. But again, one has to be careful and it is probably better to think about these up to homotopy.