Is it possible to express and analyse Bertrand's paradox with terms and tools from set/space theory?

Your background understanding is fine. The problem is to identify a space that represents the desired objects naturally and to define measure on that space, also in a natural way. “Pick a point from (geometric figure)” allows us to take said figure and use the volume/area/length as natural measure; of course this makes sense only if the total volume/area/length of the figure is finite so that we can normalise it to 1. Same goes for “Pick $n$ points from (figure)”. Note that there is no such natural way to “pick a natural number”, and while we can “pick a random real between 0 and 1” we cannot “pick a random rational between 0 and 1”.

The set of chords does form a topological space in a natural way, but there is no natural way to define a measure on that space. Note that the points (i.e., elements) of this space are not points in your standard intuition, but they nevertheless are called points in this context.

We can map the space of chords to the space of pairs of points on the circle, namely by mapping each chord to its endpoints. Or we can map our space to the interior of the circle by mapping each chord to its midpoint. Or we can map each chord to a pair of an angular direction and a length up to $2r$. All these are injective maps to nice spaces with natural measure (actually, the way I formulated it, the map to the midpoint is not injective, but this fails only for the unlikely case of diameter chords) and they all are continuous and respect the topology on our space (i.e., chords with “near” endpoints also have “near” midpoints as well as “near” angular directions and lengths). Perhaps one of these maps may occur more straightforward than the others, but neither is a really natural choice. Accordingly, the measures induced by these three maps on the space of chords are different and define (paradoxically) different probabilities for chord lengths.