Ways in which a manifold can be geodesically incomplete
Solution 1:
To your first bullet point, you certainly could make it hyperbolic near the point. The (nonunique) geodesic connecting (-1,0) to (1,0) will go around the cusp and look semicircularish (but I don't think it will actually be a semicircle, just approximately one).
Perhaps something easier to visualize is that $\mathbb{R}^2 - \{pt\}$ is diffeomorphic to $S^1\times\mathbb{R}$. If you give this space the product metric of the usual metrics, then you can easily see and work out the details.
To you second bullet point, you should modify the statement slightly (and somewhat pedantically). Any proper nonempty open subset $U$ of a complete connected manifold $M$ is incomplete. To see this, let $p\in U$ and $q\in M-U$. By Hopf-Rinow, since $M$ is complete there is a geodesic $\gamma$ starting at $p$ and ending at $q$. Since $U$ is an open subset, it is totally geodesic: what $U$ thinks are geodesics are precisely what $M$ does. Thus $U$ thinks $\gamma$ is a geodesic which doesn't stay in $U$ for all time, hence $U$ is incomplete.
To the third bullet point, (you say "surface" then use $\mathbb{R}^n$), the theorem is true for $\mathbb{R}^n$ from Cheeger and Gromoll's Soul Theorem together with Perelman's proof of the Soul Conjecture. The soul theorem states that if $M$ is complete and has nonnegative sectional curvature, then $M$ has a compact totally convex totally geodesic submanifold $K$ (called the soul) so that $M$ is diffeomorphic to the normal bundle of $K$. The Soul conjecture asks: If $M$ has nonnegative curvature everywhere and a point with all sectional curvatures positive, must $K$ be a point?
Perelman proved the answer is yes: under these hypothesis, $K$ is a point. But a normal bundle of a point in a manifold is diffeomorphic to $\mathbb{R}^n$.
Finally, I have been told, but I have no idea about references/proofs/etc that every noncompact surface has some metric (necessarily incomplete if the surface isn't $\mathbb{R}^2$ by the above) of positive curvature. I'll try to dig up a reference tomorrow, if I remember to.
Solution 2:
The OP asked in passing what new issues come up when you generalize from a Riemannian space to a semi-Riemannian one. I'll try to sketch the description.
We generally don't want to call something a singularity if it's in some sense infinitely far away so that you can't get to it. To formalize this notion, you need the idea of geodesic incompleteness. In the Riemannian case, you can characterize this by saying that a geodesic can't be extended in a certain direction past a certain length, as measured by the metric. In the semi-Riemannian case, you don't want "length" as measured by the metric, because a null curve always has zero length. Instead you need to talk about extending the geodesic past a certain affine parameter.
As in the Riemannian case, you can have both curvature singularities and conical singularities. In 2+1 dimensions, all singularities in vacuum solutions are conical, i.e., you don't get black holes with event horizons.
You can have singularities that are timelike, spacelike, or null. These notions aren't defined by default, because the singularity isn't a set of points on the manifold. E.g., to define a timelike singularity, you have to talk about a timelike curve (an observer) containing points p and q, such that the singularity is in p's causal future and q's causal past. Black-hole and cosmological singularities are spacelike, not timelike. A timelike singularity is one way of defining a naked singularity (Penrose 1973), which causes Cauchy surfaces not to exist (Geroch 1970).
The way relativists end up mentally classifying all of these things is in terms of their Penrose diagrams: http://en.wikipedia.org/wiki/Penrose_diagram
Penrose, Gravitational radiation and gravitational collapse; Proceedings of the Symposium, Warsaw, 1973. Dordrecht, D. Reidel Publishing Co. pp. 82-91, http://adsabs.harvard.edu/full/1974IAUS...64...82P (not paywalled)
Geroch, J Math Phys 11 (1970) 437