Definition of tangent
Solution 1:
The line tangent to a the graph of a differentiable function at a point is the graph of the local linear approximation of the function at that point.
"Differentiable" means exactly "locally linearly approximable," so this makes sense.
It may interest you that the word "tangent" ultimately comes from the Latin tangens, present participle of tangere "to touch", so meaning "touching".
The word "secant" likewise comes from the Latin secans, present participle of secāre "to cut", so meaning "cutting".
Solution 2:
Assuming the graph's function $\;f\;$ is differentiable at a point $\;(x_0,f(x_0))\;$, the tangent line to that graph at his point is defined to be the straight line
$$y-f(x_0)=f'(x_0)(x-x_0)$$
Why is it called "tangent line" and more is explained, usually with strong geometric intuition and diagrams, in basic calculus courses and books.
Solution 3:
There is not one formal definition; it varies from context to context which definition is most useful, and sometimes the possible definitions are not exactly equivalent for all points and all curves.
A reasonable "default" definition would be that a tangent to a curve would be that a tangent is a line that passes through a point on a curve and lies in the direction of the derivative of the curve at that point.
This definition presumes that the curve is parameterized such that it has a nonzero derivative at the point in question. There are possible ways to deal with points where the derivative is zero (such as reparameterizing by arc length, possibly considering only one-sided derivatives, and so forth), but that definitely gets us into context-specific territory.
Solution 4:
The OP seems to be correct in assuming that the most intuitive approach to the tangent line is through a pair of infinitely close points on the curve. To be completely precise, one needs to take the shadow of that line to obtain the tangent line; i.e., the line through a pair of infinitely close points is infinitesimally off the tangent line. To put it another way, one "rounds off" the line through a pair of infinitely close points to the nearest real slope to get the tangent line. See Keisler for details.