Solution 1:

There is not just one "concept of infinity in mathematics"; there are lots of them. For any one in particular, once you know more precisely what you're looking for, it's easier to search or ask for good references. I'll try to highlight/summarize all the relevant concepts I can think of, and give at least an English Wikipedia link for each. I encourage a reader to skip around, but exposure to limits in Calculus is helpful for many sections, and some paragraphs are directed to people with much more background.

  1. When dealing with infinite sequences or similar, the symbol ∞ is used as a sort of shorthand for "unbounded by natural".
  2. When dealing with real numbers (e.g. limits in Calculus), the symbol ∞ is used as a sort of shorthand for "unbounded by reals".
  3. In real analysis, building on Calculus, it's useful to give ∞ and -∞ some algebraic and topological properties and treat them as objects rather than shorthands.
  4. In some contexts, especially in complex analysis, it's useful to consider something like ∞ and -∞ that doesn't distinguish signs/directions.
  5. Phrases like "points at infinity" are used to understand ideas related to vanishing points in art.
  6. "Infinite/Transfinite ordinals" help us give rigor to ideas like "infinity plus 1 comes after infinity".
  7. "Infinite/Transfinite cardinals" help us give rigor to ideas like "infinite sets have different sizes if we can't pair up their elements".
  8. Densities help us give rigor to questions like "What proportion of the naturals are squarefree?"
  9. Non-archimedean ordered fields let us do arithmetic and compare the sizes of infinite quantities at the same time.
  10. Absorbing elements of algebraic structures act like ∞ does in senses 3 and 4 above.

1. A shorthand for "unbounded by naturals"

In a number of contexts, the word "infinite" or the lemniscate ∞ is used as a shorthand for something along the lines of "go beyond each natural number". In these contexts, it's not really an object but more like tidy notation.

1a. Limits of sequences/series

One place this comes up is in the context of the limit of an infinite sequence. For example, we might write ${\displaystyle \lim_{n\to\infty}}\frac1n=0$ to mean something like "For each positive error tolerance $\varepsilon$, there is some big-enough natural $N$ past which ($m>N$) any expression $\frac1m$ is within $\varepsilon$ of $0$.". Note that the sentence did not use the word "infinite" at all.

For more subtle properties, we use related notation like ${\displaystyle \liminf_{n\to\infty}}\,a_n$.

Similarly, infinite sums/series are typically defined as the limit of a sequence of partial sums: ${\displaystyle \sum_{n=0}^\infty}a_n={\displaystyle \lim_{m\to\infty}}\,{\displaystyle \sum_{n=0}^m}a_n$.

1b. Infinite unions/intersections

Similar-looking notation arises when we want to take the union or intersection of a bunch of sets indexed by the natural numbers. However, here we do not have the same sort of limit-based definition. The meaning of an indexed union like ${\displaystyle \bigcup_{n=0}^\infty}A_n$ is simply the set of everything that's in at least one of $A_1,A_2,\ldots$. Similarly, the intersection would be the set of everything that's in all of the $A_1,A_2,\ldots$. Note that there is no limit involved*, and there is no term $A_\infty$. Sometimes the notation $A_\infty$ might be used for the union/intersection itself, though.

*(Well, sometimes we might use a limit notation, especially when the sequence of sets is monotone with respect to inclusion, but that notation is more advanced and rare.)

1c. Other algebraic contexts

We might consider infinite cases of other operations with big symbols like Cartesian or direct products, direct sums, disjoint unions, and coproducts more generally. For those, the symbol ∞ is used with those similarly to unions and intersections.

Sometimes the symbol ∞ ends up in more compact notation for constructions above, or other direct limits. For example, $\mathbb R^\infty$ is sometimes used for a direct limit/appropriate coproduct of $\mathbb R^n$ (what you might think of as ${\displaystyle \bigcup_{n=0}^\infty}\mathbb R^n$ if you imagine that each $\mathbb R^n$ lies inside of $\mathbb R^{n+1}$). $\mathbb Z(p^\infty)$ represents a different direct limit, etc.

Finally, we often speak of "infinite" when no natural suffices. $\mathbb R^\infty$ as mentioned above can be viewed as an "infinite-dimensional" vector space (because it has no finite basis) or topological space (say, because of coverings or a way to build it).

2. A shorthand for "unbounded by reals"

Symbols like $\infty$ (or $+\infty$ for emphasis) and $-\infty$ are used as shorthand in analytic contexts as well. In those contexts, $\infty$ suggests something like "go beyond each positive real number".

2a. Intervals

Arguably the simplest such use is in interval notation. For example, $(-\infty,2]$ is just shorthand for "the set of all real numbers that are at most $2$".

2b. Limits of real functions

When we want to denote that a real-valued function grows or decreases without bound as the input approaches something, or that it does something notable as the input grows/decreases without bound, we use standard limit notation involving $\infty$ and/or $-\infty$.

For more subtle properties, we use related notation like ${\displaystyle \liminf_{x\to\infty}}\,f(x)=-\infty$. And improper integrals like ${\displaystyle \int_{-\infty}^{17}}\,f(x)$ use these symbols to suggest taking a limit of a usual integral.

2c. Measures of length, area, etc.

In measure theory, we consider abstract properties of ways to measure things like length, area (and its relationship to concepts of integral), etc.

It is reasonable and common to say things like "the length of $[-\infty,3)$ is $\infty$" in analogy to "the length of $[2,5]$ is $3$" or the length of a parabola in the plane is $\infty$. Arguably, "length is $\infty$" could be regarded as a shorthand for "contains subsets of arbitrarily large finite length". Similar things can be said about area (a "measure" of sets in the plane), etc. This is an aside, but as you might expect, the standard lengths of the rationals and the Cantor set are taken to be zero.

A special case in measure theory is the counting measure, where we just count the number of elements in a set if it's finite, and write $\infty$ otherwise. This is very similar to the talk of "infinite dimensional" at the end of 1c.

Also, sometimes we consider (extended) signed measures where maybe $-\infty$ is allowed to be the measure of something. Such an object would have subsets of arbitrary negative measure.

3. Directed infinities as genuine objects

The shorthands in 2. are a bit inconvenient for some applications, because we get a bunch of special cases. For instance, we can't say "the length of a union of two disjoint sets is the sum of the lengths", since $\infty$ (as in 2.) is not a number; we need to separate out the case when a set has infinite length. There are similar case distinctions with the limit laws. We can solve these issues by declaring $\infty$ and $-\infty$ to be objects with arithmetic and order properties compatible with the properties of real limits.

3a. Extended reals

The extended real line is the reals with two extra objects added in: $\infty$ and $-\infty$. Then we declare by fiat all properties that mesh nicely with how those symbols arose in limits. $5-\infty=-\infty$, $\frac3\infty=0$, $-\infty<-2$, etc. This leaves certain operations like $\infty-\infty$ undefined, because of indeterminate forms. (Though in measure theory, $0*\infty=0$ is often chosen for convenience.)

3b. Complex directed infinities

Sometimes this sort of idea is extended to directed infinities in the complex numbers. For instance, $i\infty$ suggests the upward direction and $(1+i)*\infty$ suggests northeast.

3a. Ends in general

The real line $(-\infty,\infty)$ is capped off "at the ends" by the extended reals $-\infty$ and $\infty$. And you might imagine an infinite complete binary tree capped off at infinitely many ends (one for each path) by something like the Cantor set. This is formalized in general in topology by the end/Freudenthal completion (universal property in "The Theory of Ends" by Georg Peschke).

4. Undirected infinities

For some contexts/applications, it doesn't matter what direction something has (if any), just that the absolute value grows without bound (or an analogous situation in more abstract contexts).

4a. Reals with an undirected infinity

There are some minor conflicts/inconsistencies in terminology and notation, but Wikipedia calls an important object the projectively extended real line. We add in to the reals a single object, often denoted "$\infty$". I prefer to distinguish it from $+\infty$ (from 2. or 3a.), e.g. "$\hat\infty$".

This additional object represents what happens to functions/sequences whose absolute value grows without bound even if they alternate sign (e.g. $\frac{x}{sin x}$ for large $x$), or inputs that are large in absolute value (e.g. $1/x$ is near $0$ if $|x|$ is large). Accordingly, we define $\frac10=\hat\infty$ since ${\displaystyle \lim_{x\to0}}\,\left|\frac1{f(x)}\right|=\infty$ when ${\displaystyle \lim_{x\to0}}\,f(x)=0$. Similarly, $\frac{1}{\hat\infty}=0$. But $\hat\infty+\hat\infty$ must stay undefined because we cannot tell what the limit of $f(x)+g(x)$ would be just by knowing $|f(x)|,|g(x)|\to\infty$. We also lose order properties (we can't reasonably write $3<\hat\infty$) since $\hat\infty$ doesn't care about sign.

Topologically/pictorially, we can imagine this $\hat\infty$ turning the real line into a circle. If you define a function from $\mathbb R\cup\hat\infty$ to itself, you could graph it on a torus. A rational function $f(x)$ can be nicely/continuously extended in that way, since it can be $\hat\infty$ when the denominator is $0$, and $f(\hat\infty)$ can be the height of the unique horizontal asymptote when one exists or $\hat\infty$ otherwise.

4b. Riemann sphere

If we add on a single $\hat\infty$ to the complex numbers instead of the reals, we didn't have a good order to lose, and things are particularly nice. If the reals became a circle, the complex numbers become the Riemann sphere. In this context, the rational functions are, in a sense, all of the differentiable functions from the sphere to itself.

4c. One-point compactification

In topology, under nice conditions (if we have a locally compact noncompact Hausdorff space), we can add on a single point and get something compact by giving the new structure the appropriate topology. This is called the "Alexandroff" or "one-point" compactification. For example, $\mathbb R^n$ becomes the $n$-sphere $S^n$.

4d. Linear relations

If we have the rationals or reals or complexes (any field), then there is a natural way to add on an undirected infinity $\hat\infty$ and two more objects ($\bot$ and $\top$) where the system is closed under addition, subtraction, multiplication, and has a generalized multiplicative inverse. Essentially, the linear subspaces of the plane are the objects (a line of slope $r$ corresponds to the number $r$) and the operations come from considering them as relations.

I first saw this in the Graphical Linear Algebra blog. The most relevant entry is Keep Calm and Divide by Zero, but the following two entries contain interesting context as well. This approach may have been discovered by Paweł Sobociński. Unfortunately, the only source I know of with a treatment divorced from graphical linear algebra is another answer of mine.

5. Projective spaces

If you think about perspective in images/how we see the world, parallel lines like railroad tracks often appear to converge/meet at a point in the image. It's as if there's a "point at infinity" down the tracks, even though we know that parallel lines don't actually meet.

Also, there are some annoying special cases in geometry: In the plane, two distinct lines determine a point (of intersection), except if the lines are parallel. Going up a degree, we might expect a line (degree 1) and a parabola (degree 2) to intersect in two points, at least if we count tangent lines as "intersecting twice" and ignore cases where we don't have points because the solutions to the equations are complex. But that fails for a line perpendicular to the directrix: $x=1$ intersects $y=x^2$ only at $(1,1)$, and there's no tangency or complex number cause to blame it on.

5a. Real projective plane

The real projective plane is a solution to the problems above. There is an algebraic construction, but I prefer to start by thinking about it geometrically, either intuitively in 2D or more rigorously in 3D.

In 2D, we take the plane and add on a special "point at infinity" or "ideal point" for each slope of lines to meet at, and one more for the vertical lines to meet at. We consider all the ideal points a "line at infinity" (so two points still determine a line). This gives us a nice duality between points and lines. Then the line represented by $x=1$ and the parabola represented by $y=x^2$ should intersect both at $(1,1)$ and at "the ideal point where all vertical lines meet" since the parabola becomes more and more vertical as you move away from the origin.

For a 3D interpretation, let's call lines in 3D that pass through the origin "projective points". And planes in 3D that pass through the origin are "projective lines". Two lines span a plane (two projective points determine a projective line), and two planes intersect along a line (two projective lines intersect in a projective point).

Now imagine $z=1$ as our regular 2d plane. A line through the origin represents the regular point where it intersects $z=1$. Except there are extra lines (those in the plane $z=0$) that don't intersect $z=1$ at all. Those lines represent "ideal points" not in the regular plane.

The lines $y=0$ and $y=1$ in our plane $z=1$ are part of the planes $y=0$ and $y=z$, which intersect in the line $y=z=0$, which doesn't itself intersect $z=1$ so it's an "ideal point". Similarly, $x=1$ and $y=x^2$ in $z=1$ lie on $x=z$ and $yz=x^2$, which intersect in the lines $x=z=0$ and $x=y=z$, the former of which is an "ideal point".

a 3D image of the described situation

This image is a view from above where you can see the parabola in the yellow plane $z=1$ (the intersection with the green surface $yz=x^2$), and the point $(1,1,1)$ on the left which is part of the line $x=y=z$ which intersects the line $x=z=0$ in the center of the picture. You can manipulate the picture at Math3D.

It's harder to visualize, but we can use pairs/triples of complex numbers instead of real numbers, to capture things like "the points where $y=-1$ intersects $y=x^2$". If we treat tangency correctly, this will always give us the number of intersection points one might expect - a result known as Bézout's theorem.

5b. Real/Complex projective lines

If we take the 3D story down one dimension, note that $y=0$ is the only line in the plane that doesn't intersect the line $y=1$. We could add just one ideal point to the real line, or to the complex equivalent (the complex plane). This gives the real projective line (basically considered in 4a.) and the complex projective line (basically the Riemann sphere from 4b.).

5c. Projective spaces

These ideas can be extended to higher dimensions (even an infinite dimensional version, analogous to $\mathbb R^\infty$ discussed earlier). And we can work not just with the real or complex numbers, but any division ring, to get many more "spaces" with "points at infinity".

There are a lot of general things in algebraic geometry that can be built upon these ideas which I am not qualified to expound. The Wikipedia page for "projective variety" is illustrative.

6. Ordinals

In none of the discussion above have we had a reason to say that one infinity is larger than another. One natural context involves focusing on the order of things and is called "the ordinal numbers" (not to be confused with ordinal numerals like "third").

Some small ordinals can be thought of on a very intuitive level: All of the naturals are ordinals. The first ordinal that comes after all of the naturals is called $\omega$. Then there are $\omega+1,\omega+2,\ldots$. And after those we have $\omega\cdot2$. These and more are illustrated in images like this one on Wikipedia.

It can be useful to think of the ordinals as representing ways things can be ordered. Specifically, an ordinal represents the order structure of the ordinals that came before. $\omega$ represents an order like $0,1,2,\ldots$ or "a","aa","aaa",…. $\omega+3$ represents orders like "a","aa","aaa",…,"b","bb","bbb". And $\omega^2$ represents orders like "ab","abab","ababab",…"abb","abbabb",…,"abbb","abbbabbb",…, etc.

With this perspective, we can understand the arithmetic operations as they are usually defined in terms of combining the two ordered "lists" together in various ways. Addition is like concatenation, and we end up with $2+\omega=\omega\ne\omega+2$, for instance. Similarly, multiplication is like replacing entries in one order with (copies of) those of the other. We have $2\cdot\omega=\omega\ne\omega\cdot2$. So these order-based operations are not commutative. For infinite exponents, exponentiation is trickier to describe, but you can read about a few ways of looking at it on wikipedia.

All of the discussion above has been fairly informal. Formally, a set $A$ is transitive if $x\in y$ and $y\in A$ imply $x\in A$. The ordinals are typically defined as transitive sets of transitive sets (or in some equivalent fashion). From this, it follows that the ordinals are well-ordered by membership (every nonempty subset has a least element) and are canonical representatives for the isomorphism classes ("order types") of well-orders in a particularly nice way.

7. Cardinals

Two sets could be considered to have "the same size" exactly when the elements could be paired up in a one-to-one correspondence. For less ambiguity, we say that such pairs of sets have "the same cardinality" or are "equinumerous". Cardinal numbers measure this sort of size.

7a. Countable versus uncountable

A set is finite if it is equinumerous with a set of the form $\{1,\ldots, n\}$ for some nonnegative integer $n$ (and then we usually say it "has $n$ elements"). Analogously, a set is "countably infinite" if it is equinumerous with the naturals. A set is "countable" if it's finite or countably infinite. There are many sets that are surprisingly countably infinite, often discussed in the setting of Hilbert's Hotel. Notably, the rationals are countable. However, Cantor's diagonal arguments show that certain sets are uncountable, such as $\mathbb R$ and the power set $\wp(\mathbb N)$.

7b. Cardinal hierarchy

Just like we do with finite sets, if a set is equinumerous with some ordinal, then we can use the smallest such ordinal to represent the size, and give it a new name (and new operations) for this cardinality purpose. Every countably infinite set is equinumerous with $\omega$, which gets the new name $\aleph_0$. $\aleph_1$ is the next biggest cardinality of an ordinal, $\aleph_2$ is next biggest, ... $\aleph_{\omega}$ is bigger, and so on through all the ordinal subscripts for all of the aleph numbers. We can also keep taking power sets of $\omega$ to get the beth numbers, which need not align with the alephs. If the axiom of choice holds, then every cardinality is represented by an aleph number. If not, we can at least use Scott's trick.

If you know about proper classes, you might wonder about measuring their sizes in this sort of way. The axiom of limitation of size, equivalent to global choice in the rest of NBG, forces there to be only one size of classes too big to be sets.

8. Densities

For this section, "naturals" means the positive integers. The set of naturals and the set of even naturals have the same cardinality. But, at times, one might want to say that half of the naturals are even. The word "density" is often used when we want to talk about the proportion of naturals in a subset. There are many inequivalent definitions.

8a. Natural densities

Let $A$ be a subset of the naturals. To understand the proportion of naturals in $A$, we might examine the proportion up to some cap. Let $a(n)$ be the number of integers in $A$ no greater than $n$. Then $\frac{a(n)}{n}$ intuitively gives an approximation of the proportion for all naturals. When it exists, we take ${\displaystyle \lim_{n\to\infty}} \frac{a(n)}{n}$, and call that the natural/asymptotic/arithmetic density. When it doesn't, we use limsup and liminf and call them the upper and lower density, respectively.

8b. Other densities

There are other ways to measure the "density" of a set of naturals. For instance, logarithmic density, and for additive number theory purposes Schnirelmann density. For the Davenport–Erdős theorem, there is also a "sequential density" considered.

9. Infinite "numbers"

None of the conceptions of infinity covered above really act like "numbers" in the sense that, say $2*H+H=H+2*H>H$ for some "infinite" $H$ satisfying $H>1,2,3\ldots$ or similar.

In general, if $x$ and $y$ are positive (in a linearly-ordered group, say), and $\underbrace{x+\cdots+x}_{n\text{ terms}} < y$ for each positive integer $n$, then $y$ is "infinite" with respect to $x$. If there is something called $1$, then "infinite with respect to $1$" is usually abbreviated "infinite" (or "unbounded", or perhaps "transfinite").

There are many non-archimedean structures where these (relatively) infinite elements exist, and a lot of attention is given to non-archimedean ordered fields where we can divide by nonzero elements, etc.

9a. Adjoining one infinity and more

To start to understand the effects of infinite elements, we can consider adding a single infinite quantity $H$, and seeing what effects closure of the operation(s) have.

If we consider the integers with addition and negatives, and then add on $H$ which is greater than all integers, then we get $\{aH+b\mid a,b\in\mathbb Z\}$ where $aH+b\le cH+d$ exactly when $a<c$ or $a=c$ and $b\le d$.

Now, instead, consider the rationals or reals with addition, negatives, multiplication and inverses, and add on an infinite $H$. Then we have the ordered field of rational functions in $H$. Since $y>x$ exactly when $y-x>0$, it suffices to describe the positive elements; those would be the ones with a positive ratio of leading coefficients.

There are a number of notable "small" non-archimedean fields that expand on these ideas, many of which are mentioned here on the Wikipedia page for the "Levi-Civita field".

9b. Robinson's hyperreals

A popular implementation is Robinson's hyperreals (not to be confused with a more general concept of hyperreal numbers). Definitions vary slightly (and unavoidably without the Continuum Hypothesis), but basically they are a field that is close enough to the reals that the new elements can be used to do calculus, as referenced in another answer by Mikhail Katz. The basic idea of a construction isn't too complicated; I like Terry Tao's voting analogy. A hyperreal is a sequence of reals that vote each time you ask about a property (like "are you bigger than 5?"). How to determine which infinite collections of voters count as good majorities is handled by technical axiom-of-choice stuff, but you don't have to worry about that to get the idea.

9c. Surreal numbers

Another popular object is the surreals, suggested in my profile picture. They are too big to fit in a set, but have a nice recursive construction as pairs of sets of surreals, and a "sign expansion" where they are functions from ordinals to $\{-,+\}$. In settings like NBG, they are "universal" in the sense that they contain a copy of every (set-sized) ordered field.

10. Absorbing elements

When we have $\infty$ in senses like 4a/4b, it has a special absorbing property where $x+\infty=\infty+x=\infty$ for all $x$. This is similar to $0*r=r*0=0$ for all real/complex $r$. An absorbing element is often written as $0$ and a semigroup with such an element is called a null semigroup or a nulloid (by Pete L. Clark) in analogy to monoid. However, in some contexts where the operation is written additively, the symbol $\infty$ is still used. For example, it is used this way in Three-player impartial games by James Propp.

Solution 2:

I prefer the original formulation of your question before the question was closed. Some illuminating ideas about infinite numbers can be found in the very accessible book Keisler, Elementary Calculus. An approach using infinitesimals which was recently reissued by Dover.