How do you find the Lie algebra of a Lie group (in practice)?

Here a worked out example: What is the Lie algebra of the group of rotations in 3-dimensional space, $SO(3)$?

Matrices $A\in SO(3)$ are defined by the property that they are invertible and that the scalar product is invariant $\langle A\vec x,A\vec y\rangle = \langle \vec x,\vec y\rangle$ for all vectors $\vec x,\vec y\in \mathbb R^3$. Expanding the latter condition into coordinates, you can see it is equivalent to $A^TA = I$.

To find the Lie algebra, take a smooth path $A(t)$ with $A(0) = I$. In first order, it can be written as

$$A(t) = I + t·H + \mathcal O(t^2) .$$

Plugging this into the condition $A^TA=I$, we get in first order

$$I = (I+t·H)^T(I+t·H) = I + t·(H^T + H) .$$

Hence, the condition on the tanget vector $H$ is that it is antisymmetric

$$ H^T = -H .$$

In other words, the Lie algebra of the rotation group $SO(3)$ consists of antisymmetric 3x3 matrices, which must have the form

$$ H = \begin{pmatrix}0 & -\omega_3 & \omega_2 \\ \omega_3 & 0 & -\omega_1 \\ -\omega_2 & \omega_1 & 0\end{pmatrix} .$$

It is not difficult to show that exponentiating any such matrix will yield an element that preserves the scalar product. Hence, this is the whole Lie algebra.

By the way, the elements of the Lie algebra $so(3)$ are usually represented by the angular velocity $\vec\omega=(\omega_1,\omega_2,\omega_3)$ such that multiplication becomes the cross product

$$ H·\vec x = \vec\omega \times \vec x .$$

A similar method applies to your original problem. For instance, you can embed affine transformations of $\mathbb R^n$ into $GL_{n+1}(\mathbb R)$ by

$$ (x \mapsto Ax + v) \iff \begin{pmatrix} A & v \\ 0 & 1 \end{pmatrix} \in GL_{n+1}(\mathbb R).$$

You can express this as an algebraic condition: a matrix $B\in GL_{n+1}(\mathbb R)$ represents an affine mapping of $\mathbb R^n$ if and only if

$$ (0,0,…,1)·B = (0,0,…,1) .$$

This will induce an equation for the first derivative of a path $B(t)$ and as above, you will obtain the Lie algebra as a set of matrices.

To expand Qiaochu's comment: Your $G$ can be seen as the (closed) subgroup of $\operatorname{GL}_{n+1}(\mathbb{R})$ given by the matrices of the form $$\begin{pmatrix} A & v \\\ 0 & 1 \end{pmatrix}$$ with the usual group multiplication and acting on the affine subspace of vectors of the form $\begin{pmatrix} x \\\ 1 \end{pmatrix}$. If you understand why $\mathfrak{gl}_{n}(\mathbb{R}) = \operatorname{Mat}_{n}(\mathbb{R})$ with the commutator bracket $[X,Y] = XY - YX$ then you should have no difficulty in proving that $\mathfrak{g} = \operatorname{Lie} G$ is given by the matrices of the form $\begin{pmatrix} X & y \\\ 0 & 0 \end{pmatrix}$ with the commutator bracket.

For me, the easiest way to determine the Lie algebra of a matrix group is to think of it as the tangent space at the identity, constructed by tangent vectors of paths through the identity.

Like Qiaochu says, thinking of this as a matrix group is a good first step. Then you need to find one-parameter smooth paths $A(t)$ through the unit element $I$ corresponding to every tangent direction, where $A(0)=I$. Taking the derivative of these smooth paths at $0$ yields a basis for the Lie algebra.