Show $4x^2+6x+3$ is a unit in $\mathbb{Z}_8[x]$ (inverting unit + nilpotent)

Show that $4x^2+6x+3$ is a unit in $\mathbb{Z}_8[x]$.

Once you have found the inverse like here, the verification is trivial. But how do you come up with such an inverse. Do I just try with general polynomials of all degrees and see what restrictions RHS = $1$ imposes on the coefficients until I get lucky? Also is there a general method to show an element in a ring is a unit?

If $R$ is a commutative ring: the units in $R[x]$ are the polynomials whose constant term is a unit, and whose higher order coefficients are nilpotent. You can apply this directly to your example.

Write $$ 4x^2+6x+3 = 3(4x^2+2x+1) = 3((2x)^2+(2x)+1) = 3 \frac{(2x)^3-1}{2x-1} = \frac{-3}{2x-1} $$ Therefore, $$ \frac{1}{4x^2+6x+3} = \frac{2x-1}{-3} = 3(1-2x) = 3-6x = 2x+3 $$

Hint: As in the hinted paper, a possible ansatz would be

$(4x^2+6x+3) (ax+b) = 4ax^3+(4b+6a)x^2+ (6b+3a)x+3b=1$.

This requires $4a\equiv 0\mod 8$ (so $a$ must be even), $4b+6a\equiv 0\mod 8$, and $6b+3a\equiv 0\mod 8$ and $3b\equiv 1\mod 8$ (so $b=3$).

The cases left are $a$ even with $b=3$.

To find an inverse polynomial for that holds $p(x)(4x^2+6x+3)=1$ so it has to be $3y=1\mod 8$ [edit: For more context on $y$, see the comments below]. So $y=3$ and the polynomial might look like this:

$p(x)=(ax+3)$

Then $(4x^2+6x+3)(ax+3)=4ax^3+(6a+12)x^2+(3a+18)x+9$. Now it has to be $4a\equiv 0\mod 8$ and $6a+12\equiv 0\mod 8$ and $3a+18\equiv 0\mod 8$.

Is there such an $a$?. Yes indeed. For $a=2$ we have $8\equiv 0\mod 8$

$24\equiv 0\mod 8$ and $24\equiv 0\mod 8$.

If we would fail to find this $a$ in this step, we would have to try with $p(x)=(ax^2+bx+3)$ and proceed as above, which gets more and more complicated.

So it is $(4x^2+6x+3)(2x+3)\equiv 1\mod 8$