Can I move layers one pixel at a time (using arrow keys for example) in GIMP?

Yes, it is possible.

First of all, push M to select the "move" command (or just pick it from the toolbox), then in the "tool options" panel, choose the proper options:

As you can see, the small icons let you move a whole layer, a selection or a path; make sure to select the "layer" icon; then, you have two options, to manually pick the layer that you want to move, or to move the currently selected one.

In the first case, you have to click with your mouse on the image on the layer that you want to move; it may seem complicated, but if you have a layer whose dimensions are smaller than the full image or with transparencies, with this method is possible to manually pick the desired one.

In the second case, of course you are working with the currently selected layer, which I suppose is the thing that you want to achieve; even now, after having selected the proper tool options, you have to click into the image window (even in the grey area outside the proper image) to give focus to that precise window instead of leaving it to the side panels.

Now you can move your layer around with the arrow keys; and if you keep the Shift key pressed, you can move it of a larger amount of space.