Singular word for gravel?
Solution 1:
A piece of gravel would indeed be a 'piece of gravel'.
The reason for this is that your other suggestions rule themselves out for various reasons, the primary of these being that not all gravels are alike.
'Grain' would not apply to pieces of stone larger than a couple of millimeters at most, 'pebble' would not apply to a piece of crushed rock and a piece of crushed rock would not be called a pebble.
So while all those terms come under the wider set of 'gravel', they are not interchangeable.
Gravels come in two principal forms; 'sharp' and 'pea' or 'pebble' gravel.
'Sharp gravel' is crushed stone, ie the angular broken pieces you referred to as 'rock fragments'. Sharp gravels are quarried from monolithic rock beds. Mostly from igneous or metamorphic rocks, though there are sedimentary rocks which produce useful gravels, but you wouldn't use the softer sandstones.
The crushed rock is generally screened to a tight range of sizes so that all of the pieces are much of a muchness, but it also covers what variously gets called 'scalpings', quarry bottoming', 'quarry process', 'crusher run' etc which is unscreened gravel and will include all the sizes below the stated maximum and be described by its maximum size and 'to dust'. So an unscreened, unwashed gravel might be specified as '50mm to dust'. You might call the dust 'grains' but you wouldn't apply the term to a larger piece.
A surface laid with a single size sharp gravel will remain a relatively loose and free draining surface. A gravel which goes down 'to dust' can be laid to a more compacted surface which will still drain, but if there are dips puddles can form. Once you have puddles forming the smaller 'dust' will tend to migrate to the surface and create a seal, reducing the free draining properties.
Pea and pebble gravels are quarried from deposits of stone which has been rounded by natural processes, ie by water in river, glacial or wave situations. The rounded shape gives it different characteristics in use and it will not lock under loading as sharp gravels will - imagine the difference between ball bearings and pyramid shaped blocks if you were trying to roll something flat across them. There are the gravels whose individual pieces you would call 'pebbles'.
So, a piece of sharp gravel is a fragment of broken rock and a piece of pea gravel, shingle or pebble gravel is a pebble, or a piece of shingle.
There is no overarching term with which to describe a piece of gravel, which encompasses sharp, pea, pebble and crusher run other than 'piece of gravel' or perhaps 'piece of aggregate' if you are more construction-industry minded.
I'm sure the Wikipedia entry covers the same ground.
Solution 2:
The only real alternative to "piece of gravel" and even then only in some situations, is "stone".
If (for example) it's stuck in your shoe scratching the floor, or if you've stepped on one and hurt your foot, you don't really care that it came from a mass of gravel. This covers both pea and sharp gravel, but isn't appropriate if you're using it in construction or gardening