Are there any general terms that encompass a large number of things such as soil, sand, silt, clay, mud, and gravel?
Solution 1:
I looked in Wikipedia's Glossary of geology and found some words that might work:
Detritus is particles of rock derived from pre-existing rock through processes of weathering and erosion. A fragment of detritus is called a clast or clastic rock.
Solution 2:
In terms of haulage and shipping, that kind of stuff is sometimes referred to as Aggregate