Residual soil or surplus soil

Consider three words:

Cambridge

Residual:

remaining after most of something has gone

Cambridge

Surplus:

an amount that is more than is needed

Merriam Webster

Leftover:

something that remains unused or unconsumed

All have the meaning of remaining from an initial amount. Surplus has the additional association of not being needed.

Your soil is certainly residual from the construction.

Is it surplus? Many would reasonably say so but, because it was never needed (that’s why it was removed), I suggest residual is a slightly better word.

Also consider leftover, which merely states that it remains after the end of work, without any overtones of need.

The most common term is excess soil. This can apply to soil that is reused on the same site, or hauled off. Surplus soil is also a common term, but is more strongly associated with soil that is sold off and hauled away. In construction, surplus tends to indicate something has a resale value.

Cut-Fill/Borrow/Excess/Dirt Balance: refers to how many cubic yards of dirt will be excavated, moved, placed and compacted on a job site. After the excavation (Cuts) are completed, the site may have too much dirt (Excess) or not enough (Borrow). Ideally, an owner wants to have a Balanced site to avoid the costs of hauling Excess off-site or buying Borrow material.

https://graynson.com/glossary-sitework-common-terms/

Surplus is the right word as shown for example here on this Canadian government website.

There they refer to surplus earthwork material which includes more than just soil. I think the potential difficulty with 'surplus soil' is the alliteration but in context I think it will be fine. You may need to differentiate between 'fill' soil that you brought to site and surplus excavated soil that you removed.

Ciria has

Sustainable management of surplus soils and aggregates (RP1124)

but this and the Canadian government's site is more general. As Phil Sweet points out, once you decide you don't want it, excess soil is probably a better word. See the Ontaria govt. site title Handling Excess Soil with

Excess soil is soil that has been dug up, typically during construction activities. It must be moved off-site because it can't or won't be reused at the development site.

Where I live (upstate NY), at least, this would be called "clean fill," which can be used in other construction projects which require soil to fill some spaces and gaps. You'll see signs such as "Clean fill needed." People pay for it. It's okay if there are some rocks but if there's trash mixed in, then it's not clean fill. You'll also find "Clean fill for sale."