What's the origin of "nest" implying *multiple* levels of containment?

Solution 1:

You ever nest two or more boxes together? That nest may have inspired later mathematicians and (eventually) everyone else to use nesting to suggest hierarchies of multiple nested things.

The Oxford English Dictionary has this definition for what I'll call the hierarchical nest (in "nest, v."):

9. b. transitive. To place or fit (an object) inside another, esp. of the same kind; to place (an abstract element or entity) within the scope of another, esp. in a hierarchical structure; to arrange (abstract elements or entities) in a hierarchical conceptual structure. Usually in passive.

and this example:

1961 D. V. Huntsberger Elem. Statist. Inference ix. 230 Situations of this sort, where every classification is nested within the next larger one, are called nested or hierarchal classifications.

In the mid-twentieth century, as structuralist and systems-based thinking proliferated, hierarchical nest may have emerged, so that it applied to sets or hierarchies. For example, here is a 1943 mathematics paper that uses nested when working with sets (Wiener, Norbert, and Aurel Wintner. “The Discrete Chaos.” American Journal of Mathematics, vol. 65, no. 2, 1943, pp. 279–298):

Since m(E*) is independent of 1, it follows that in the sequence of successive decompositions of E* not more than m(E*) nested sequences of parallelepipeds contain points P. (p.288)

The mathematical usage extends even further back. In JSTOR, the earliest result for a search of "nested AND math*" rests in Ragsdale, V. “On the Arrangement of the Real Branches of Plane Algebraic Curves.” American Journal of Mathematics, vol. 28, no. 4, 1906, pp. 377–404. Ragsdale has to define the concept of nesting:

Hilbert proved that for n even, not more than (1/2)(n -2) of the p + 1 ovals can be nested; that is, so situated that the first lies inside a second, the second inside a third, and so on; and that curves of even order do exist having p + 1 ovals, (1/2)(n - 2) of which are nested; similarly, that for n odd, not more than (1/2)(n -3) ovals can be nested, if the maximum number of circuits is present, and that curves of odd order do exist having p + 1 circuits, (1/2)(n - 3) of which are nested ovals. (p. 378)

Ragsdale explains iterative nesting in order to emphasize how multiple, progressively smaller ovals can fit within the larger ones. In turn, this concept of nesting would possibly come from the practice of putting one container within another for storage or transport. The Annual Report - Iowa State Commerce Commission from 1889 contains several examples, including these items on the same page referring to baskets:

"baskets, not nested. in bdls ..."

"baskets, N.O.S., nested in bdls ..."

"Baskets, over-handled, covers and handles taken off and packed separately. Baskets nested in bundles."

Solution 2:

(Adding to TaliesinMerlin)

Oxford English Dictionary under nest as a noun, has

5.a. A set or series of similar objects, esp. such a set or series designed to be contained in the same receptacle, or so made that each smaller one is enclosed in, or fits into, the larger one nearest in size to it.

and provides some historical examples:

nest of counters was common in the 15th and 16th cent., nest of goblets in the 16th cent., and nest of drawers in the first half of the 18th cent.; now frequently in nest of tables.

added What is a "nest of counters"?
Google finds the following in the Glossary of the book "Accounts of the Lord High Treasurer of Scotland: A.D. 1500-1504" ...

Another quote found by Google

§ A nest of counters was a set of several counting-boards fitting one with- in the other, like the Chinese toys known as nests of boxes. Nests of boxes of a larger kind and made of tir, the bisgest about ten inches long, are fre- quently imported from Germany to Hull. The word nest has been used in connexion with various other articles. We still sometimes hear of a nest of weights ; our ancestors possessed nests of goblets, bowls, and cups. We llnd from the inventory of Bertram Anderson, a north country gentleman, who died in 1570, that he possessed "a nest of white gobblettes with one cover," and "one nest of boules with a cover duble gilte." — Ullls, cjr. of Northern Cos., ii. 339.