Etymology of the use of "Drive" to refer to a digital storage medium

Originally, the drive was not the storage device. It was the mechanism that you mounted the storage device onto.¹ Spools of tape were mounted on the motor spindle of a tape drive, and disk packs and floppy disks were mounted on the motor spindle of a disk drive. The drive imparted to them a spinning motion:

As disk technology matured, it became common for disks to be permanently sealed and non-removable – that is, the disk and the drive became one device. People referred to this device informally as either the disk or the drive. It also became common to refer to it as the hard drive to distinguish it from the floppy drive installed in the same cabinet.²

When companies came along with devices which performed the same function as sealed disk drives, they called them drives too, in order to make clear that they were marketing disk drive replacements: RAM drives, ZIP drives, USB drives, thumbdrives. And when remote file storage functionally replaced these storage devices, it often got the drive label as well: network drive, Google Drive, OneDrive.³

In short, while the word drive still has its original meaning, it now also has a figurative meaning: it can refer to any storage device that takes the place of a traditional storage drive.

Many end users have begun referring to the entire tower or system unit of a personal computer as the hard drive, a further broadening of the term.⁴

Notes

For more about the history of disk drives, see the Wikipedia article “History of hard disk drives”. Much of what I’ve written here also relies on personal knowledge: I have been in the industry for 40 years.

1. Drive in this original sense is attested in American English by Collins English Dictionary:

• Drive, American English sense 28, “a device that communicates motion to a machine or machine part”. (Collins)
• See also the example in American English sense 27a, “any apparatus that transmits power in a motor vehicle: a gear drive”. (op. cit.)

2. The derivative, computing-specific senses of drive are also attested by Collins. You can see the progressive shift of meaning from the device that transmits motion to the assembly as a whole:

• Drive, American English sense 29, “(computing) a unit that reads and writes data on magnetic tape, a disk, etc.” (op. cit.)
• Disk drive, “(computing) the controller and mechanism for reading and writing data on computer disks”. (Collins)
• Hard drive, “hard disk, or the part that contains the hard disk … a computer drive for hard disks, often, specif., such a drive as distinguished from a floppy-disk drive in the same computer”. (Collins)

3. See for example in Collins:

• Thumb drive, “a thumb-sized portable computer hard drive and data storage device”. (Collins)

4. This usage is not yet attested by a dictionary, but is frequently attested by professionals in information technology who work with the public. Representative samples from the wild:

He said he was having a problem with a port on the hard drive... of course he'd actually blown the controller chip on his USB card and the hard drive was just fine (Urban Dictionary, 2003)

When I used to do IT support, the terms "CPU" and "hard drive" were common. As an IT guy I cringed, but at least I knew what people were talking about. (English Language & Usage, 2014)

the blinking lights I'm seeing on my hard drive 2 blinks pause then 3 blinks pause (personal correspondence, 2017)

To add to and strengthen the accepted answer:

Publications from the 1950's (when computers had drum memory instead of disk drives, and tape drives for long-term storage) refer to a tape unit, tape reader, or tape recorder for the whole assembly (tape+drive). See, for example, Frizzell, C. (1953). Engineering Description of the IBM Type 701 Computer. Proceedings of the IRE. 1275ff. and the manual for the IBM 705 EDPM. I find mention of a tape drive as early as 1955 in Eckert's book Faster, Faster. Context makes it clear that drive refers only to the drive proper, and not to the whole unit.

Under some circumstances it would be desirable to erase an old block in the middle of the record and to replace it by a new block of information. This operation requires some care. Obviously one couldn't replace a block with a larger one without running into one of the adjacent blocks. Moreover, slight variations in the speed of the tape drive or slight slippage of the tape would prevent an accurate fit of a new block of data.

As others have stated, "drive" originally referred to the machinery for moving storage media within the device, and has since come to refer to the entire storage device. Here are two early "drive" references of its original sense:

1. In Figures 9 & 10 and nearby text, of an article describing the circa 1951 Uniservo-Tape Reader and Recorder, at https://www.computer.org/csdl/proceedings/afips/1952/5041/00/50410047.pdf , the configuration of the tape reel drives and central drive are described.

2. In Figure 2 of the first image of IBM's 1970 patent for a Direct Access Disk Magnetic Disk Storage Device, available at https://www.google.com/patents/US3503060 , a "Head Drive System" interfaces with "Magnetic Disks".

What is fascinating to me is that the term "I/O Driver" seems to derive from a different usage of the word drive. Compare meanings 1 and 5 from Noah's 1844 Dictionary (available online):
DRIV'ER, n. 1. One who drives; the person or thing that urges or compels any thing else to move. 2. The person who drives beasts. 3. The person who drives a carriage; one who conducts a team. 4. A large sail occasionally set on the mizzen-yard of gaff, the foot being extended over the stern by a boom. – Mar. Dict. 5. In machinery, a wheel which communicates motion to another; used in composition, as in screw-driver. 6. A substance interposed between the driving instrument and the thing driven. A cooper drives hoops by striking upon the driver.

In US patent records, the term drive spindle is attested from at least 1893. In mechanical engineering, a drive spindle is a shaft which rotates in order to drive a disc or wheel.

The earliest appearances of the term in the patent records have to do with mechanical systems for processing agricultural products. Indeed, in the 19th century, factories were generally powered by a single steam engine. This engine drove a wheel, from which a belt drove a shaft, which in turn drove many other belts, each driving a piece of machinery. See, for example, the Gorreana tea factory (now driven by an electric motor, but originally by steam).

Drive spindle appears in connection with sound reproduction in a patent from 1912. The term phonograph drive also appears commonly in the early 20th century.

From there it is a short step to tape drive, since tape was also used for audio recording, and to disk drive, and as mentioned in other answers and comments: motorized devices for moving a tape or a disk so it may be read.

My best guess:

Magnetic tape, of the type used for computer data storage, was invented in pre-WWII Germany but did not become known in most of the world until after the war ended. So any use of magnetic tape would be post-1945.

This is consistent with the appearance of the "magnetic tape drive" associated with the EDVAC computer ca 1948. Likely there were several competing designs for such tape systems, and one would have to research the lines of IBM, Fairchild, Honeywell, Burroughs, and several British and German manufacturers to find the first that used the term "drive".

But, as to the use of "drive", the "tape drive" consists of mainly four components: Two hubs which mount tape reels (one "supply", one "take-up"), a "capstan", and a "read-write head". (In some design there were two capstans, one forward, one reverse.)

The details varied, but basically the hubs contained motors which were pulling in opposite directions, keeping the tape taut, while the capstan rotated against the tape, causing the tape to move forward or backwards across the read-write head. In essence, the capstan would "drive" the tape one direction or the other, and this likely led to the use of the term "drive" to refer to the whole box.

This naming got transferred to disks when removable disk packs came into use in the 60s. The early disk storage units did not have removable packs and, to my recollection, were just called "disks" or "disk storage units". (I tried to find a reference to the hydraulic monstrosity that was attached to the 7094 at Wright-Patterson AFB ca 1970, but I had no luck.)

The removable disk pack units had "packs" or "cartridges" which were the actual disk platters, but they had to be "mounted" to be used (just as tapes were "mounted" for use). Given the conceptual similarity to "tape drives" with regard to this mounting process, calling the receiving units "drives" was a natural extension of the nomenclature (even though colloquially the drives were often referred to as "ovens", due to a remarkable physical resemblance).

Note that the "drives" themselves, both tape and disk, contained no storage, but rather the storage medium was attached to them.

Things sort of got off-track, then, when smaller self-contained "fixed-disk" storage units started being produced, first for mini-computers, then for "personal computers", and the term "drive" was transferred to these. From there the use of the term "drive" in a more figurative sense was transferred to virtual disk partitions, "non-rotating" storage devices (SSDs), and the rest.