Why does USB 3.1 only use two of the four available data lines?

Solution 1:

USB 3.1 Gen 2 (SuperSpeed+, 10 Gbps) was designed to work over both existing USB 3.0 cables (the ones with the 5 extra contacts), as well as USB Type C cables.

Since existing USB 3.0 cables (the ones with Type A and B connectors, as well as the micro A and B variants) only contain one super-speed pair-of-pairs (Tx pair and Rx pair), USB 3.1 Gen 2 could only use that one pair-of-pairs and still work over existing USB 3.0 cables. So even when you run USB 3.1 Gen 2 over a cable with Type C connectors, it only uses the one super-speed pair-of-pairs. This also makes it possible to have USB 3.0/3.1-capable cables with a Type C connector on one end, and the earlier USB 3.0-style Type A, B, micro A, or micro B connectors on the other end.

Now you might ask a follow-up question, "Why didn't the USB Implementers Forum (USB-IF, the USB standards consortium) define an even-faster-than-10Gbps flavor of the USB protocol, that uses both super-speed pairs-of-pairs in the Type C connector?" That's a valid question, but I'm unwilling to speculate. It would certainly have been a bigger departure from previous USB PHY designs, in that it would have two separate send and receive data streams that would have to be coordinated. In effect, it would be a kind of parallel interface whereas USB has traditionally been nominally serial.

The way you asked your question exposed a few potential misunderstandings that I'd like to address here:

I know that it is possible to run Thunderbolt 3 over any high-quality USB-C cable

That's not quite true. There are many high quality USB Type C compliant cables that are not suitable for Thunderbolt 3. Thunderbolt 3 is limited to ≤ 0.5m cable lengths if you have a passive cable. To go longer than that (like 2m), you need a more expensive active cable (a cable with special IC chips in it to assist in signal handling).

Why doesn't standard USB-C use both of the pairs

USB-C is not a protocol. USB Type C is the name of a connector and cabling specification; it's not the name of the protocols that are used over those connectors and cables. When doing USB protocols over Type C cables, you're doing USB 3.1 Gen 2 (10Gbps "SuperSpeed+"), or earlier flavors of USB.

Solution 2:

USB 3.1 uses one lane (Tx pair and Rx pair) because it is USB standard, to use only one Rx+Tx. All original USB connectors (A, B, microAB) have only one pair of SS (SuperSpeed) contacts, and the entire USB hardware architecture focuses only on one, single-lane design.

To use more lanes in parallel, there should be additional architectural elements in hardware data pipe on how to deal with lane synchronization and other issues associated with individual link-layer management, buffer credit exchanges and error recovery, link training and individual channel electrical optimization. Once you do all this, it is turned into "Thunderbolt". Or something like MIPI.

The Type-C connector is a new standard for CONNECTOR, which was meant to have wider applicability than USB alone. The need for Type-C was largely driven by system design, to fit into smaller form-factors of portable electronics. In some sense it has nothing to do with USB, and your request to re-use all available super-speed lanes to change the standard USB architecture is unfounded.