How do I require one field or another or (one of two others) but not all of them?

The problem is the "not" semantics. "not required" does not mean "inclusion forbidden". It just means that you don't have to add it in order to validate that schema.

However, you can use "oneOf" to satisfy your specification in a simpler way. Remember that it means that "just one of these schemas can validate". The following schema achieves the property switching you are attempting to solve:

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "object",
    "required": [
        "unrelatedA"
    ],
    "properties": {
        "unrelatedA": {
            "type": "string"
        },
        "fileNames": {
            "type": "array"
        },
        "copyAll": {
            "type": "boolean"
        },
        "matchesFiles": {
            "type": "array"
        },
        "doesntMatchFiles": {
            "type": "array"
        }
    },
    "oneOf": [
        {
            "required": [
                "copyAll"
            ]
        },
        {
            "required": [
                "fileNames"
            ]
        },
        {
            "anyOf": [
                {
                    "required": [
                        "matchesFiles"
                    ]
                },
                {
                    "required": [
                        "doesntMatchFiles"
                    ]
                }
            ]
        }
    ]
}

If the property having a value of null is as good as it not being there, then something like this might be suitable. commonProp must be provided, and only one of x or y can be provided.

You might get a couple of similar error messages though.

{
    $schema: 'http://json-schema.org/draft-07/schema#',
    type: 'object',
    required: ['commonProp'],

    oneOf: [
        {
            properties: {
                x: { type: 'number' },
                commonProp: { type: 'number' },
                y: {
                    type: 'null',
                    errorMessage: "should ONLY include either ('x') or ('y') keys. Not a mix.",
                },
            },
            additionalProperties: { not: true, errorMessage: 'remove additional property ${0#}' },
        },
        {
            properties: {
                y: { type: 'number' },
                commonProp: { type: 'number' },
                x: {
                    type: 'null',
                    errorMessage: "should ONLY include either ('x') or ('y') keys. Not a mix.",
                },
            },
            additionalProperties: { not: true, errorMessage: 'remove additional property ${0#}' },
        },
    ],
}

const model = { x: 0, y: 0, commonProp: 0 };

// ⛔️ ⛔️ ⛔️ ⛔️ ⛔️ ⛔️
// Model>y should ONLY include either ('x') or ('y') keys. Not a mix.
// Model>x should ONLY include either ('x') or ('y') keys. Not a mix.

const model = { x: 0, y: null, commonProp: 0 };

// ✅ ✅ ✅ ✅ ✅ ✅

const model = { x: 0 };

// ⛔️ ⛔️ ⛔️ ⛔️ ⛔️ ⛔️
// Model must have required property 'commonProp'