Is there a solid reference/rule on when not to put article after 'of'

In this you would have first to do a distinction between countable and uncountable (substance) things. Perhaps there could be a reference text, but I will try a roundup.

This shouldn't be mysterious:

A countable object can be counted in units (1, 2, 3, 4, 5...). One can enumerate one car, two cars, etc (the equivalent in mathematics would be a discrete value). So it can be used in singular and plural form.
Milk being a substance, in most cases you don't count it in units such as 1, 2, 3: if you add milk into a pot you don't add "one milk" or "two milks*, but a quantity such as 1 gallon or 1.3 gallons. It is still milk (singular). It is described as uncountable (the equivalent in mathematics, would be a continuous value).

Uncountable Names

"Of" is often used with uncountable names. The rule is simple: if this refers to the substance in general, there is no article:

It is made of silk.

We have an increase of temperature.

I want two pints of water.

If the substance is specific (typically to a certain place and time), it is possible to specify this:

I want more of the silk from that shop.

Today, we have an increase of the temperature.

We noticed a change in the color of the water in the fish tank.

Countable Names

With countable names, the rules is similar: omit "the" when it refers objects of a certain category, and use "the" when it refers to specific objects.

So, in general:

He has a good opinion of writers.

Or in particular:

He has a good opinion of the writers of that article.

I want a review of the contract.

For your question about hotel, there is a nuance, which is quite perceptible:

I need a change of hotel.

A "change of hotel" is an often practiced action. You don't want to be particularly specific. You just want a new hotel, for whatever reason.

I need a change of the hotel.

You are being specific about that hotel. In that particular case, you are conveying there is something wrong with it. Perhaps you would avoid it if you want to avoid confrontation ("What's wrong with that hotel?"), or else you would use it to make yourself extra-clear.

Similarly, when you open a conversation with a travel agent:

I want a change of reservation.

(you want to activate a procedure called "change of reservation")

But if the travel agents already knows what reservation you are talking about:

I want a change of the reservation.

Conclusion

Regardless of countable or uncountable, it's a about being specific or not specific. In many cases, the context will dictate which one you need to use. In others, you could decide whether you want to use one or the other, so as to express the exact nuance.

As a caveat about uncountable versus countable, this is not necessarily a feature connected to the word itself, but to the meaning you are giving to it.

For the example of "oil":

Uncountable: Add two table-spoons of olive oil.

Countable: This supermarket stocks several vegetable oils (olive, soja, etc.).

Uncountable: This supermarket stocks different types of vegetable oil.

If you wanted to be specific: