How does Content-Security-Policy work with X-Frame-Options?

The frame-src CSP directive (which is deprecated and replaced by child-src) determines what sources can be used in a frame on a page.

The X-Frame-Options response header, on the other hand, determines what other pages can use that page in an iframe.

In your case, http://a.com with X-Frame-Options: DENY indicates that no other page can use it in a frame. It does not matter what http://b.com has in its CSP -- no page can use http://a.com in a frame.

The place where X-Frame-Options intersects with CSP is via the frame-ancestors directive. From the CSP specificiation (emphasis mine):

This directive is similar to the X-Frame-Options header that several user agents have implemented. The 'none' source expression is roughly equivalent to that header’s DENY, 'self' to SAMEORIGIN, and so on. The major difference is that many user agents implement SAMEORIGIN such that it only matches against the top-level document’s location. This directive checks each ancestor. If any ancestor doesn’t match, the load is cancelled. [RFC7034]

The frame-ancestors directive obsoletes the X-Frame-Options header. If a resource has both policies, the frame-ancestors policy SHOULD be enforced and the X-Frame-Options policy SHOULD be ignored.

An older question indicated this did not work in Firefox at that time but hopefully things have changed now.

UPDATE April 2018:

Content Security Policy: Directive ‘child-src’ has been deprecated. Please use directive ‘worker-src’ to control workers, or directive ‘frame-src’ to control frames respectively.

Looks like child-src is now the deprecated one and frame-src is back.

None of your hypotheses are universally true.

• Chrome ignores X-Frame-Options.
• Safari 9 and below ignore CSP frame-ancestors.
• Safari 10-12 respect the CSP frame-ancestors directive, but prioritize X-Frame-Options if both are specified.