As you will read here in the doc, the Reference Data Type is used to store DocumentReferences.

If you want to use it in a query, you cannot use a simple string, neither the UID of the document (i.e. '5gF5FqRPvdroRF8isOwd'), nor the string value that is stored in the field (i.e. '/categories/5gF5FqRPvdroRF8isOwd').

You have to build a DocumentReference and use it in your query, as follows:

const categoryDocRef = firebase.firestore()
   .collection('categories')
   .doc('5gF5FqRPvdroRF8isOwd');

const files = await firebase
  .firestore()
  .collection('tickets')
  .where('category', '==', categoryDocRef)
  .get();

With Firebase Version 9 (Dec, 2021 Update):

You must make a document reference with "categories/5gF5FqRPvdroRF8isOwdand" then use it in your query:

import { doc, query, collection, where, getDocs } from "firebase/firestore";

const categoryDocRef = doc(db, "5gF5FqRPvdroRF8isOwd");

const q = query(
  collection(db, "tickets"),
  where("category", "==", categoryDocRef)
);

const ticketDocsSnap = await getDocs(q);