What protocols does a typical home router use when talking to ISP?

My home router connects to the ISP via coax cable (Verizon FIOS or Comcast Xfinity, here in the US). Fitting into the OSI 7 layer model, what layer 1 (coax?) and layer 2(still Ethernet?) protocol does it use to talk to the ISP?

Comcast Xfinity, like all ISPs that are also cable TV providers, uses DOCSIS. Most likely version 3.0.

DOCSIS defines both layer 1 and 2. It defines the physical layer as RF signals between 5MHz and 1GHz over 75ohm coaxial cable at certain power levels, and channelized into 6MHz channels (for North America and other NTSC/ATSC TV markets) or 8MHz channels (EuroDOCSIS, for PAL TV markets) just like local TV channels, so it can use unused TV channel frequencies on the cable TV cable so it doesn't interfere with your cable TV reception. It also defines the link layer but that's a bit more complicated to describe here.

DOCSIS stands for "Data Over Cable System Interface Specification", and was developed by CableLabs, a consortium of the cable TV industry. I believe the DOCSIS standards are freely downloadable from CableLabs, and fairly readable as technical standards go (which isn't saying much).

Verizon FiOS started as BPON but is transitioning to GPON ([Broadband|Gigabit] Passive Optical Network). The BPON standard is ITU-T G.983. GPON is ITU-T G.984. These PON standards also specify both layers 1 and 2.

If you get your broadband Internet service from a landline telephone company that hasn't gone to BPON/GPON yet, then it is probably some flavor of DSL (Digital Subscriber Line), most likely ADSL2+ or VDSL2 nowadays. Again, the *DSL standards specify both the physical and data link layers.

Note that I'm really saying what protocol your broadband modem or fiber ONT speaks to the ISP. If your router does not have an integrated modem or ONT, then it uses its Ethernet WAN port to speak to the standalone modem or ONT.