Can I send a Wake On LAN packet without broadcasting?

Wake-On-LAN uses a "magic packet" sent at Layer 2 of the OSI model. Routers need information that is contained at Layer 3 of the OSI model in order to route packets. Switches on the other had are Layer 2 devices, which is why it works when you are just going through a switch. In short, you need to have IP information in order to cross a router, WOL only has MAC address information, so it is not possible to cross the router.

Further, the computer that you are wanting to wake is not turned on, and therefore does not have an IP address, so it would be impossible to send a packet and have it cross the router and reach your computer short of configuring your router to statically route the packet to your NIC's MAC (see Wake-on-Internet for additional options).