Earliest I've been to work
At least in US English, "been" wouldn't be narrow enough for what you're trying to convey. So here are some slight variants of what you proposed, that would be clearer:
This is the earliest I've ever gotten to work or ever started work or ever arrived at work
These seem fine. However, I would say: This is the closest to being on time I've ever been!