How do I check for a session variable in flask repeatedly across the application?

Your guess is correct. A decorator is a way to review the session and react to the result. In the example below, the user is redirected to the appropriate route based on the decorator if the user is not logged in. If a username is stored in the session, the decorated route is called.

If the login is optional for the respective route, it is public and no decorator is required. But it might be necessary to ask if the user is logged in.

from flask import (
    redirect,
    session,
    url_for
)
from functools import wraps

# ...

def is_authenticated():
    username = session.get('username')
    return username and len(username.strip()) > 0

def login_required(f):
    @wraps(f)
    def wrapper(*args, **kwargs):
        if not is_authenticated():
            return redirect(url_for('login'))
        return f(*args, **kwargs)
    return wrapper

# Add the state query function to the jinja context.
app.jinja_env.globals.update(is_authenticated=is_authenticated)

@app.route('/login', methods=['GET', 'POST'])
def login():
    # your login code here!

@app.route('/secret')
@login_required
def secret():
    return 'You are logged in.'

@app.route('/public')
def public():
    if is_authenticated():
        return 'User is authenticated'
    return 'User is not authenticated'

This code is used to check within the template whether the user is logged in.

    {% if is_authenticated() %}
      User is authenticated.
    {% else %}
      User is not authenticated.
    {% endif %}

If you really want to ask for the session variable before each route, the before_request decorator might be a solution. In this case you cannot avoid storing variables in the g object to use them in your routes. However, this goes beyond your example or unnecessarily complicates the code as it only becomes necessary to use additional data.
I think the code I gave should be enough for simple purposes. For more complex environments I recommend you take a look at Flask-Login or Flask-Security.