Create a figure that is reference counted

If you create the figure without using plt.figure, then it should be reference counted as you expect. For example (This is using the non-interactive Agg backend, as well.)

from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
from matplotlib.figure import Figure

# The pylab figure manager will be bypassed in this instance.
# This means that `fig` will be garbage collected as you'd expect.
fig = Figure()
canvas = FigureCanvas(fig)
ax = fig.add_subplot(111)

If you're only going to be saving figures rather than displaying them, you can use:

def savefig(*args, **kwargs):
    plt.savefig(*args, **kwargs)
    plt.close(plt.gcf())

This is arguably no less hacky, but whatever.