Is this passage an example of a metaphor, analogy, or both?

I would say it's an analogy, which generally in literature implies the primary subject being compared to something more familiar, in hopes of conveying insights into the true nature of the subject.

In Tolstoy's case, it doesn't matter that his audience probably aren't particularly familiar with differential calculus. He explicitly defines the feature relevant to his analogy; exacting measurement of tiny differences enables one to graph a complex function to present a revealing image.

His purpose being to show that a true understanding of human history in the global and long-term sense can best be achieved by focussing on the minute particulars of specific incidents involving particular people at some given place and time.

Today he'd be more likely to use the analogy of a hologram, which has the interesting property that if you shatter it, you can still retrieve the whole original picture from each small part.

Unfortunately for Tolstoy (who had no great reputation as a mathematician), measuring exact values anywhere on the line of a complex function doesn't actually tell you much about what may happen elsewhere on the graph. And the smaller the piece of broken hologram you look at, the fuzzier the image you can get from it becomes. It's not an analogy that can be taken too far.

I would call this an implied metaphor. Without actually declaring the comparator, Tolstoy is likening humanity to something massive and moving, possibly a river or a glacier — in any case, something composed of minute forces that push the mass along. He then suggests that a kind of calculus could be applied to the progress, measuring smaller and smaller bits, until a larger pattern emerges.