Why was "a world" used in this sentence of Melville?

OED:

World 5b. A period or age of human history associated with particular cultural, intellectual, or economic characteristics or conditions, or indicated by the character of those living in it.

1849 T. B. Macaulay Hist. Eng. I. iii. 401 These were men whose minds had been trained in a world which had passed away.

1991 R. Ferguson Henry Miller viii. 154 With such gestures he had already begun his obsessive lifelong concern with the ‘unre-enterable’ world of the past.

A world = an example of a world from many possible worlds.

Melville recognises that "the world" changes and at each stage there is "a world" as opposed to "the world" that makes no reference to change or distinctive society.

Melville was free to choose the definite or indefinite article here. In general, it means very similar things either way. It's hard to say why Melville chose the indefinite article, but we can guess at distinct nuances of using the indefinite article here.

A world ~ a kind of world

The paragraph before a whale describes ships avoiding a specific place because of the fear struck by the floating corpse, followed by years of habit where other ships will also avoid the place. This obedience to past habit is compared to precedent, tradition, old beliefs, and orthodoxy. It is this place (where the corpse is) or this world (a world of orthodoxy) that the ghost instills powerless panic in.

It could be that a world makes the literal or metaphorical place sound hypothetical or partial, thereby contrasting an imagined world of orthodoxy with another world ungoverned by orthodoxy. (The world beyond orthodoxy? The real?) The OED documents a few uses of "a world" under "world, n." I.1.c. that seem similar, each referring to a possible world with an implicit contrast to this world now:

c. gen. Any state or realm of existence, esp. one regarded in contrast to that of contemporary human life (sense 1a). Cf. other world n. 2, possible world at possible adj. 4.

a1797 M. Wollstonecraft Posthumous Wks. (1798) IV. lxvii. 3 A world in which self-interest..is the principal mobile.

2007 Washington Post 29 July (Home ed.) (Book World section) 5/1 Among Other Things, I've taken up Smoking is a coming-out novel about a world we don't quite live in yet, a world in which the dividing line between straight and gay looks..faint.

So a world would refer to something somewhat defamiliarized from the usual associations with the world, namely a world under the effects of an apparition.

A world as generalizing something specific

It's also possible to read this as a generalization of the effects of one whale corpse and its memory in one place. In that sense, II.6.b. in the OED fits particularly well, since many things are likened to a world:

b. In generalized sense: any material or other system likened to but usually distinct from the earth. Usually with indefinite article or in plural. Cf. universe n. 3a.

1676 J. Dryden Aureng-Zebe iii. 33 Too truly Tamerlain's Successors they, Each thinks a World too little for his sway.

1822 Ld. Byron Heaven & Earth i. iii, in Liberal 1 181 When the hot sun hath baked the reeking soil Into a world.

1917 A. Quiller-Couch Notes Shakespeare's Workmanship xvi. 294 So he sleeps, and awakes—to be rewarded..with the best earthly thing that could betide in a world he has served worldlily yet well.

In this sense, a world may help specify that this is not literally the world but rather something less definite, that could be taken for many things: a period of time, a spot of ocean, the mind of a ship crew, and so on.

All the answers so far focus on the meaning of the word "world", ignoring the contrasting context in which it was used in the original sentence. The "world" puts emphasis on the difference between a narrow and a wide influence the whale has; on "his (immediate) foes" and "a (whole) world". In life, the whale only terrified his would-be attackers, while in death its ghost terrifies everyone whoever hears of its story. The "world" is thus a measure of reach, as in, "a person, a family, a country, a world".