What does "worm of yellow convicts" mean?

a worm of yellow convicts

I would understand this to be a metaphoric usage.

The worm refers to a line (or queue), a group of individuals moving in the fashion of a snake or worm (think of the "dragons" seen in Chinese New Year celebrations, how they move).

In "yellow convicts" the term yellow might refer to the color of their prison garb or else a possibly jaundiced (due to malnutrition) look of their skin (though this seems less likely).

In any case, convicts definitely refers to prisoners¹, most likely on a work detail under the supervision of prison guards. Probably (as Xanne notes below) they are chained together in a coffle.

convict n Law

1. A person found or declared guilty of an offense or crime.
2. A person serving a sentence of imprisonment.
   _{TFD Online}

_{¹ Originally I had speculated that these could possibly be prisoners of war, since I had only ever read one book by Crane, A Red Badge of Courage, which was about the Civil War, but I looked up Maggie and discovered that the story takes place much later, so I have deleted that.}

As Robusto says, the "worm of convicts" is a line or queue, snaking out in an undulating and wormlike fashion.

As for the adjective "yellow," I agree that its exact meaning is mysterious. From the small amount of context you gave, I did wonder if it could have meant either "convicts of Asian race" or "convicts of mixed race"; however, the full context (e.g. on Project Gutenberg) makes it clear to me that in this case the word "yellow" does not refer to race.

My best guess is that it refers to the canary-yellow color of the convicts' prison uniforms. I haven't found any evidence that yellow was used in New York prisons, but (A) absence of evidence is not evidence of absence, and (B) Crane's research may not have been super meticulous anyway.

Honestly I can't even figure out what "Island" is being referred to here. I think this opening scene takes place somewhere in the Bowery, and I don't think Rikers Island (est. 1884) would be visible from there... Blackwell's Island (now Roosevelt Island) would be closer but I think still not visible. It might refer to some island being worked by the labor of convicts who were housed somewhere else; or, again, Crane's research might not have been super meticulous. For some reason, I instinctively assume that the "river" in question is the East River.

Notice that throughout this book Crane repeatedly uses the word "yellow" for atmospheric effect, with varying shades of literal and figurative meaning:

• "A wind of early autumn raised yellow dust from cobbles..."

• "She received a stool and a machine in a room where sat twenty girls of various shades of yellow discontent."

• "An orchestra of yellow silk women and bald-headed men..."

• "On a corner a glass-fronted building shed a yellow glare upon the pavements."

• "Some hidden factory sent up a yellow glare, that lit for a moment the waters lapping oilily against timbers."

For the literary connotations of "yellowness" (in all of the above cases except the "yellow silk women"), I suggest "unsavoriness," "filth," and yet (in all cases) "deliberate conspicuousness." See also The Yellow Wallpaper (1892) and The King in Yellow (1895), which neatly bookend Maggie: A Girl of the Streets (1893).

(Per davidbak's comment above: The word "worm" has matching connotations of uncleanliness and filth; as of course does the reference to "squat, ignorant stables." "Worm" also omits the sense of menace that you might infer from a "yellow snake of convicts," and indicates that the queue appears small because it is distant.)

A line of brightly dressed prisoners, moving to or from work, would appear like a worm when seen from a distance.

A simple search confirms that orange or yellow can be used as clothing for convicts. This is partly to make them conspicuous if they should escape. Images of convict uniforms both genuine and for fancy dress