What does AoS mean?

It stands for "Aeon of Strife", which was a really popular map from Starcraft. It had a mission where you rescue someone using only heroes, which was the idea that gave birth to DotA (Defence of the Ancients, A popular Warcraft 3 total modification, which has spawned an entire new multiplayer genre) and similar maps.

An AoS map in Warcraft 3 is a map which is set up in a similar way. You rescue someone using only heroes, combatting both AI and Player Controlled characters. Source

Иво Недев's answer is plain wrong in the context of Warcraft 3's online custom maps scene. The sourced Yahoo Answer is misinformed. tl;dr "AoS" or "AoS map" is what we called MOBAs before that term existed; an AoS map is symmetrical head-to-head; co-op maps where you adventure towards some end goal like rescuing someone are called other things.

The map/game "Aeon of Strife" in particular did indeed involve a team of players against waves of AI enemies. It's reasonably well-documented elsewhere, though there's a lot of misinformation that implies that the earliest versions were competitive - not the case. Probably nobody has the original anymore, but this one is an almost exact match - a 4 player co-op vs. AI. However, "AoS map" in Warcraft 3 was generally understood to mean only "Aeon of Strife-style maps", a wholly separate concept and kind of a misnomer. Aeon of Strife itself, confusingly, would not be considered an "AoS map".

If you look under the "AoS" category in Epicwar, Hive Workshop(+), WC3 Campaigns (registration required for search, but see example), or Nibbits, some of the major portals of WC3 modding, you'll find that in the era of Warcraft 3, the acronym was commonly used to mean the type of game described below - which you'll see is almost exactly identical to "MOBA".

There's general agreement that a set of maps (primarily inspired by Aeon of Strife) with "Defense of the Ancients" in their name first made the small step of replacing the co-op format with symmetrical player teams, and managed to achieve a following sometime around 2003. (There must have been one single earliest "Defense of the Ancients", but it never won dominance and copycats appeared so soon that it doesn't mean much to call special attention to it.) Later on DotA Allstars was created as a collection of the best hero ideas from the wider set of DotA maps. The rest of DotA's story should be rather familiar.

In the time between the first DotA and the first standalone MOBAs (led by Heroes of Newerth), there emerged a vast profusion of maps which coalesced into a recognizable formula early on. There's a huge amount of fascinating, undocumented, and now lost history about these developments, and I recommend https://lanepushinggames.com/ for some insights. Though there was a lot of diversity in the details, they all shared roughly these characteristics:

• Two (usually) opposing teams of players each in control of one or a very small number of units, usually a Hero, with little to no base-building element. A few games (most notably Eve of the Apocalypse: Twilight (+)) had a more involved strategy layer with outpost building and such.
• AI-controlled mob units for each team, which periodically spawned and moved down roughly symmetrical "lanes" towards the enemy base (the unqualified term "creeps" originally indicated these only, with "jungle creeps" or “forest creeps” used to indicate what MOBAs now call “jungle camps”)
• Usually a significant emphasis on power progression over the course of a match, especially through acquiring hero items
• A symmetrical victory condition involving destruction of an AI-controlled base. Some maps had stronger asymmetry - games like Advent of the Zenith and Rise of Winterchill featured a unique set of heroes and statistically different creeps for each team, and a few like Aerie of Ruin had a significantly asymmetrical strategy layer. However, the teams were almost always intended to have even chances of victory.

In short, ”AoS" specifically indicated games where opposing teams use heroes to fight each other with creep support on similar footing to achieve their side of a mirrored primary objective. These are generally called "MOBAs" nowadays.

"DotA-like" never gained much traction because it couldn't be shortened without being conflated with DotA Allstars itself, many players held great disdain for DotA for its relative lack of technical sophistication and innovation, and there was no single dominant progenitor of the head-to-head iteration of the genre, so a more exact term never appeared, leaving "AoS-style map" as the de facto name for the archetype. In time this was shortened to just "AoS", and people who wanted to talk about the original Aeon of Strife used the full name. It was common to see game lobbies marked with “AoS” to attract players of the genre who were unfamiliar with the specific map being hosted. The new standalone games following this format understandably didn't want to self-identify their genre with a term so specifically tied to WC3, and so "MOBA" replaced "AoS" in wider usage.

A game like Starcraft's Aeon of Strife in which "you rescue someone using only heroes, combatting both AI and Player Controlled characters" (i.e. a goal-oriented co-op game) in that time would usually have been agreed to fall under one of the following categories:

• "Hero Defense"/"Hero Survival"/"Hero Siege", if you were mostly on the defense against waves of incoming enemies. "Hero Siege", like "AoS", refers both to a few specific maps with "Hero Siege" in their name (Hero Siege X was the most popular, I believe, though even that name was attached to several different maps maintained by different authors) and to a category of game. The term is meant to differentiate these maps from other defense-type maps, the difference being that you use primarily heroes rather than bases or towers to defend. "Hero Siege" tended to involve some offense objectives, "Hero Defense" was most similar to tower defense often with a base to defend, and "Hero Survival" usually meant a less spatially organized map where the goal was direct survival rather than protecting some objective, but the categorizations are very muddy (see e.g. Enfo's Team Survival, which I'd say is an extremely orthodox hero defense, rather than a hero survival).
• "RPG" if it was longer, or had a significant story component. Games with enough content to warrant implementing some sort of save code system (Warcraft 3 itself had no provision for storage of any data between matches, and the vanilla multiplayer save system was too limited to use in this role) for continued play between different sessions also tended to be grouped here. There was a different generally accepted term, "Open RPG" or "ORPG", to refer to maps like The Black Road or Gaias Retaliation which would more closely match what the rest of the gaming world considers an RPG, though the two terms intermixed frequently.
• "Dungeon Crawler", if the game had a more regimented format usually involving separate zones interspersed by safe "prep areas" where you purchased resources, etc. Famous examples include Obsidian Depths and Tomb of Jarahcon. Also used for RPG games that involved actual dungeon crawling, like Sunken City. Impossible Bosses (+SC2) would also fall under this category by the defined criteria, but it's a much less obvious fit for the name itself.

The various copies of the Starcraft version of Aeon of Strife that made their way into WC3 were considered hero defense maps. The boundaries of the Hero (X) category were generally pretty clear, but Dungeon Crawler and especially RPG leaked all over each other and into other genres.

However, the common point was that these were primarily cooperative, asymmetrical games, with a team of many players against AI. Though somewhat rare, there were examples of each genre where a player, not an AI, controlled all or part of the asymmetric opposing forces.

(It's interesting to note that unlike in Starcraft, Aeon of Strife itself never had a "canonical" iteration in the Warcraft 3 custom maps scene - i.e. there was no single map in the set of those with "Aeon of Strife" in the name that was clearly more popular or considered a progenitor to the others. Though the vast majority of these maps are lost to time, you can get a cursory sample of the survivors with a search through Epicwar. The perception from the earliest days that "AoS" was a category containing considerably different games rather than one game might have contributed to the community's willingness to use it to name a whole genre.)