What does it mean for a set to have "structure"?

Structure is an additional information on the set. It means that there are some relations, constants and operations associated to that set.

For example $\Bbb N$ is a set, but we can give is structure like addition or order, or both. Then the elements of $\Bbb N$ may have importance relative to the structure. If we added the order, then $0$ is a minimum; if we added multiplication then $1$ is a unit; and so on.

It is important to remember that while addition and so on are very natural for us on $\Bbb N$ we can also take relations and operations which make no sense for us, and it will be mathematically valid to do. We can of course give structure to strange sets, and it may not make much sense at first but it is still a valid thing that we can do.

To be able and use the structure mathematically we have a language which allows us to express wanted properties of the structure, or its elements, or subsets, and so on. This language contains symbols for our constants, for any relations we need, as well function symbols.

For example, if we consider $\Bbb Z$ with only the [usual] order for structure then we might as well be talking about it in the language which contain a single relation symbol which we will interpret as $\leq$. If we decide to talk about $\Bbb Z$ with addition and multiplication as well, this structure is called a ring, and we use the language of rings which includes symbols for $+$ and $\cdot$ (and sometimes symbols for the constants $0$ and $1$).

It is important to point out that there is a big difference between language and structure, but they are tied together. Often when we work in mathematics we implicitly assume that we have a language which contains the needed symbols and we interpret it in such way that we give a compatible structure to a particular set.

The basics of this idea can be found in many introductory logic (or model theory) books, where we study language and structure, also Wikipedia.