What was the definition of "set" that resulted in Russell's paradox?
The axiom that told Russell that he could consider that to be a set is called the comprehension axiom, which says that for any property $P$, there is a set $X_P$ such that $$ x\in X_P\iff P(x).$$ (We usually write $X_P$ using comprehension notation: $X_P=\{x:P(x)\}$).
The other important axiom here is extensionality, which says two sets are equal if and only if they have the exact same elements: this tells us that the above condition is a valid definition of $X_P.$
Russell applied the comprehension axiom to the property $P(x)=x\notin x$ and then derived his contradiction that $$ X_P\in X_P\iff X_P\notin X_P.$$
The lesson we learned from this is that the comprehension axiom is inconsistent. Thus we do not use it in axiomatic set theory. Instead we use weaker versions, most commonly the separation axiom of ZF that says for any set $A$ and property $P$ that $ \{x\in A:P(x)\}$ is a set.
The reason that Russell thought he could use the comprehension axiom is because it seemed naively true, had been used implicitly before in mathematics with no problems, and had even been singled out as a formal axiom by Frege. His discovery that this seemingly innocuous bit of mathematical reasoning led straightforwardly to an obvious contradiction led to a lot of worry about the formal foundations of mathematics in the ensuing decades, and many interesting things were discovered by logicians trying to interrogate exactly why comprehension failed and how to avoid similar problems.
Russell started from the very trivial axiomatisation of Frege of the concept of set. It consists of only two axioms: one is the estensionality axiom and the other is the comprehension axiom. The point is this axiomatisaton seems very simple and, in the same time, powerful and seems it has captured the real essence of set. In fact the comprehension axiom (often called "unrestricted")
If $\mathcal P$ is a property, then there exists a set $A$ such that $$\forall x \big( x \in A \Leftrightarrow \mathcal P(x)\big)$$
expresses nothing else than our natural (nearly genetical) habit to build any set giving a property to satisfy. Mathematicians before Russell used that axiomatization before Frege wrote it down, but did so tacitly. This axiom, let me say, is very interesting: it essentially says logic and set theory are the different faces of the same medal, that is, every logical question can be translated in term of sets and vice versa, statements correspond to sets, and sets correspond to statements.
Philosophically speaking, this axiom is to too powerful, I can build everything I happen to think, i.e. set of all sets (so $x \in x$?). Briefly, I loose very quickly the control of my constructions. The kiss of death of all that mathematical building is Russell's Antinomy. Just consider the predicate $$x \notin x\,,$$ which allows, cause the comprehension axiom, to consider the set $$R:=\{x \mid x \notin x\}$$ that has the following contradictory property $$R \in R \Leftrightarrow R \notin R\,.$$ This antinomy essentially says you have to pay attention because carefree constructions (as naïve set theory is) can lead to contradictions and antinomies. This paradox is far more than a sophism or a trifle and has an heavy relevance in Mathematics and in Philosophy of Mathematics: we need something with more restrictions, with which you can control what you are doing and avoid antinomies. These are the reasons why axiomatic set theories were born.