To what extent were mathematicians in previous centuries aware of the lack of rigour in their methods?

By modern standards, much of pre-modern mathematics isn't rigorous. Famous examples include Euler's solution to the Basel problem or literally anything involving sets before Cantor and Russel came along, when a "set" was simply a handwaving notion of "all things that have some property", later found to be quite problematic.

To what extent were the mathematicians of the past aware of these shortcomings? Did they even feel mathematics needed to be a rigorous subject, or was there an idea that anything goes as long as it works?

I feel this question is not just of historical interest. Modern mathematics is usually judged (by modern mathematics) to be rigorous, but we have no reason to believe we are able to assess our methods correctly unless our predecessors were capable of assessing theirs.

Lagrange launched a contest at the end of the 18th century whose goal was to clarify the notions of infinity and infinitesimal. There was no clear winner, but Lazare Carnot submitted an entry which eventually became a popular book. This is only one episodes illustrating the fact that rigor has indeed been a concern historically.

A century earlier, there was a debate at the French academy focusing on similar issues, opposing Rolle and Varignon.

In the 19th century, Cauchy lists rigor as one of the objectives of his approach to analysis.

So to answer your question, mathematicians in previous centuries were aware of lack of rigor, but in most cases attributed it to their predecessors only. This seems to be the case today, as well.