number of a book in a series

The words book or volume can be used in this manner. Two examples:

[Foundation and Empire] is the second book in the Foundation Series. 1

The Two Towers is the second volume of J. R. R. Tolkien's high fantasy novel The Lord of the Rings. 2

Some users differentiate between the two: using volume may connote that the larger work is one long story, perhaps published in multiple parts. So for a book series where the stories are more self-contained, book would likely be more appropriate.

The word book has many related uses: according to definition 4 in the OED entry, a book is "a main subdivision of a larger written work". This definition occurs in uses like "books of the Bible" or "books of the Iliad". While the Iliad and Bible are usually published as one volume, they can still be separated into books.