How can I open a System 9 FileMaker document in OS X 10.9?

Which version of FIleMaker created the file that you wish to open?

Presumably, you are talking about a file created by FileMaker 5 or prior, which was the last version not to support OS X, which was introduced in FileMaker 5.5 (April 2001). MacOS 9 support was dropped in FileMaker 7 (March 2004). Source Wikipedia: FileMaker Version History.

Regardless, the current versions of FileMaker (v14 was released just this month) only offer backwards compatibility with files created with earlier recent versions of FileMaker (v13 and v12). v11 and earlier need to be converted to the latest file format in order for them to be opened with FileMaker Pro 14. Thanks to user3439894 for that information.

However, I guess that you are looking for a free database application that offers compatibility with FileMaker. I would have suggested OpenOffice or LibreOffice. However as user3439894 has commented below, you can not by default, open FileMaker files with these two applications, unfortunately.