database vs. flat files
Solution 1:
- Databases can handle querying tasks, so you don't have to walk over files manually. Databases can handle very complicated queries.
- Databases can handle indexing tasks, so if tasks like get record with id = x can be VERY fast
- Databases can handle multiprocess/multithreaded access.
- Databases can handle access from network
- Databases can watch for data integrity
- Databases can update data easily (see 1) )
- Databases are reliable
- Databases can handle transactions and concurrent access
- Databases + ORMs let you manipulate data in very programmer friendly way.
Solution 2:
This is an answer I've already given some time ago:
It depends entirely on the domain-specific application needs. A lot of times direct text file/binary files access can be extremely fast, efficient, as well as providing you all the file access capabilities of your OS's file system.
Furthermore, your programming language most likely already has a built-in module (or is easy to make one) for specific parsing.
If what you need is many appends (INSERTS?) and sequential/few access little/no concurrency, files are the way to go.
On the other hand, when your requirements for concurrency, non-sequential reading/writing, atomicity, atomic permissions, your data is relational by the nature etc., you will be better off with a relational or OO database.
There is a lot that can be accomplished with SQLite3, which is extremely light (under 300kb), ACID compliant, written in C/C++, and highly ubiquitous (if it isn't already included in your programming language -for example Python-, there is surely one available). It can be useful even on db files as big as 140 terabytes, or 128 tebibytes (Link to Database Size), possible more.
If your requirements where bigger, there wouldn't even be a discussion, go for a full-blown RDBMS.
As you say in a comment that "the system" is merely a bunch of scripts, then you should take a look at pgbash.