How can I view log files in Linux and apply custom filers while viewing?
- Reading through large log files is not a good idea
- you need to first filter them to the necessary lines and then look at them
- even then, if possible (and if the filtered lines are a lot too),
you should run secondary filters and counting scripts to analyze them
- When you have a large static file and know the filters to eliminate lines from it,
it is always a good idea to store the filtered output (rather than trying to look at it immediately).
- This way you can run secondary filters on the filtered output and they will not need to run on the entire thing again
- so, in your example, store the first output to a new file, and when you know another filter apply it on that stored file
- of course, this implies some storage space for the filtered file.
It works better if your filters will reduce the actual file by a large amount
- Regular linux tool like '
grep', 'sed', 'AWK' usually suffice to process text log file very nicely.
I have processed log files in the order of 10GB frequently with these things.
You can make your own tools with these things in 'bash scripts'.
- Don't underestimate '
vim', it can handle large files too (but will take time, so give it filtered files)