Debezium Connector tries to open old log file
I have a debezium connector that works fine, for a limited time. These errors occur in log file:
Caused by: java.sql.SQLException: ORA-00308: cannot open archived log '+RECO/XXXXXXXX/ARCHIVELOG/2022_01_04/thread_1_seq_53874.3204.1093111215'
ORA-17503: ksfdopn:2 Failed to open file +RECO/XXXXXXXX/ARCHIVELOG/2022_01_04/thread_1_seq_53874.3204.1093111215
ORA-15012: ASM file '+RECO/XXXXXX/ARCHIVELOG/2022_01_04/thread_1_seq_53874.3204.1093111215' does not exist
I've learnt in this database log files are deleted daily. Is my connector trying to read an old log file, which does not exist anymore? How can I tell my connector to check only last 12 hours, for example. Or should I do something in database side?
Solution 1:
I've learnt in this database log files are deleted daily. Is my connector trying to read an old log file, which does not exist anymore?
It is fine to delete archive logs that are no longer needed, but it's critical that you make sure that you are not deleting logs that the Oracle Connector still requires in order to perform mining. In your particular case, the connector still required thread_1_seq_53874.3204.1093111215 but the log is no longer on the file system and therefore the connector will stop with an error. This error happens with any other connector such as MySQL if you remove the binlogs before the connector is done reading them.
How can I tell my connector to check only last 12 hours, for example.
You cannot.
The way the Debezium connectors are designed is that they're meant to read all changes from the logs in chronological order to guarantee that there is no change data event loss. If a log were to be deleted that was needed and we did not throw an error, then you would have gaps where changes from the source database would not be represented as change events and so your consumers wouldn't be kept in sync.
Or should I do something in database side
Archive logs need to be retained for as long as they're needed by the connector. The latency of the Oracle connector is dependent both on the volatility of your database but also on a number of factors such as the performance of the database server hardware (disk and cpu), the size of your redo logs, etc.
Some environments may not be able to keep archive logs available in the default destination location for extended periods of time due to space constraints. This is why we introduced a way that you can set up Oracle to write archive logs to a secondary destination location that is capable of retaining the logs for a longer period of time, often via a network mount, and then you can explicitly tell the connector use that archive destination name rather than the first valid/default location of the system.