Database XP / Over time

Retention is a design decision

Keeping everything forever is a choice with consequences.

Systems accumulate. Rows are inserted and rarely removed, because deleting requires deciding and keeping requires nothing. Tables grow past the point where operations that were routine become difficult, and backups and restores lengthen accordingly.

There are also obligations in both directions. Some data must be retained for a defined period, and some must not be kept longer than necessary, particularly personal information under privacy law. Those requirements differ by jurisdiction and sector and are a matter for the applicable law rather than for engineering preference.

The practical arrangement is to decide, per table, what the retention position is, and to implement it as a routine process rather than as an occasional clean-up. A process that runs monthly and removes a little is unremarkable; one that runs after five years removes a great deal and is frightening.

Archival rather than deletion is often the right answer where the data may be needed but not operationally. Moving it out of the working tables keeps operations fast without losing anything.

There is a retention consideration specific to backups that organisations frequently miss. Deleting personal data from the live database does not remove it from the backup set, and where a deletion obligation applies, the treatment of historical backups is a question that needs a considered answer rather than an assumption.