In real-world settings like cybersecurity, data arrives continuously over time. New threats emerge, behaviors evolve, and attack patterns shift, rendering static machine learning models ineffective. Continual Learning (CL) offers an adaptive solution by training models on a sequence of tasks X_1, X_2, ..., X_T without starting from scratch. However, a key challenge in CL is catastrophic forgetting, where learning new tasks leads to severe performance drops on previous ones due to the overwriting of learned representations. In this talk, I will introduce a formal perspective that casts Continual Learning for neural networks as a numerical optimization problem, an approach still uncommon in the literature. Specifically, given task-specific loss functions, the goal is to minimize the total loss over all tasks while being restricted to data from the current task X_t, and limited samples from past ones. This constraint demands optimization algorithms that balance plasticity (learning new information) with stability (preserving prior knowledge). I will reinterpret key continual learning methods through this optimization lens, including: - Regularization-based approaches (e.g., Elastic Weight Consolidation), - Replay-based strategies (e.g., Experience Replay), - Gradient-based constraints (e.g., A-GEM). The second part of the talk focuses on applying this framework to AI-assisted Intrusion Detection Systems (IDS), a domain where adaptability and memory are critical. I will present a continual learning pipeline for IDS, analyze its unique challenges (e.g., class imbalance, shifting distributions), and compare strategies to mitigate forgetting in this dynamic setting. This talk bridges continual learning, neural networks, and cybersecurity, aiming to provide a principled foundation and practical tools for building resilient AI systems.