By default, Monitor will only ignore certain alerts during maintenance windows. You can see the list here: List of alerts suspended during maintenance windows.
You can obtain a list of the available alerts by running the following query in SSMS:
SELECT * FROM [RedGateMonitor].alert.Alert_Type
where [RedGateMonitor] is the name of your SQL Monitor database.
Then simply add the alert you wish to exclude to the exclusion list using: