Upon further investigation restarting the SQL Monitor BaseMonitor service resolved this issue. We're investigating why a restart was required, but for the time being if somebody else experiences the same problem with alerts please restart your service.
Edit In fact I would say the newly created Metric looks like it doesn't run at all
Regards
Andrew Lackenby