Since you posted this in a PowerShell forum, I will assume you are wanting to have a scripted solution to monitor the process.
You can use get-process to monitor the process and it's current memory usage, then if member is above the declared value you can use restart-service to restart the service.
You can set the process in an infinite loop so it will continuously run. Would recommend sleeping if doing this to keep from chewing up resources. Or you could make it a once through script that kicks off periodically using a scheduled task.