So I found that in my Suspended Messages queue I was getting a lot of the same message. The failed message was not coming out of an orchestration(the orchestration it was generated in from was catching all exceptions) but instead was coming from the send port the orchestration was bound to. What happens in these circumstances is that the orchestration doesn't suspend since that's being caught but the message that goes out from the port(in this case a web service) was failing and ending up in the queue. I wanted to make a note of this in the originating orchestration and then somehow make these suspended messages disappear.
First of all I made a note in the originating orchestration (or rather a coworker did) by simply parsing the exception generated's message property in the catch scope and writing to the event viewer(and/or log4net).
Second, I marked the send port in question, the one bound to the orchestration, such that failed messages from this port were routed to the message box. The messages in question are the requests that got sent out, not the responses(which didn't really exist since the requests were what was responsible for gumming up the works). Here's a pic;
and here is an explanation of what happens(though it could be more clear).
Basically the failed message will not be put into the suspended queue but instead will be put into the message box for routing. Interestingly enough the normal promoted properties(that orchestrations would use for routing I believe) are not promoted. This means it goes back to the message box and things aren't subscribed to it that might normally be. There is however an ErrorReport object with a bunch of properties(listed in the link above). One of these is ErrorType which is, for all such 'failed messages' that get put in the message box, set to "FailedMessage".
What this all means is that we can mark these ports, and then either create a send port that has got a filter(see above) and have the send port just dump the message to a folder(that we then clean up).
Alternatively we can write an orchestration that has a receive shape with the same filter. There's a short explanation of how best to go about this orchestration here near the bottom(post by Greg.Forsythe);
The nice thing about this last option in general is that nothing is generated; the formerly suspended message will disappear into the orchestration and no record will be left. The downside is that you cannot modify the filter expression in the receive shape at runtime(for example if you want to further filter what gets picked up by the orchestration).