I would definitely look into Micah's suggestion which involves using a static singleton instance of your converter. But another thing to consider is that if you're using a separated presentation pattern like MVVM you can often avoid the requirement for a value converter by implementing the conversion in the ViewModel.
There's a lot of reasons you might want to do this.
For one, it is much more testable. Your unit tests can be sure that whatever is coming out of the ViewModel is what will be displayed by the UI. You can imagine testing a requirement that dollar values must follow the current culture's currency format, two decimals must be used, etc.
Another good reason is that exceptions in value converters will not be treated as validation errors which can be a huge pain in the butt in Silverlight. Even if you set ValidatesOnExceptions to true in the binding, if your value converter throws an exception, Silverlight will just let it propagate. If however you use the ViewModel to do the conversion, an exception will be treated as a validation error.
The downside is that you lose some of the "reusability" of a general purpose value converter.