For Managed resources, this should not be a problem, they will get Garbage Collected. For Unmanaged resources, make sure you have a Finalizer defined for your object, that will ensure the Unmanaged resources are cleaned up.
Also, throwing exceptions from a constructor is considered very bad manners, It's better to provide a Factory Method to do the construction and error handling or to equip your object with an Initialize method that will throw the actual exception. That way the construction always succeeds and you're not left with these type of issues.
Correct, Dispose isn't called by the Garbage Collector, but a Finalizer is, which in turn needs to call Dispose. It's a more expensive technique, except when used correctly. I didn't say so in my answer, did I ;).
You can call into the GC to force a collection run and you can wait for all pending finalizers. It's still better to not put the exception generating code in the construstor or by placing a try/catch around the code in the constructor to ensure Dispose is called on these files in case of an error. You can always rethrow the exception later.