c# - Membership.DeleteUser() fails with RI Constraint

Question

I'm missing a concept here. I was assuming that Membership.DeleteUser() would expunge a user from my membership tables. My code:

// remove all but 'admin' from Membership
MembershipUserCollection users = Membership.GetAllUsers();

foreach ( MembershipUser user in users )
{
    if ( user.UserName != "admin" )
    {
        Membership.DeleteUser( user.UserName );
    }
}

DeleteUser() fails with exception:

The DELETE statement conflicted with the REFERENCE constraint 
"FK__aspnet_Me__UserI__58D1301D". The conflict occurred in database 
"MyDatabase", table "dbo.aspnet_Membership", column 'UserId'.
The statement has been terminated.

In my Web.config:

<membership defaultProvider="MyMembershipProvider">
  <providers>
    <clear/>
    <add name="MyMembershipProvider" type="System.Web.Security.SqlMembershipProvider" connectionStringName="MembershipConnectionString" enablePasswordRetrieval="false" enablePasswordReset="true" requiresQuestionAndAnswer="false" requiresUniqueEmail="false" maxInvalidPasswordAttempts="5" minRequiredPasswordLength="5" minRequiredNonalphanumericCharacters="0" passwordAttemptWindow="10" applicationName="MyApplication"/>
  </providers>
</membership>

I get it that there's a foreign key relationship between Membership.UserId -> Users.UserId but would have assumed that the whole point of DeleteUser() is to remove all the records for this user in the Membership, User and UsersInRoles tables, to name a few.

Of course, I could go straight into the Membership table and delete the appropriate record, but that's defeating the purpose of using the API. What am I doing wrong? What is the correct way to delete a user from the Membership tables?