It is really down to how you are building your screens.
I don't know how you are building your screens but say I had an edit contact page, and part of that page displayed the list of activities and then allowed you to add/edit/delete activities. I would use Ajax to handle this and then create an Activity WebApi controller (if you are using MVC4) to handle the Ajax calls.
If you are using MVC3 then I would add the methods to the Contact controller.
EDIT:
In support of @Husein who presented his case much better than I did - it is definitely about how you want to deal with the object graph. My description above makes the assumption that you are not dealing with activities in isolation - they are dealt with only as part of a Contact object.