As AOE, area of effect, is always a circle, the first thought I came up with was to calculate the distance between each unit in the map to the center of the circle, and to determine a unit is in the range of the circle by the formula (Xunit - Xcenter)2 + (Yunit - Ycenter)2 < R 2 , where R is the radius of the circle.
Apparently, it is not an efficient algorithm. Maybe it can be improved by downsizing the calculating area first, and then use the formula above. But calculations are still time consuming operations, and hashing may be an efficient way to solve this problem while I actually don't know how to do :( . And I REALLY wonder what the algorithms are used in the games.