There isn't going to be a completely non-hacky way.
The ICCCM recommends identifying toplevel windows (as opposed in particular to WM frames) by looking for the WM_STATE property. So you could distinguish application windows with that. However, when the WM reparents the window into a frame it'll get tricky to find the app window inside the frame window. You'd have to implement a little program to do something like xwininfo -tree but only check for windows with WM_STATE set.
A relatively nice solution would be to use libwnck, or one of the tools based on it (wmctrl, devil's pie). However, this requires an EWMH window manager (which includes anything remotely modern or sane, but it sounds like you are dealing with some really old curmudgeonly stuff).
The advantage of these tools is that they use EWMH to identify application windows and even the semantic type of those application windows (dialog, etc.).
Also, libwnck removes the need to poll, it can just watch for changes in the toplevel window list.