This is not only a good idea, but it is a pretty standard practice (particularly in Direct3D)!
OpenGL implementations tend to use a standard box filter (uniformly weighted) when you generate mipmaps. You can use a nice tent fiter (bilinear) or even cubic spline (bicubic) when downsampling textures in image editing suites. Personally, I would prefer a lanczos filter since this is going to be done offline.
You may already be aware of this, but Direct3D has a standard texture format known as DDS (Direct Draw Surface) which allows you to pre-compute and pre-compress every mipmap level at content creation time instead of load-time. This decreases compressed texture load time and more importantly allows for much higher quality sample reconstruction during downsampling into each LOD. OpenGL also has a standardized format that does the same thing: KTX. I brought up Direct3D because although OpenGL has a standardized format very few people seem to know about it; DDS is much more familiar to most people.
If you do not want to use the standardized format I mentioned above, you can always load your levels of detail one-at-a-time manually by calling glTexImage2D (..., n, ...)
where n is the LOD (0 is the highest resolution level-of-detail). You would do this in a loop for each LOD when you create your texture, this is actually how things like gluBuild2DMipmaps (...)
work.