I need some help. I've been back and forth on which direction I should go and there are some options of which none I like or can use.
I wrote a generic data dump tool that pulls data from a specified server and dumps it to a comma delimited file. It's configuration and what query to run comes from a SQL table created specifically for this tool. However, I have a new requirement that there are some data dumps that need data pulled from different servers and merged together, but I don't want to alter the tool for this "custom" type of pull/dump. I'm trying to keep it generic so I'm not constantly coding on it. My thought is to create a lib in which my reporting tool can use for each of these custom type of pulls and the data returned by this lib is a SqlDataReader object. However, since this lib will have to pull from different servers and merge the data, is it possibly for the lib to create a SqlDataReader of it's own with this pulled data and returned to the data dump tool or am I think too much into this?
I don't want to return an array because it's not how the tool loops through data now, mainly because some of my existing data dumps are millions of rows, so my existing loop is a datareader loop to keep memory down. However, the libs may create a two dimensional array, as long as it can be converted to a SqlDataReader object before returning. This way I don't have to change much on the looping within my application.
Hope that all makes sense. I have it in my head bouncing around so I ended up writing this like 10 times.
Edit: Keep in mind, each record will be scattered across 3 servers and have to be merged. These are three different processes that work together, but have their own servers. ID from server 1 will relate to Server1ID on Server2 for example.