mongodb - How to store query output in temp db?

Question

I am really new to the programming but I am studying it. I have one problem which I don't know how to solve. I have collection of docs in mongoDB and I'm using Elasticsearch to query the fields. The problem is I want to store the output of search back in mongoDB but in different DB. I know that I have to create temporary DB which has to be updated with every search result. But how to do this? Or give me documentation to read so I could learn it. I will really appreciate your help!

Answer 1

Mongo does not natively support "temp" collections.

A typical thing to do here is to not actually write the entire results output to another DB since that would be utterly pointless since Elasticsearch does its own caching as such you don't need any layer over the top.

As well, due to IO concerns it is normally a bad idea to write say a result set of 10k records to Mongo or another DB.

There is a feature request for what you talk of: https://jira.mongodb.org/browse/SERVER-3215 but no planning as of yet.

Example

You could have a table of results.

Within this table you would have a doc that looks like:

{keywords: ['bok', 'mongodb']}

Each time you search and scroll through each result item you would write a row to this table populating the keywords field with keywords from that search result. This would be per search result per search result list per search. It would probably be best to just stream each search result to MongoDB as they come in. I have never programmed Python (though I wish to learn) so an example in pseudo:

var elastic_results = [{'elasticresult'}];
foreach(elastic_results as result){
    //split down the phrases in this result and make a keywords array
    db.results_collection.insert(array_formed_from_splitting_down_result); // Lets just lazy insert no need for batch or trying to shrink the amount of data to one go or whatever, lets just stream it in.
}

So as you go along your results you basically just mass insert as fast a possible create a sort of "stream" of input to MongoDB. It can do this quite well.

This should then give you a shardable list of words and language verbs to process things like MRs on and stuff to aggregate statistics about them.

Without knowing more and more about your scenario this is pretty much my best answer.

This does not use the temp table concept but instead makes your data permanent which is fine by the sounds of it since you wish to use Mongo as a storage engine for further tasks.

Answer 2

Actually there is MongoDB river plugin to work with Elasticsearch...

Answer 3

db.your_table.find().forEach(function(doc) { b.another_table.insert(doc); } );