Skip to content

Conversation

@ocean1
Copy link
Contributor

@ocean1 ocean1 commented Apr 17, 2014

a little more flexible approach on storing data effectively on mongodb allowing to map-reduce/aggregate later:

  • added a groupmeta_collection option to save group results in a different collection than results
  • added a 'bson' fake kombu encoder to allow pymongo to serialize natively data in mongodb
  • modified the way groups are saved/rebuilt to avoid serializing the whole object but just the id and recreating the AsyncResult object(s) when restoring the group.

…t collection than results and add a 'bson' fake kombu encoder to allow pymongo to serialize natively data in mongodb
@alyssaq
Copy link

alyssaq commented Apr 24, 2014

Yes please. It would be nice for celery to have the option to store the results as native json and not binary into mongodb. There was a similar issue raised in the past but had no pull request #1200.
This pull request seems to address it.

@ocean1
Copy link
Contributor Author

ocean1 commented Apr 24, 2014

This address the first request of #1200, the second one is not really needed, since you would need the object id nevertheless (or task id for results), so the best option is to save to a collection all the data we need before the task is started and reference this in the results with its object id.

@ask ask merged commit 0762057 into celery:master Aug 19, 2014
ask added a commit that referenced this pull request Aug 19, 2014
@ask
Copy link
Contributor

ask commented Aug 19, 2014

Thanks! Merged into master.

I don't think you need to register a kombu serializer for this, so I changed it to override the backend methods instead.

I also don't think one should expect to get stores group results by doing AsyncResult(group_id).get().

Can you also add your name to https://github.com/celery/celery/edit/master/CONTRIBUTORS.txt?

ask added a commit that referenced this pull request Aug 19, 2014
@AlJohri
Copy link

AlJohri commented Mar 7, 2016

Is there any reason this hasn't been backported to the 3.1 release? The part of the code that does

if self.serializer in BINARY_CODECS:
    payload = Binary(payload)

fixes this issue: #2081

@AlJohri
Copy link

AlJohri commented Mar 7, 2016

actually, this still doesn't quite fix it

'result': self.encode(result) needs to be changed to 'result': result as shown in #2081 (comment)

@AlJohri
Copy link

AlJohri commented Mar 8, 2016

@ask what are your thoughts on adding a configuration option allowing to store to content as a raw dict instead of doing self.encode(result)?

@matt-snider
Copy link

@AlJohri @thedrow Any update on this, or a new issue where we can discuss it? I definitely think having the raw json in mongodb makes sense, otherwise there isn't a whole lot of benefit to using mongo in the first place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants