Python3+mongoDB follows the monthly, quarterly, half-yearly and full-year methods and ideas.

problem description

constant: number of active people, number of registrants, recharge amount, etc.
proportion: proportion of paid people, proportion of lost users, etc.
increase or decrease: the number of registered people rises year on year, the number of lost users decreases compared with last month

the environmental background of the problems and what methods you have tried

at present, constant data can be obtained or "timed time interval" data can be obtained according to the find () syntax in mongodb
but how to batch cycle the annual, quarterly and monthly data for calculation?

if there is source code in the previous question, it is no longer copy and paste. This question is just looking for ideas and methods

Mongodb python

Apr.07,2021

in my practice, this kind of statistical data is made into intermediate data.
A service is scheduled to do statistics at a time point to generate statistics (month, quarter, half year, year) to a new collection.

has also seen online references to streaming (frameworks such as stom ). This is relatively complicated. I haven't put it into practice. We can communicate together.

Previous: How springboot uses redis caching

Next: How to delete a field in MongoDB

Pymongo queries the first 1w of data for a specific condition, and the count result is the number of records of the entire collection. What's wrong with me?
the code is as follows: recode1 = table_out.find ({}). Sort ([( "_ id ", 1)]) .limit (10000) print ( total: , recode1.count ()) directly understand that the number of count () given by print should be 10,000, but what is currently output is the tota...

Mongodb python3.x

Mar.02,2021
How to save the cursor location locally when python traverses mongodb
there is a requirement in the project that a field extracted from the mongodb library at the level of tens of millions of records needs to be transferred to postgresql. because the mongodb is constantly updated, the data needs to be reversed at regular...

Mongodb python

Mar.03,2021
How to associate query data with multiple tables through Pymongo?
MongoDB database multi-table query data before, you could associate two tables with $lookup in Robo 3T . related codes db.users.aggregate([ { $match : { regDate : { "$gte" : ISODate("2018-05-01T00:00:00Z"), "$lt"...

Mongodb python

Mar.28,2021
Time conversion in Python3
the date format stored in MongoDB is 2018-07-01T00:00:00Z the environmental background of the problems and what methods you have tried related codes collection.find_one( { "regDate": { "$gte":ISO...

Mongodb python

Mar.30,2021
Pymongo aggregation module multi-condition, multi-table association, de-duplicate query
problem description 71 related codes 8.3 Update: -sharp def get_registered_paying_user(): unwind = { $unwind : $recharge } -sharp match1 = { $match : { regDate : regDate}} -sharp m...

Mongodb python

Apr.03,2021
Callback in scrapy is useless. I have read the relevant problems on seg, but I can't solve them. I hope you can answer them.
problem description crawl the list of Amazon products, save the data into mongodb crawl the first page and pass the next page link to Request. You can get the next page link in shell but you can only see the first page of data in the database after...

Scrapy mongodb python

Apr.05,2021
How python lists are filtered using a similar method of mongodb aggregation $group
[ { _id : ObjectId( 5abb4f9ca7e2c54c757b3e48 ), amount : 8400, buyerEmail : otzYzwMh24edWk8NxSJOqCSZREe0 , from : weixin , orderDate : datetime.datetime(2018, 3, 28, 8, 17, 28, 940000), orderid ...

Mongodb python

Apr.17,2021
Pymongo aggregates the method for situations where the query condition is not in the table.
user table associated with consumption table want to query the conditions of users who have not consumed for 3 months match2 by the way, how to optimize the following conditions? match0 = { $match : { regDate : { $gte : day_30, $lt : today}...

Mongodb python

May.05,2021
Pymongo merges a collection (table) into another collection
there are currently three collections user consume recharge just want to completely modify the User collection instead of creating a temporary collection like lookup, is there any efficient way, after all, there is a lot of data. currently used $repl...

Mongodb python

May.06,2021
Mongodb $replaceRoot
found mergeObjects and replaceRoot aggregation methods in order to merge collections. examples of official documents are copied down and modified and found that there are no errors available, but there are a few areas that I do not understand. Ple...

Mongodb python

May.10,2021
Scrapy mongodb Joint Group Keys in python to remove duplicate file_url and name Joint Primary Keys
def process_item(self, item, spider): print( ) print(item[ file_url ], item[ name ]) key_word = {"file_url": item[ file_url ], "name": item[ name ]} res = self.db.find(key_word) if...

Mongodb python

Aug.12,2021
Build mongdob index in python code
def process_item(self, item, spider): print( ) print(item[ file_url ], item[ name ]) key_word = {"file_url":item[ file_url ] , "name": item[ name ]} res = self...

Mongodb python

Aug.18,2021
Python indexed mongodb, right?
{ "_id" : ObjectId("5bd6c4f3c3474d25ef31e224"), "unique_charcter" : [ "FqT9lRq7wdoSqjbfvQALU7mf4R0M" ], "name" : "-_2017-2018.3-14.pdf" } { "_id" : ObjectId("5bd6c4f3c3474d25ef31e...

Mongodb python

Sep.27,2021
Problems related to Python+Mongodb and Pandas data Statistics
def getuser(): users = user.find({ totalRecharge : { $gt : 0}, $or : [ { recharge.from : weixin }, { recharge.from : alipay }, { recharge.from : { $ne : jCode }}, ...

Mongodb python

Dec.23,2021
How mongodb updates data
> db.lagou_cookie.find() { "_id" : ObjectId("5c21f6ab65adddf40b330fb3"), "cookie" : "user_trace_token=20180909010719-4eb82332-59f2-4979-b7ba-4a96de35eb40; _ga=GA1.2.1153938840.1536426437; LGUID=20180909010720-a5755fe...

Mongodb python

Apr.01,2022
An error is reported during the operation of scrapy, ModuleNotFoundError: No module named 'pymongo'
I run the single file directly without import errors. In addition, it is normal for me to use mongodb in the py file alone, but when I run it in the scrapy project, I will say that the import failed. Why? import json import pymongo from scrapy.utils.pr...

Mongodb python scrapy python-crawler

Jul.02,2022