The official account has hundreds of thousands of followers. How to quickly run Synchronize to the local database when docking?

I try to develop a third-party platform using EasyWechat, and then I will help local merchants or official accounts run some activities. How to quickly transfer the official account fan data Synchronize to the local database to facilitate operation after the official account is authorized and docked? My previous idea was that Synchronize would be triggered when users interact with each other. At present, manual methods and command line methods are used, which are as follows:

/**
     * 
     */
    public function sync()
    {
        $account = $this->request->param("account", 0, "intval");
        $account = AccountService::getDataById($account);
        if (empty($account)) {
            $this->error("");
        }
        //
        $page = $this->request->param("page", 0, "intval");
        $page_size = 50;
        //
        $wechat = WechatService::applicationInit($account);
        
        //
        $fans = cache("wechat_fans_".$account->id);
        if (!$fans) {
            echo "";
            //
            $datas = $wechat->user->lists();
            $total = ceil($datas["total"] / $datas["count"]);
            $fans = $datas["data"]["openid"];
            for ($i = 1; $i < $total; $iPP) {
                $datas = $wechat->user->lists($datas["next_openid"]);
                $lists = $datas["data"]["openid"];
                foreach ($lists as $k => $v) {
                    array_push($fans, $v);
                }
            }
            //
            cache("wechat_fans_".$account->id, $fans);
        }
        foreach ($fans as $k => $v) {
            if ($k <= $page * $page_size) {
                continue;
            }
            if ($k > ($page + 1) * $page_size) {
                return $this->success("", url($this->request->controller()."/sync", ["account" => $account->id, "page" => $page + 1]));
            }
            echo "".$k."<br/>";
            try {
                UserService::syncDataByServer($account, $v);
            } catch (\Exception $e) {
                echo "".$k."<br/>";
                continue;
            }
        }
        //
        cache("wechat_fans_".$account->id, null);
        return $this->success("", url($this->request->controller()."/index", ["account" => $account->id]));
    }

this way of using paged Synchronize to execute in the browser is very slow, about tens of thousands of fans have to run for a long time, and then similar to this I wrote a command line way, the execution is also very slow.

<php
namespace app\wechat\command;
use app\wechat\service\AccountService;
use app\wechat\service\WechatService;
use app\wechat\service\UserService;
use think\console\Command;
use think\console\Input;
use think\console\Output;
use think\console\input\Option;

/**
 * 
 * Class SyncUser
 * @package app\wechat\command
 */
class SyncUser extends Command
{
    /**
     * 
     */
    protected function configure()
    {
        //
        $this->setName("Wechat:SyncUser")
            ->addOption("account", 0, Option::VALUE_REQUIRED, "ID.")
            ->setDescription("");
    }

    /**
     * 
     * @return int|null|void
     */
    protected function execute(Input $input, Output $output)
    {
        ini_set("memory_limit", "1024M");
        if (!$input->hasOption("account")) {
            $output->writeln("");
            return null;
        }
        $account = $input->getOption("account");
        $account = AccountService::getDataById($account);
        if (!$account) {
            $output->writeln("");
            exit;
        }
        //
        $wechat = WechatService::applicationInit($account);
        //
        $fans = cache("wechat_fans_".$account->id);
        if (!$fans) {
            $output->writeln("");
            //
            $datas = $wechat->user->lists();
            $total = ceil($datas["total"] / $datas["count"]);
            $fans = $datas["data"]["openid"];
            for ($i = 1; $i < $total; $iPP) {
                $datas = $wechat->user->lists($datas["next_openid"]);
                $lists = $datas["data"]["openid"];
                foreach ($lists as $k => $v) {
                    array_push($fans, $v);
                }
            }
            //
            cache("wechat_fans_".$account->id, $fans);
        }
        foreach ($fans as $k => $v) {
            try {
                UserService::syncDataByServer($account, $v);
            } catch (\Exception $e) {
                $output->writeln("{$k}");
                continue;
            }
        }
        //
        cache("wechat_fans_".$account->id, null);
        $output->writeln("");
    }
}

ask the gods if they have a better plan. Please be more detailed. I am not a professional programmer. I am too brief to understand. Thank you!

Php
Mar.09,2021

after users have authorized access, there are two main parts for carrying out fan data Synchronize, one is historical fan data Synchronize, which uses the backend program to start Synchronize after the official account is authorized. The API for obtaining user list can only get 10,000 openid, one official account and one thread at a time. There is no need to finish running immediately. There is a frequency limit on the official account API Wechat itself. If you think the speed is really slow, you can also open a few processes to deal with it. In fact, the official accounts of tens of thousands of fans will soon be finished, and there is no need to perform tasks manually in the background.
the second is the new fan data. In following and unfollowing events, you can process the fan data in the event callback, so that you don't have to pull all the fan data again.
in fact, tens of thousands of fan data was quickly processed. I ran through the data of more than 20 million fans in just a few days, and the frequency of API calls was very slow.


do you want to pull official account fans to your local database? If a single entry to the city is slow, you can start multiple processes to run


is it UserService::syncDataByServer ($account, $v); this method is slow? Try it in bulk Synchronize? And putting hundreds of thousands into a fans array consumes a lot of memory. It feels like it can be combined with every 10, 000 pieces of data obtained by Wechat and then processed in batches.

Menu