fetch items from database in steps and perform operations sequentially
I have a database with a lot of items and I need to perform an action on each of these items.Unfortunatly, I have to run these actions sequentially and also delay every action to avoid rate limit.
My approach does not wait for the previous actions to be finished and I end up getting rate limited. What do I have to change in order for it to run sequentially?
setInterval(async () => {
await this.processQueue();
}, 1500)
private async processQueue() {
try {
//Only 3 requests per second allowed by the API so I only take 3 items from the database on every call
const bids = await getRepository(Bid).find({ order: { created_at: "ASC" }, take: 3, skip: 0 })
if (bids) {
for (const bid of bids) {
//check if active order exists
const activeOrder = await this.accountService.hasActiveOrder(bid.tokenId, bid.tokenAddress, 0);
if (!activeOrder) {
//perform async functions with that db item
const startTime = Date.now();
await this.placeBid(bid);
//delete from database so the next call to processQueue does not return the same itemsagain
await getRepository(Bid).delete({ id: bid.id })
const endTime = Date.now() - startTime;
}
}
}
} catch (error) {
console.error("TradingService processQeueu", error.message);
}
}
There's no coordination between your interval timer and the work being done by the processQueue
function. Passing an async
function to setInterval
as its callback is misleading and not useful; setInterval
doesn't use the callback's return value, so the promise it returns isn't used for anything.
Instead, the minimal change would be to use something that does wait for processQueue
to finish, perhaps a chained series of setTimeout
callbacks:
function processAndWait() {
processQueue().then(() => setTimeout(processAndWait, 1500));
// (Normally I'd have a `catch` call as well, but `processQueue`
// ensures it never rejects its promise)
}
Note that that waits 1500ms after the queue has been processed. That's probably overkill, if the API allows up to three calls per second. You could probably trim it to 1000ms.
Or if this is a method (public or private) in a class:
processAndWait() {
processQueue().then(() => {
this.queueTimer = setTimeout(() => this.processAndWait(), 1500);
});
// (Normally I'd have a `catch` call as well, but `processQueue`
// ensures it never rejects its promise)
}
Note that I added something saving the timer handle to a property, so you can use clearTimeout(this.queueTimer)
to stop the process.