Valerio Tesei
2018-05-24 11:31:10 UTC
Hi There,
I'm trying to learn node and I've come across a problematic question, used
to use ml during my C days, I thought it may be good to ask the community.
As exercise I've built a small system to read from a database and push data
to a remote server.
Something simple, I thought...
I've 20.000 elements that need to go from a local database to a remote rest
api.
in PHP I would:
1. load from database
2. process the data
3. post digest to remote endpoint
Attempting to do this in NodeJS I've encountered a big issue, the remote
server was denying the requests due a too high request rate.
Didn't took me too much time to understand I was flooding the remote
endpoint.
The only solution I've found is to execute a cascade of promises with a
timer polling the requests status and allowing only 1 request at the time.
1. load from database
2. push to an array
3. use setTimeout to execute the polling fn
4. every time a request is completed, execute callback for next element
1. each callback sets either error or done for the element in the q
The pace becomes 1 request at the time, as fast as possible (within ~tickMs
intervals)
I could add a concurrency counter to execute X requests at the time, but it
does not address my concerns just works them around.
I know is barbaric, that's why I am here.
https://pastebin.com/jZZZhSyf
I did not use events, setTimeout was faster to use ;)
My point is, how do you manage efficiently how often execute asynchronous
operations? how would you solve my problem?
Any comment about the code is welcome, I'd love feedback from experienced
developers,
Cheers,
--
Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/8c6342f8-be9a-4366-b9ce-4526e852d9ec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
I'm trying to learn node and I've come across a problematic question, used
to use ml during my C days, I thought it may be good to ask the community.
As exercise I've built a small system to read from a database and push data
to a remote server.
Something simple, I thought...
I've 20.000 elements that need to go from a local database to a remote rest
api.
in PHP I would:
1. load from database
2. process the data
3. post digest to remote endpoint
Attempting to do this in NodeJS I've encountered a big issue, the remote
server was denying the requests due a too high request rate.
Didn't took me too much time to understand I was flooding the remote
endpoint.
The only solution I've found is to execute a cascade of promises with a
timer polling the requests status and allowing only 1 request at the time.
1. load from database
2. push to an array
3. use setTimeout to execute the polling fn
4. every time a request is completed, execute callback for next element
1. each callback sets either error or done for the element in the q
The pace becomes 1 request at the time, as fast as possible (within ~tickMs
intervals)
I could add a concurrency counter to execute X requests at the time, but it
does not address my concerns just works them around.
I know is barbaric, that's why I am here.
https://pastebin.com/jZZZhSyf
I did not use events, setTimeout was faster to use ;)
My point is, how do you manage efficiently how often execute asynchronous
operations? how would you solve my problem?
Any comment about the code is welcome, I'd love feedback from experienced
developers,
Cheers,
--
Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/8c6342f8-be9a-4366-b9ce-4526e852d9ec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.