This is more of a question than a thread.
This is the scenario. I am working with nodes in node-RED. Basically I am requesting a list of files from One Drive and then sending this list (about 1000 files) through to a split node. A download of said files is then requested.
This request is made with Fetch and via the M365 javascript SDK.
The SDK has a get stream method that firstly gets the download URL and then requests the file as a binary stream.
What happens ocassionally with this request is that I get a fetch failed error for multiple downloads. Sometimes fetch fails when a relatively small number of downloads are requested for example 500.
I have monitored node-RED with Prometheus and have found that there is no noticable increase in event loop lag or excessive CPU usage.
Part of my question is can a node in node-RED spawn a worker thread for the download operation, in theory reducing main thread work https://nodejs.org/api/worker_threads.html
As far as I understand fetch failed means no response received. A failed request. So another question is, does the problem relate to the number of fetch requests being made.
Is the node being swamped with that many requests that it is just getting some kind of throw back from Fetch. If this is the case what is the best diagnostic tool to get some qualitative data?
Part of me thinks that this is a network error. However I am new to software such as shark wire so do not know how to identify possible issues.
Top comments (0)