I am using flowise with railway. I use pinecode to store data.

I want to upload text files and my website pages. I am using the Text Node and Puppeter web crawler Node.

I faced a first issue :

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

So after investigation, I changed the chank size to 1000 and the bug was fixed for a text file. Then I added puppeteer web scrapper. I used the sitemap method, I get the urls in puppeteer web scraper. and the issue came back. Again a problem of memory limit.

So I tried with another web scraper node and the same issue.

I have 30 text files that represent 30 books. It is about 50 Mo in total.

I have some screenshots to see my configuration if necessary. I don't know what I am doing wrong. I am a newbie in Flowise and Pinecone.

here is railway logs :

2024-07-08 11:52:17 [INFO]: ⬆ POST /api/v1/vector/internal-upsert/ceee0b68-64a4-4f7b-b684-445d5ffb3c19

Jul 08 13:54:11

Jul 08 13:54:11

<--- Last few GCs --->

Jul 08 13:54:11

Jul 08 13:54:11

[36:0x7fa3236226c0] 483060 ms: Mark-Compact 245.4 (258.7) -> 244.7 (258.7) MB, 210.03 / 0.00 ms (average mu = 0.597, current mu = 0.017) allocation failure; scavenge might not succeed

Jul 08 13:54:11

[36:0x7fa3236226c0] 483211 ms: Mark-Compact 245.7 (258.7) -> 245.0 (259.2) MB, 147.53 / 0.00 ms (average mu = 0.435, current mu = 0.025) allocation failure; scavenge might not succeed

Jul 08 13:54:11

Jul 08 13:54:11

Jul 08 13:54:11

<--- JS stacktrace --->

Jul 08 13:54:11

Jul 08 13:54:11

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Jul 08 13:54:11

----- Native stack trace -----

Jul 08 13:54:11

Jul 08 13:54:31

container event container died

Can you help me to find a solution, please?