Website Crawler Problem · AI Automation Society

Website Crawler Problem

hi friends, i have question and i appreciate anyone who can help me

i want to create a website crawler which insert all data and news related to one website into my supabase vector store. do you know any free or open-source crawler that i can give it my website URL, then with API of that crawler, in n8n i can call the crawler and add the data of site in my supabase vector store.

also i want to set this crawler to go and get all news and data or update of my website every minutes.

please give me easiest and fastest way which i do not need to coding

thanks for your watching 🙏

1 comment