Next, you need to make a decision how you would like the crawler to manage various World wide web systems. There's an ongoing debate regarding the intelligence of search engine crawlers. It is not entirely obvious If they're full-blown headless browsers or just glorified curl scripts (or a thing https://youtu.be/EVeTP0aV8gY?si=5FHbTy4kbLwUVoJD