Activity
Mon
Wed
Fri
Sun
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
What is this?
Less
More
Memberships
ChatGPT Users
12.6k members • Free
AI Developer Accelerator
10.8k members • Free
2 contributions to AI Developer Accelerator
Feb 3 •
[New Video] Scrape Any Website for FREE Using DeepSeek and Crawl4AI
Hey everyone 👋 Web scraping is one of the most in-demand AI skills right now, and DeepSeek is taking over the AI world. In my latest tutorial, I’ll show you how to scrape any website for free using DeepSeek, Crawl4AI, and Groq. We will walk through the entire process step by step. You’ll learn how to tweak a prebuilt template to scrape any site effortlessly, extract leads and save them to an organized file, and automate the entire process using these cutting-edge AI tools. 🎁 Free Source Code: Because you guys are awesome, I’m giving away the full source code for free. You can download it here: https://github.com/bhancockio/deepseek-ai-web-crawler DeepSeek is making AI smarter, faster, and cheaper. I can’t wait to see what you all build with it. Let me know your thoughts in the comments. Cheers, Brandon Hancock 🧑💻
0 likes • Feb 3
@Brandon Hancock I tried the LLM Extraction Strategy, but it's not foolproof. This is where Crewa can come in. We can programmatically identify a repeating container in the HTML DOM structure, like a product tile that repeats across the page. This container can then be sent to an agent to generate the appropriate selector to extract the required fields. Next, we use a separate validator agent to check if the generated selector is correct. If it’s not, the validator will provide feedback to the first agent, and the process will repeat, with a maximum of three retries. Once the validator confirms the selector is correct, we can use it to extract the data programmatically.
1 like • Feb 3
@Brandon Hancock That would be helpful. We should set up a validator agent to validate the selectors before passing them to the next crew for data extraction. This step will be the final key element we need.
Feb 3 •
CrewAI structuring.
Am I approaching this correctly? I feel like I'm wrong, but if I'm not, it seems like it lacks intelligence since it involves simple if-else logic. I was expecting Crewai to automatically perform entity extraction from the user's question and, based on the tool's input parameters, prompt the user for any missing information. Even regular function calls can achieve that. So, what exactly does Crewai solve in this scenario? question = self.inputs.get("question", "Show me product information for Bosch A1242G") # Classify the query using the LLM query_type = self.classify_query(question) print(f"Classified query type: {query_type}") # Dynamically select the task sequence based on the classified query type if query_type == "info": tasks_to_run = [ self.product_info_task(), self.replacement_check_task() ] elif query_type == "match": tasks_to_run = [ self.product_matching_task(), self.replacement_on_match_task(), self.replacement_info_task() ] elif query_type == "compare": tasks_to_run = [ self.multi_product_info_task(), self.comparison_task() ] elif query_type == "replacement": tasks_to_run = [ self.get_replacement_task() ] else: tasks_to_run = [ self.product_info_task(), self.replacement_check_task() ] return Crew( agents=self.agents, tasks=tasks_to_run, process=Process.sequential, verbose=True, memory=True )
1-2 of 2
