Quick start
- Navigate to AI Agents from your sub-account dashboard.
- Click on the Knowledge Base tab.
- Create a new Knowledge Base or select an existing one to edit.
Follow the detailed steps below to complete the setup.
You just built a beautiful, high-converting website for a client. It has sleek dropdown menus, hidden accordions for FAQs, and dynamic lazy-loaded pricing tables. It looks incredible.
Then, you plug your AI chatbot into the site to handle customer support. A hot lead asks a basic pricing question, and the bot replies, “I do not have that information.” You lose your mind because the answer is literally right there on the page inside an FAQ accordion.
The problem? Basic bots only read the surface level. They do not know how to click tabs or open dropdown menus. They are completely blind to the dynamic content that makes modern websites work.
HighLevel just solved this massive bottleneck with the Enhanced Web Crawler. It acts like a real human, automatically clicking, scrolling, and expanding every hidden section to extract the data so your bot actually knows what it is talking about. Let’s upgrade your bot’s brain.
Part 1: Access the Brain (Knowledge Base)
You need to point the crawler in the right direction. Ready to streamline this? Here is exactly how to do it in 60 seconds:
- Navigate to AI Agents from your sub-account dashboard.
- Click on the Knowledge Base tab.
- Create a new Knowledge Base or select an existing one to edit.
- Click the + Add Source button.
- Select Web Crawler from the options to launch the extraction engine.
Part 2: Target the Right Data
You have to tell the crawler exactly how deep to dig into the website.
- Choose your Domain Type based on how much data you actually want the bot to learn.
- Select Exact URL if you only want the bot to crawl one specific webpage (like a dedicated sales page).
- Select All URLs with the Path if you want it to crawl a specific category (like everything under /marketing).
- Select All URLs in this Domain if you want the bot to scrape and learn the entire root domain.
- Enter your URL into the field.
- Click the Extract Data button and let the system run its parallel detection strategies.
Part 3: Finalize the Training
Once the crawler finishes pulling data from all the hidden tabs and accordions, you need to lock it in.
- Click the View All Pages option when the URL crawling is complete.
- Review the list of discovered pages. You can select all URLs at once or cherry-pick individual pages by clicking the checkboxes next to the specific URLs you want to add into your training data.
- Click the Train Bot button.
Stop letting your AI give generic “I don’t know” answers. Let the Enhanced Web Crawler scrape the hidden data and turn your bot into an absolute expert on your client’s business.
Frequently Asked Questions
How do I access the Brain (Knowledge Base)?
You need to point the crawler in the right direction. Ready to streamline this? Here is exactly how to do it in 60 seconds: Navigate to AI Agents from your sub-account dashboard. Click on the Knowledge Base tab. Create a new Knowledge Base or select an existing one to edit. Click the + Add Source button. Select Web Crawler from the options to launch the extraction engine.
What should I know about target the Right Data?
You have to tell the crawler exactly how deep to dig into the website. Choose your Domain Type based on how much data you actually want the bot to learn. Select Exact URL if you only want the bot to crawl one specific webpage (like a dedicated sales page). Select All URLs with the Path if you want it to crawl a specific category (like everything under /marketing). Select All URLs in this Domain if you want the bot to scrape and learn the entire root domain. Enter your URL into the field. Click the Extract Data button and let the system run its parallel detection strategies.
What should I know about finalize the Training?
Once the crawler finishes pulling data from all the hidden tabs and accordions, you need to lock it in. Click the View All Pages option when the URL crawling is complete. Review the list of discovered pages. You can select all URLs at once or cherry-pick individual pages by clicking the checkboxes next to the specific URLs you want to add into your training data. Click the Train Bot button. Stop letting your AI give generic “I don’t know” answers. Let the Enhanced Web Crawler scrape the hidden data and turn your bot into an absolute expert on your client’s business.
Start Your HighLevel Free Trial
Browse all GoHighLevel guides · AI & Chatbots.