Question 1

What types of websites can I import with Web Import?

Accepted Answer

Web Import works with any publicly accessible HTTP or HTTPS website—no authentication required. It captures HTML pages and linked PDF documents. Common use cases include competitor websites, regulatory agency pages, industry news sites, documentation portals, and public knowledge bases. The content must be publicly viewable without login credentials.

Question 2

How do scheduled imports work and what frequencies are available?

Accepted Answer

Scheduled imports run automatically in the background at your chosen frequency: Hourly (every hour at a specified minute), Daily (once per day at a specified time), Weekly (on specific days of the week at specified times), or Monthly (on specific dates or expressions like "first Monday"). All schedules are timezone-aware and can be activated, deactivated, or modified at any time.

Question 3

How does version management track changes to imported content?

Accepted Answer

Clear Ideas maintains a one-to-one mapping between source URLs and imported files. When content changes, a new version is automatically created while preserving all previous versions. You can view any historical version, compare versions side-by-side to see exactly what changed, and track the complete evolution of external content over time. Versions are only created when actual content changes are detected, avoiding unnecessary duplication.

Question 4

Can I control which parts of a website get imported?

Accepted Answer

Yes, you have granular control over import scope. Use URL masking to restrict imports to specific sections (e.g., only pages under "/docs/"), set link depth to control how many levels deep the crawler follows links (1-3 levels), specify destination folders for organized storage, and choose whether to include linked PDF documents. These controls help you capture exactly the content you need without excess.

Question 5

How do I trigger AI Workflows automatically when content changes?

Accepted Answer

AI Workflows can be configured to run on schedules that align with your import schedules. For example, set a weekly import for Monday at 8 AM, then schedule a workflow for Monday at 9 AM to process the imported content. The workflow can compare versions, extract changes, summarize updates, identify risks, and send notifications—all automatically without manual intervention.

Question 6

Can I test Web Import before setting up a schedule?

Accepted Answer

Absolutely. Every site includes an immediate Web Import option for testing. Navigate to your site, click Web Import, enter the URL, configure your settings (URL masking, depth, destination folder), and run it immediately. Review the results to verify settings work as expected, then create a schedule using those same settings for automated recurring imports.

Question 7

How does Web Import integrate with Public AI Chat?

Accepted Answer

Content imported via Web Import is automatically indexed and made available to Public AI Chat widgets configured to use that site. When you schedule daily imports from your external documentation site, your public chat widget automatically stays synchronized—visitors always get answers from your latest documentation without any manual updates or synchronization steps.

Question 8

What happens if a website blocks automated crawling?

Accepted Answer

Web Import respects robots.txt files and standard web crawling protocols. If a website explicitly blocks automated access, the import will fail with a clear error message. For websites that require authentication or have anti-bot protection, Web Import cannot access that content. In these cases, consider manual uploads or using the Clear Ideas Sync desktop application to transfer content.

Question 9

How are imported files organized and stored?

Accepted Answer

Imported files are organized in your Clear Ideas site following the source website structure. You specify a destination folder, and Web Import creates subfolders mirroring the URL path structure. Each URL maps to one file location, with versions tracked automatically. Files are searchable via full-text and AI-Enhanced Search, and subject to the same role-based permissions as all other content in your site.

Question 10

Can I import content from multiple websites into one site?

Accepted Answer

Yes, you can create multiple import schedules within a single Clear Ideas site, each targeting different source websites. Specify different destination folders for each source to keep content organized. This is ideal for competitive intelligence (monitoring multiple competitors), regulatory tracking (multiple government agencies), or news aggregation (multiple industry publications).

Question 11

How much storage do scheduled imports consume?

Accepted Answer

Storage consumption depends on the volume of content imported and change frequency. Because versions are only created when content actually changes, storage is used efficiently. Most organizations find that monitoring 5-10 competitor websites with weekly imports consumes minimal storage. Enterprise plans include higher storage limits and the ability to configure retention policies for older versions if needed.

Question 12

Can I be notified when specific content changes are detected?

Accepted Answer

Yes, through AI Workflows. Configure a workflow that runs after each import, analyzes changes, and sends notifications based on your criteria. For example, create a workflow that compares versions, identifies new pricing information on competitor sites, and emails your sales team. Or detect regulatory language changes and alert your compliance team. The workflow automation makes intelligent monitoring hands-free.

Scheduled Content Automation for External Websites

Manual Content Monitoring is Inefficient

Automated Website Monitoring & Content Synchronization

Complete Web Content Automation

Real-World Automation Scenarios

Automate Analysis with AI Workflows

Start Automating in Minutes

Powerful When Combined