Scheduled Content Automation for External Websites

Web Import works with any publicly accessible HTTP or HTTPS website—no authentication required. It captures HTML pages and linked PDF documents. Common use cases include competitor websites, regulatory agency pages, industry news sites, documentation portals, and public knowledge bases. The content must be publicly viewable without login credentials.

Scheduled imports run automatically in the background at your chosen frequency: Hourly (every hour at a specified minute), Daily (once per day at a specified time), Weekly (on specific days of the week at specified times), or Monthly (on specific dates or expressions like "first Monday"). All schedules are timezone-aware and can be activated, deactivated, or modified at any time.

Clear Ideas maintains a one-to-one mapping between source URLs and imported files. When content changes, a new version is automatically created while preserving all previous versions. You can view any historical version, compare versions side-by-side to see exactly what changed, and track the complete evolution of external content over time. Versions are only created when actual content changes are detected, avoiding unnecessary duplication.

Yes, you have granular control over import scope. Use URL masking to restrict imports to specific sections (e.g., only pages under "/docs/"), set link depth to control how many levels deep the crawler follows links (1-3 levels), specify destination folders for organized storage, and choose whether to include linked PDF documents. These controls help you capture exactly the content you need without excess.

AI Workflows can be configured to run on schedules that align with your import schedules. For example, set a weekly import for Monday at 8 AM, then schedule a workflow for Monday at 9 AM to process the imported content. The workflow can compare versions, extract changes, summarize updates, identify risks, and send notifications—all automatically without manual intervention.

Absolutely. Every site includes an immediate Web Import option for testing. Navigate to your site, click Web Import, enter the URL, configure your settings (URL masking, depth, destination folder), and run it immediately. Review the results to verify settings work as expected, then create a schedule using those same settings for automated recurring imports.

Content imported via Web Import is automatically indexed and made available to Public AI Chat configured to use that site. When you schedule daily imports from your external documentation site, your public chat automatically stays synchronized—visitors always get answers from your latest documentation without any manual updates or synchronization steps.

Imported files are organized in your Clear Ideas site following the source website structure. You specify a destination folder, and Web Import creates subfolders mirroring the URL path structure. Each URL maps to one file location, with versions tracked automatically. Files are searchable via full-text and AI-Enhanced Search, and subject to the same role-based permissions as all other content in your site.

Yes, you can create multiple import schedules within a single Clear Ideas site, each targeting different source websites. Specify different destination folders for each source to keep content organized. This is ideal for competitive intelligence (monitoring multiple competitors), regulatory tracking (multiple government agencies), or news aggregation (multiple industry publications).

Storage consumption depends on the volume of content imported and change frequency. As versions are only created when content actually changes, storage is used efficiently.