Scheduled Content Automation for External Websites

AI Workflows can be configured to run on schedules that align with your import schedules. For example, set a weekly import for Monday at 8 AM, then schedule a workflow for Monday at 9 AM to process the imported content. The workflow can compare versions, extract changes, summarize updates, identify risks, and send notifications—all automatically without manual intervention.

Content imported via Web Import is automatically indexed and made available to Public AI Chat widgets configured to use that site. When you schedule daily imports from your external documentation site, your public chat widget automatically stays synchronized—visitors always get answers from your latest documentation without any manual updates or synchronization steps.

Web Import works with any publicly accessible HTTP or HTTPS website—no authentication required. It captures HTML pages and linked PDF documents. Common use cases include competitor websites, regulatory agency pages, industry news sites, documentation portals, and public knowledge bases. The content must be publicly viewable without login credentials.

Yes, you can create multiple import schedules within a single Clear Ideas site, each targeting different source websites. Specify different destination folders for each source to keep content organized. This is ideal for competitive intelligence (monitoring multiple competitors), regulatory tracking (multiple government agencies), or news aggregation (multiple industry publications).

Yes, you have granular control over import scope. Use URL masking to restrict imports to specific sections (e.g., only pages under "/docs/"), set link depth to control how many levels deep the crawler follows links (1-3 levels), specify destination folders for organized storage, and choose whether to include linked PDF documents. These controls help you capture exactly the content you need without excess.

Absolutely. Every site includes an immediate Web Import option for testing. Navigate to your site, click Web Import, enter the URL, configure your settings (URL masking, depth, destination folder), and run it immediately. Review the results to verify settings work as expected, then create a schedule using those same settings for automated recurring imports.

Web Import respects robots.txt files and standard web crawling protocols. If a website explicitly blocks automated access, the import will fail with a clear error message. For websites that require authentication or have anti-bot protection, Web Import cannot access that content. In these cases, consider manual uploads or using the Clear Ideas Sync desktop application to transfer content.

Yes, through AI Workflows. Configure a workflow that runs after each import, analyzes changes, and sends notifications based on your criteria. For example, create a workflow that compares versions, identifies new pricing information on competitor sites, and emails your sales team. Or detect regulatory language changes and alert your compliance team. The workflow automation makes intelligent monitoring hands-free.

Imported files are organized in your Clear Ideas site following the source website structure. You specify a destination folder, and Web Import creates subfolders mirroring the URL path structure. Each URL maps to one file location, with versions tracked automatically. Files are searchable via full-text and AI-Enhanced Search, and subject to the same role-based permissions as all other content in your site.

Scheduled imports run automatically in the background at your chosen frequency: Hourly (every hour at a specified minute), Daily (once per day at a specified time), Weekly (on specific days of the week at specified times), or Monthly (on specific dates or expressions like "first Monday"). All schedules are timezone-aware and can be activated, deactivated, or modified at any time.

Storage consumption depends on the volume of content imported and change frequency. Because versions are only created when content actually changes, storage is used efficiently. Most organizations find that monitoring 5-10 competitor websites with weekly imports consumes minimal storage. Enterprise plans include higher storage limits and the ability to configure retention policies for older versions if needed.

Clear Ideas maintains a one-to-one mapping between source URLs and imported files. When content changes, a new version is automatically created while preserving all previous versions. You can view any historical version, compare versions side-by-side to see exactly what changed, and track the complete evolution of external content over time. Versions are only created when actual content changes are detected, avoiding unnecessary duplication.