How We Evaluate AI Tools
Tools and agents on Intelloro are evaluated through a structured, source-cited pipeline. No star ratings guessed from marketing pages. No paid placements inflating scores. Every field traces back to a URL, a review platform, or a reviewed data source.
Quick Answer
Intelloro evaluates AI tools across 70+ fields using a 3-layer pipeline: automated web scraping (up to 10 pages per tool), Google Search Grounding for gap-filling, and AI-verified cross-referencing (Approach 3, run by Claude and reviewed by an admin) against 15 external sources including G2, Capterra, Trustpilot, and compliance databases. Dimension scores combine algorithmic calculations and LLM assessment. All sources are logged and visible on each tool's verification record.
The 3-Layer Evaluation Pipeline
Layer 1 — Web Scraping
We scrape up to 10 pages per tool (homepage, pricing, features, security, integrations, API docs, customers, changelog). A Gemini LLM extracts 44 structured fields from the content. Fields require direct evidence — no guessing.
Layer 2 — Google Search Grounding
For any fields still null after scraping, we run a Google Search Grounding pass. Gemini searches the web and fills gaps using live search results. All grounded values are tagged with source attribution.
Layer 3 — AI-Verified Cross-Referencing (Approach 3)
Approach 3 is run by Claude (an AI), which executes 15 mandatory external source checks: G2, Capterra, TrustRadius, Product Hunt, Trustpilot, GetApp, Make.com, Zapier, compliance searches (SOC 2, GDPR, HIPAA), status page verification, and enterprise customer lookup. Every source is logged, and an admin reviews and approves each session before it is saved to our database with a full audit trail.
How Dimension Scores Work
Each tool is scored across 9 dimensions (12 for agents) on a 1–10 scale. We use two methods depending on what can be measured objectively:
- Integration Power — counted from verified integrations list (0=1, 1–3=4–6, 10–29=8, 30+=10)
- Customizability — additive: API (+3), open-source (+2), webhooks (+1), MCP (+1), SSO (+1)
- Compliance & Data Protection — additive: SOC 2 + GDPR + HIPAA + WCAG AA/AAA + PII auto-redaction + licensed-only training (caps at 9; 10 reserved for admin-verified compliance)
- Performance — direct enum map from response time (INSTANT=10, FAST=8, MODERATE=5, SLOW=2)
- Planning Quality (agents only) — clamp-shaper on Gemini estimate by planning_capability enum (NONE→cap 3, BASIC→4–7, ADVANCED→floor 7)
- Ease of Use — LLM assessment from onboarding, docs, UI complexity signals. Upgraded with G2/Capterra user ratings when available.
- Output Quality — AI-estimated from public product content. Upgraded with G2/Capterra user ratings when available.
- Value for Money — LLM assessment from pricing vs. features. Upgraded with Capterra user ratings when available.
- Support & Ecosystem — LLM assessment from docs, community size, support channels. Upgraded with G2 user ratings when available.
- Accuracy & Reliability — AI-estimated from public product content. Published benchmark scores (MMLU/SWE-bench/HELM), when the item is on a public leaderboard, are shown separately.
- Task Completion (agents only) — AI-estimated from public product content.
- Tool Use Correctness (agents only) — AI-estimated from public product content.
When Approach 3 verification finds G2 or Capterra per-dimension ratings (e.g. “Ease of Use 8.9 · Quality of Support 8.7”), these real user data points override the AI estimates for the matching dimensions.
15 Mandatory External Sources
Every Approach 3 verification runs all 15 sources. No skipping. Results are logged in the session record.
Ratings
- G2
- Capterra
- TrustRadius
- Product Hunt
- Trustpilot
- GetApp
Compliance
- SOC 2 search
- GDPR search
- HIPAA search
- Status page verification
Adoption
- Enterprise customer search
Integrations
- Make.com listing
- Zapier listing
- MCP compatibility (dev tools)
- Uptime SLA from status page
What We Don't Do
- No paid placements — sponsored tools are clearly labeled and never get inflated scores
- No inference for boolean fields — HIPAA, SOC 2, MCP require explicit statements, not marketing language
- No facts guessed from AI memory — compliance, customer names, funding, pricing, data residency, PII & data-deletion handling, security isolation, uptime, SSO, regional availability, copyright & licensing, and underlying-model attribution come only from the vendor's site, live web verification, or manual review; left blank when there is no evidence, never filled from a model's training knowledge
- No cross-tool contamination — competitor comparison articles are never used as evidence for the tool being evaluated
- No thin content — pages with fewer than 300 characters of relevant content are marked UNVERIFIABLE, not used as evidence
- No crowdsourced ratings without verification — user review counts and ratings come only from verified platforms (G2, Capterra, Trustpilot)
Verification Coverage Score
Every tool has a Verification Coverage Score (0–100%) and grade (A–F) computed from field coverage, source reliability, and verification status. Fields verified by Approach 3 are markedVERIFIEDand weighted higher in the score. Fields filled only by the Smart Generator are markedLLMand weighted lower. Empty fields that are confirmed correct (e.g., no integrations for a standalone tool) are markedVERIFIEDrather than penalized.
A
90–100%
B
70–89%
C
50–69%
D
30–49%
F
<30%
What Does “Largely Verified” Mean?
Every listing carries a verification-level badge derived from how many of its data points we have verified via Smart Generator extraction + Approach 3 cross-checking. The badge you see ("Fully Verified", "Largely Verified", etc.) reflects our coverage of this listing's data — it is not a rating of the tool itself.
Fully Verified
≥ 90% of data points verified — comprehensive coverage across all tiers.
Largely Verified
70–89% verified — strong coverage with minor gaps in optional fields.
Partially Verified
50–69% verified — core information present, several optional fields incomplete.
Limited Data
30–49% verified — basic information only, significant gaps remain.
Unverified
< 30% verified — minimal data available; pending Smart Generator + Approach 3 pass.
Important: "Unverified" describes the state of our data about a tool, not the tool itself. A tool with "Limited Data" may be excellent — we simply have not yet completed extraction + cross-checking. Vendors can request faster verification via the claim flow.
Industry pattern reference: LinkedIn Profile Strength (5 levels), Crunchbase Profile Strength, Wikipedia Content Assessment letter grades.
FAQ
How often are tool profiles updated?
Smart Generator profiles are generated once and updated when significant website changes are detected. Approach 3 verified profiles are re-verified on a rolling schedule. Sponsored tools are prioritized for frequent updates.
Can tool vendors update their own profiles?
Vendors can claim their listing and submit corrections. All vendor-submitted changes are reviewed by our team before going live — they do not auto-publish.
How are integrations verified?
Integrations are extracted from the tool's official integrations page, then cross-checked against Make.com and Zapier listings. Each integration is verified individually — bulk arrays are never auto-passed.
What does UNVERIFIABLE mean?
A field is UNVERIFIABLE when no source in the 15-source checklist could confirm or deny the value. The original DB value is preserved unchanged. UNVERIFIABLE is not the same as false — it means we couldn't confirm.