The right AWS generative AI partner is the one that can show you a running production agent resembling your workload, with named evaluation gates, per-tool credential scoping, and traces you can read after a bad run. Everything else on a partner shortlist measures the ability to start work, not the ability to keep a system running under real load. Competency badges, logo walls, and a polished pilot demo are floor checks. They are not differentiators.
I have sat on both sides of these evaluations, and the pattern repeats: the partner with the best deck wins the RFP, and six months later the agent is answering questions nobody asked while a human quietly does the actual work behind it. The failure almost never traces back to model choice. It traces back to the absence of an evaluation suite, a shared high-privilege credential wired into every tool, and nobody owning the system after the launch celebration.
This article gives you a weighted scorecard you can paste into an RFP or a sole-source justification. It does not rank named firms. It ranks evidence. Use it with three to five partners, because verifying evidence properly takes real calendar time per firm.
The Shortlist Question You Should Be Asking Instead
Stop asking "do you have generative AI experience on AWS." Every firm on your shortlist will say yes, and they will all be telling a version of the truth. Ask instead: show me an agent you built that is in production today, still changing, and shaped like my problem, then walk me through the last time it failed.
That question works because published agent research has given buyers a vocabulary that did not exist two years ago. Benchmark design has shifted from single-turn text scoring to multi-step, tool-using, environment-interacting tasks. tau-bench scores agents in simulated user-and-tool conversations against explicit domain policy rules [1]. GAIA covers real assistant questions that require reasoning plus tool use [2]. SWE-bench grades whether an agent actually resolved a real repository issue, decided by a deterministic check rather than by plausibility [3].
The implication for procurement is direct. You can now ask a partner how they measure task completion, what their graders check, and what their policy-violation rate is, and a serious partner will have answers. A partner who only has demo footage will change the subject to model benchmarks or industry trends.
This scorecard is written for CTOs, VP Engineering, and enterprise architects running an agent procurement on AWS. It assumes you have at least one workload where an agent will touch a system of record, which is where evaluation and permission design stop being theoretical.
If you are earlier than that and still deciding which workflow to put out to bid, start with our guide on how to choose an AWS agentic AI implementation partner, which covers writing the one-page workflow brief, comparing partner types, and the evidence request to send every finalist. This article picks up where that one ends, at the point where you have a shortlist and need to score it.
The Weighted Scorecard (Copy This Into Your RFP)
Score each criterion 0 to 3. A 3 requires a verifiable artifact you can inspect. A 2 requires a reference who confirms it on a call. A 1 is a claim in a deck. A 0 is silence or deflection. Multiply by weight, total the score, and keep the worksheet because you will want it at contract negotiation.
| Criterion | Weight | What Passes (3) | What Fails (0-1) | How You Verify |
|---|---|---|---|---|
| Production references | 25 | Live system, still shipping changes, reference names what broke | Pilot that ended, or a reference who only remembers launch day | Reference call with the engineer who owns it now |
| Evaluation and release gates | 25 | Seeded eval environment, deterministic graders, pass rate and policy-violation rate reported separately [1] | One blended "accuracy" number, or human spot checks only | Screen share of a CI run on a prompt change |
| Security and governance practice | 15 | Per-tool least-privilege credentials, action classification enforced in orchestration [5] | Shared high-privilege identity, guardrails written in prompt text | Architecture review of an existing agent's auth model |
| Delivery model fit | 15 | Named engineers, on-call path through month four, written continuity terms | Senior architects in the pitch, unnamed rotation in delivery | Ask who is on call the week after go-live |
| Post-launch outcome tracking | 10 | Baseline measured before build, measurement method you own after exit | Projected ROI percentage with no stated measurement method | Request the instrumentation plan, not the business case |
| Commercial model | 5 | Risk sits with the party that controls the variable | Fixed price on undefined agent scope, or open-ended time and materials | Model two failure scenarios against the contract |
| AWS validated competencies | 5 | Verified by you in the Partner Solutions Finder, plus named AWS contacts | Badge images in a deck you never checked | Search the partner directly, do not accept a slide |
Adjust the weights to your constraint. Regulated or public-sector workload: raise governance to 25 and drop commercial model to zero, since procurement rules already fix it. Aggressive timeline with no internal agent platform team: raise delivery model fit. Strong internal engineering with a decision bottleneck: raise evaluation and lower delivery.
Tie-break rule: when two partners score within a few points, the one whose evidence you could verify without their help wins. Independently verifiable evidence is a proxy for how the relationship will behave when something goes wrong.
Verifying AWS Credentials Yourself in the Partner Solutions Finder
Open the AWS Partner Solutions Finder and search the partner by name before your second meeting. You are checking three distinct things that partners routinely blur together in slides: tier status, validated competencies, and service validations or deliveries. They are not interchangeable.
Tier status (Select, Advanced, Premier) reflects accumulated certifications, customer evidence, and program investment. Competencies are audited validations in a named area, which means AWS reviewed delivery evidence against a technical bar for that specialty. Service validations and deliveries are narrower, tied to a specific AWS service practice. A partner with a generative AI or machine learning competency has passed a review relevant to your work. A partner whose only validation is in general migration has passed a review about something else.
What these designations establish is audited delivery practice in a defined area. What they do not establish is an outcome on your specific workload, with your data quality, your legacy integration surface, and your approval chain. Score competencies as a floor, not as proof of fit. A Strategic Collaboration Agreement or Advanced and Premier tier indicates joint investment and audited depth, which is meaningful signal about practice maturity, and still only a floor.
One cheap test: ask for the AWS solutions architect and partner development manager who support the account. A partner with genuine AWS engagement names them without hesitation and will often offer to put them on a call. A partner reselling a logo will not.
Reference Calls: The Questions That Separate a Pilot From a Running System
Insist on a reference whose system is still in production and still changing. A reference who fondly remembers a successful launch two years ago is describing a delivery event, not an operating practice. Ask to speak to the engineer who owns the agent now, not the executive who signed the SOW.
Work this checklist in order:
- Who owns the agent today, and did ownership transfer cleanly or by attrition?
- What broke first after go-live, and how was it detected: a trace, a monitor, or a customer complaint?
- How many prompt, tool-schema, or model changes have shipped since launch, and what gate did each pass?
- What does the evaluation suite contain, and do sampled production traces get promoted into the regression set?
- What is the on-call path at 2am, and who answers it in month four versus week one?
- What got turned off or restricted after launch, and why?
The regression-set question is the single clearest signal of operating discipline. Promoting sampled production failures into a permanent test set is what separates a team running an evaluation program from a team that ran a one-time accuracy check [1]. If the reference does not know whether this happens, the partner probably built a demo and left.
- Eval environment exists
- A seeded environment with mocked tools and deterministic graders means runs are comparable across prompt and model versions
- [1]
- Two numbers, not one
- Task completion rate and out-of-policy action rate reported separately keeps a safety regression from hiding behind a quality gain
- [1]
- Tool schemas versioned
- A protocol-standard tool interface with versioned schemas lets you trace a behavior change back to an interface change
- [5]
- Traces correlated by run ID
- GenAI semantic conventions on model and tool spans let an operator reconstruct a bad multi-step run after the fact
- [4]
The red flag to listen for is a reference who cannot describe a single failure. Every production agent fails. A reference with no failure story was either never close to the system or is managing the narrative, and both answers should cost the partner points.
Delivery Models: Forward-Deployed Engineers, Advisory, or Staff Augmentation
Three delivery models dominate agent work, and the mismatch between model and constraint causes more failed engagements than technical incompetence does.
Forward-deployed engineering puts partner engineers inside your environment building and operating the system with your team. You get a running system. You still own product decisions, data access approvals, and the organizational change around the workflow. It fails when you have no internal technical counterpart, because the engineers end up guessing at requirements. Best for teams that need a production agent and have no agent platform group yet.
Advisory gives you architecture, evaluation design, governance structure, and decision support. You still own all the building. It fails when you hire advisory to compensate for missing engineering capacity, which produces excellent documents and no shipped system. Best for strong engineering teams stuck on a decision: which orchestration pattern, how to design the release gate, how to scope tool permissions.
Staff augmentation fills named roles on your team. You still own the platform, the standards, and the review process. It fails when there is no platform to plug into, because individual contractors cannot establish engineering standards from inside a staffing contract. Best for a working platform with a capacity gap.
Partner types map imperfectly onto these. Global systems integrators bring breadth, procurement familiarity, multi-country coverage, and the ability to absorb a program larger than one team. AWS specialist consultancies bring depth on Bedrock, agent orchestration, and account-level AWS relationships, with a narrower delivery surface. Product-led partners bring a running codebase and faster time to a first working system, constrained to the shape their product supports. None of these is better. They answer different constraints.
Evaluation, Release Gates, and Governance Evidence
Make every finalist describe their release gate for prompt, tool-schema, and model changes. This is the highest-signal technical question in the entire evaluation, and the answers separate partners more sharply than any architecture discussion.
A passing answer includes four things. Seeded evaluation environments with mocked or sandboxed tools so runs are reproducible. Deterministic graders that score final outcome state, not just whether the trajectory looked reasonable, which is the design pattern behind published agent benchmarks [1][2][3]. Execution in CI on every prompt, tool-schema, or model change. And pass rate plus policy-violation rate reported as two separate numbers, because a blended score lets a safety regression hide behind a quality improvement [1].
On instrumentation, require OpenTelemetry generative-AI semantic conventions for model and tool spans, correlated by a run identifier, with inputs and outputs redacted per your data-handling rules [4]. This is what makes a multi-step failure debuggable after the fact instead of reconstructed from scattered application logs. It also means the same traces feed offline scoring and regression tracking rather than requiring a second instrumentation effort.
On authorization, the standard is per-tool least-privilege credentials, read-only and state-changing tools separately authorized, and every action classified as auto-execute, requires-confirmation, or prohibited, with that classification enforced in the orchestration layer rather than in prompt text [5]. A partner that treats prompt instructions as a permission boundary has not operated an agent against a system of record. Our write-up on capping agent blast radius with permission tiers covers the enforcement pattern in detail, and agent identity and least privilege covers the credential side: one identity per agent, short-lived session-scoped credentials, and authorization checked at the tool boundary on every call.
For regulated and public-sector workloads, ask for one internal control set rather than four compliance binders. The practical structure is a single AI use-case register as system of record, with controls organized on the NIST AI Risk Management Framework functions [6], mapped to the generative-AI profile actions in NIST AI 600-1 [7], to ISO/IEC 42001 management-system requirements [8], and to EU AI Act obligations for your role as provider or deployer [9]. Deployment gates on a fixed artifact bundle: risk assessment, evaluation report with known failure modes, human-oversight design, data provenance notes, and rollback plan. Ask for GovCloud and FedRAMP-aligned delivery experience where relevant, and get release gate thresholds and override approvers in writing. We walk through the mapping mechanics in one control set with two mappings.
Commercial Models and Post-Launch Outcome Tracking
Commercial structure matters less than most RFPs assume, but the wrong structure quietly distorts behavior. Time and materials puts schedule risk on you and rewards duration. Fixed-scope outcome milestones push risk to the partner and reward narrow scope definitions that exclude the integration work that actually consumes the timeline. Managed service retainers align on uptime and incident response but can dull the incentive to reduce the agent's operating cost. Build-operate-transfer front-loads partner control and depends entirely on whether the transfer terms are specific.
Pick the structure where risk sits with the party controlling the variable. If data quality is your unknown, do not accept fixed price on the data work. If orchestration complexity is the unknown, do not accept time and materials on it.
The post-launch question most RFPs omit entirely: what business metric does the partner commit to instrument, and does the instrumentation survive the engagement? Insist that outcome tracking be a measured baseline plus a measurement method you own. A projected savings percentage in a proposal is a sales artifact. A baseline you can re-measure next quarter is an asset. If the partner cannot help you measure the current state before building, their ROI claim is arithmetic performed on an assumption.
Negotiate these transfer terms explicitly rather than discovering them at exit:
- 1Source code and infrastructure-as-code ownership, including the agent orchestration layer and tool definitions
- 2Evaluation suite ownership, including the seeded environments, graders, and promoted production test cases
- 3Trace retention and access, with the retention window and export format named
- 4Runbook handover, with a dry-run incident exercise before the exit window closes
- 5A named exit window with specific handover deliverables, not a termination clause
Score any of these as a red flag: no evaluation artifacts of any kind, agents authorized with a shared high-privilege identity, governance delivered as a document rather than an enforced gate, outcome claims with no stated measurement method, or refusal to provide a production reference.
How Tactical Edge Scores Against the Same Scorecard
Apply the scorecard to us. Tactical Edge is an AWS Advanced Tier Services Partner and holds a Strategic Collaboration Agreement with AWS, which you can verify in the Partner Solutions Finder rather than taking from this page; our AWS partnership detail lists the validations. We deliver through forward-deployed engineering on agentic AI systems, cloud and application modernization, and enterprise data platforms, described in our agentic AI approach, cloud modernization practice, and platform overview.
The practices we apply, stated as practices rather than as promised outcomes: an evaluation harness that reports pass rate and policy-violation rate as separate numbers, standardized tool interfaces with versioned schemas, per-tool credential scoping with read-only and state-changing tools separately authorized, OpenTelemetry-instrumented agent traces correlated by run identifier, and a use-case register mapped to one internal control set. Our agent release gate write-up shows the harness design we actually use.
Frequently Asked Questions
Does an AWS competency guarantee production capability?
No. A competency establishes that AWS audited delivery evidence against a technical bar in a named area. It does not establish fit for your workload, data quality, or approval chain. Treat it as a floor check worth about 5 percent of your scoring weight.
How many references should I ask for?
Two per finalist, and at least one must be a system still in production and still receiving changes. Ask to speak with the engineer who owns it today, not the executive sponsor.
Should I pick a global integrator or a specialist?
Match the partner to your constraint. Integrators fit multi-country programs, complex procurement, and scope larger than one team. Specialists fit depth on Bedrock and agent orchestration with account-level AWS access. Product-led partners fit when speed to a first working system matters more than breadth of shape.
What is a fair pilot budget structure?
Fund a pilot that produces three transferable artifacts regardless of outcome: an evaluation suite, a measured baseline for the target metric, and a working tool interface with scoped credentials. Pay for artifacts you keep, not for a demo you cannot run yourself.
How do I evaluate a partner for GovCloud or regulated workloads?
Ask for one control set mapped across the NIST AI RMF functions [6], the generative-AI profile actions [7], ISO/IEC 42001 requirements [8], and EU AI Act obligations by role [9], plus named release gate thresholds and override approvers. Also ask how they reviewed the target architecture against the AWS Well-Architected pillars for recovery objectives, encryption, and access control [10].
What should the contract say about evaluation suite ownership?
That you own the seeded environments, the graders, the test cases including those promoted from production traces, and the CI configuration that runs them, with no license dependency on the partner after exit.
Your Next Working Session
Four actions, in order. Open the AWS Partner Solutions Finder and verify every claimed competency and tier on your shortlist yourself. Send the reference-call checklist from this article to each finalist and require a production reference, not a launch reference. Require one verifiable artifact per scorecard criterion before the next meeting, and let silence score as zero. Then score and total.
Track one number through the process: the share of scorecard criteria backed by an artifact you inspected versus a claim in a deck. If that share is under half for your leading candidate, you are not evaluating partners, you are comparing sales materials.
Go back to the question at the top. The partner worth choosing can show you a running agent, name the gate it passes through, and tell you exactly how it failed last month. That conversation takes twenty minutes and tells you more than any proposal.
References
[1]Yao et al., "tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains," 2024. https://arxiv.org/abs/2406.12045
[2]Mialon et al., "GAIA: A Benchmark for General AI Assistants," 2023. https://arxiv.org/abs/2311.12983
[3]Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?," 2023. https://arxiv.org/abs/2310.06770
[4]OpenTelemetry, "Semantic Conventions for Generative AI," 2025. https://opentelemetry.io/docs/specs/semconv/gen-ai/
[5]Model Context Protocol, "Specification," 2025. https://modelcontextprotocol.io/specification
[6]NIST, "AI Risk Management Framework," 2025. https://www.nist.gov/itl/ai-risk-management-framework
[7]NIST, "AI 600-1: Generative AI Profile," 2024. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
[8]ISO, "ISO/IEC 42001: Artificial intelligence management system," 2023. https://www.iso.org/standard/81230.html
[9]European Union, "Regulation (EU) 2024/1689 (Artificial Intelligence Act)," 2024. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
[10]AWS, "AWS Well-Architected Framework," 2025. https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html