Migrating Off Informatica and DataStage to Event-Driven AWS Pipelines

The fastest path off Informatica or DataStage is not a rewrite. It is a classification exercise. Before you migrate a single mapping, sort every workload into four tiers by latency and complexity, model per-pipeline cost against AWS consumption pricing, then use the strangler-fig pattern to move workloads incrementally onto EventBridge, Step Functions, and Glue. Teams that classify first cut their migration timelines roughly in half and avoid the big-bang cutovers that stall in reconciliation hell.

I have run this migration three times for enterprises with 300 to 900 active pipelines. The pattern that works is always the same: understand what you have, price the target precisely, then peel workloads off the legacy platform in waves while both systems run in parallel. This article gives you the framework to do exactly that.

The trigger is almost always a renewal quote. That is where we start.

The Renewal Quote That Should Have Been a Wake-Up Call

A financial services client forwarded me a renewal quote last year. Seven figures for another three-year Informatica PowerCenter term, roughly 22 percent higher than the prior cycle. Their data volume had doubled in eighteen months, and three business units were demanding sub-minute latency for fraud scoring and inventory sync that the batch grid simply could not deliver.

The quote was not the problem. The problem was what the quote represented: a locked-in capacity model paying premium rates for compute that sat idle most of the day, plus mounting technical debt in 400-plus mappings that only two people fully understood.

Renewal timing creates a narrow window. You have use with procurement, budget attention from finance, and executive willingness to fund change. Miss it, and you sign another multi-year term that guarantees the same conversation later at a higher number. The goal is not to panic-migrate before the deadline. It is to arrive at that renewal meeting with a classified inventory, a defensible cost model, and a phased plan that does not require a risky cutover.

Here is the promise of the rest of this article: a repeatable method to classify workloads, build a cost model that survives finance scrutiny, and execute the migration incrementally.

Why Legacy ETL Costs More Than the License Line Item

The license is the number everyone stares at. It is rarely the largest number. When we do a full cost accounting on legacy ETL, the license typically accounts for 40 to 60 percent of total cost of ownership. The rest hides in places finance never itemizes.

Start with dedicated grid compute. Informatica PowerCenter and IBM DataStage run on fixed capacity you provision for peak batch windows. That hardware or reserved cloud instance sits at 15 to 25 percent utilization outside the nightly window, and you pay for all of it, all the time.

Then add the human cost. Dedicated DBAs tuning the repository database, ETL developers maintaining PowerCenter workflows and DataStage job sequences, and the change-review overhead that turns a simple field addition into a two-week ticket. Change lead time is the silent killer. When a new source column takes ten business days to flow through a mapping, the business routes around IT and builds shadow pipelines.

The architectural constraint is worse than the money. A fixed batch grid forces contention. Every new reporting job competes for the same nightly window, and real-time use cases simply cannot exist because the platform was designed around scheduled runs. Consumption-based AWS pricing inverts this. You pay for a Glue job only while it runs, a Lambda only per invocation, and a Step Functions workflow only per state transition. Idle capacity stops being a line item.

That shift is the entire business case. But you cannot make it credibly until you know exactly what you are running.

Step 1: Classify Every Workload Before You Touch It

Do not migrate by team, by schema, or by whoever complains loudest. Migrate by workload class. Every pipeline falls into one of four tiers, and the tier dictates the target service, the migration effort, and the risk profile.

The four tiers:

• Real-time streaming: Sub-second to sub-minute latency requirements. Fraud scoring, event enrichment, live inventory. These belong on Kinesis or EventBridge with Lambda consumers.
• Micro-batch: One-to-fifteen-minute cadence. Near-real-time dashboards, CDC replication. EventBridge Scheduler plus Glue or Lambda handles these well.
• Scheduled batch: Hourly to daily runs with hours of tolerance. Classic reporting extracts and warehouse loads. Glue jobs orchestrated by Step Functions are the natural fit.
• Complex transformation-heavy: Multi-stage joins, slowly changing dimensions, heavy data quality logic. Step Functions orchestrating multiple Glue jobs, sometimes with Glue DataBrew for the transformation-dense stages.

Classify on four axes: latency SLA, data volume per run, transformation complexity (count of joins, lookups, and aggregations), and downstream dependency count. A pipeline with a 24-hour SLA, 5GB volume, two joins, and one consumer is a low-risk scheduled batch. A pipeline with a 30-second SLA feeding six downstream systems is a high-effort streaming migration you tackle later.

One more classification outcome matters: retire and consolidate. In every migration I have run, 10 to 20 percent of pipelines are dead. They feed a report nobody opens, or they duplicate a job another team already built. Kill those before they cost you a single DPU-hour. The classification exercise pays for itself just by revealing what you should never migrate.

Step 2: Build the Cost Model That Survives Finance Scrutiny

Finance will not approve a migration on a promise of savings. They approve on a per-pipeline unit cost model tied to a payback window. Build it bottom-up.

For a Glue-based batch pipeline, the formula is straightforward. Glue charges per DPU-hour. A standard job with 10 DPUs running 12 minutes costs roughly 10 DPUs times 0.2 hours times the DPU-hour rate. Model each pipeline's DPU allocation and average runtime, then multiply by monthly run frequency. For Lambda-based micro-batch, price against GB-seconds: memory allocation times execution duration times invocations. For Step Functions orchestration, count state transitions per workflow and multiply by the per-transition rate, which for most pipelines is a rounding error against compute.

Here is a representative before/after for two real workloads from the financial services client.

The legacy costs here allocate a share of the fixed grid to each pipeline based on runtime and resource share. That allocation method is what makes the model defensible: it maps real infrastructure cost to real workloads instead of hiding it in a lump license fee.

Tie the model to renewal timing. If your total annual AWS run-rate for migrated workloads lands at $180K against a $470K annual legacy cost, your gross annual savings is $290K. Subtract migration labor, and you have a payback window measured in months, not years. That single slide closes the funding conversation.

The Reference Architecture: EventBridge, Step Functions, and Glue

The target pattern replaces monolithic mappings with composable, event-driven pipelines. Three services do most of the work. EventBridge is the router that reacts to events (a file landing in S3, a schedule firing, an upstream job completing). Step Functions is the orchestrator that sequences steps, handles retries, and manages error paths. Glue is the transformation engine that runs the actual Spark or Python work.

A file-arrival pipeline looks like this. An object lands in S3, which emits an event to EventBridge, which matches a rule and starts a Step Functions state machine that runs Glue jobs in sequence.

{
  "Name": "s3-arrival-triggers-pipeline",
  "EventPattern": {
    "source": ["aws.s3"],
    "detail-type": ["Object Created"],
    "detail": {
      "bucket": { "name": ["raw-ingest-prod"] },
      "object": { "key": [{ "prefix": "orders/incoming/" }] }
    }
  },
  "Targets": [
    {
      "Id": "start-order-pipeline",
      "Arn": "arn:aws:states:us-east-1:1234:stateMachine:order-etl",
      "RoleArn": "arn:aws:iam::1234:role/eventbridge-invoke-sfn"
    }
  ]
}

The Step Functions definition then orchestrates the stages that a single Informatica mapping used to bundle together, but now each is independently observable and retryable.

{
  "Comment": "Order ETL pipeline",
  "StartAt": "ValidateSchema",
  "States": {
    "ValidateSchema": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": { "JobName": "order-schema-validate" },
      "Next": "TransformAndLoad",
      "Retry": [{ "ErrorEquals": ["States.ALL"], "MaxAttempts": 2 }]
    },
    "TransformAndLoad": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": { "JobName": "order-transform-load" },
      "End": true
    }
  }
}

This structure maps cleanly onto legacy constructs. What used to be tangled inside a single tool now becomes explicit, versioned infrastructure.

The gain is decoupling. Change one Glue job without touching the orchestration. Add a downstream consumer by adding an EventBridge rule, not by editing a monolithic workflow. This is the same composable thinking behind our agentic AI architecture work, where small, independently deployable units beat monoliths every time.

Step 3: Execute With the Strangler-Fig Pattern, Not a Big Bang

Never flip a switch. The strangler-fig pattern runs the legacy pipeline and the new AWS pipeline in parallel, compares their outputs, and only decommissions the old one after the new one proves correct over multiple cycles. The name comes from the vine that gradually envelops a tree until the tree is gone, but the structure never collapsed during the transition.

The reconciliation check is the heart of it. For each parallel run, compare row counts, checksums on key columns, and business-metric totals between the legacy target and the new target. Only when they match across a defined number of consecutive runs (I use seven for daily jobs, three for streaming) do you cut over reads and stop the legacy pipeline.

def reconcile(legacy_df, new_df, key_cols, metric_cols):
    checks = {}
    checks["row_count_match"] = legacy_df.count() == new_df.count()
    for col in metric_cols:
        legacy_sum = legacy_df.agg({col: "sum"}).collect()[0][0]
        new_sum = new_df.agg({col: "sum"}).collect()[0][0]
        checks[f"{col}_sum_match"] = abs(legacy_sum - new_sum) < 0.01
    key_diff = legacy_df.select(key_cols).subtract(new_df.select(key_cols))
    checks["no_missing_keys"] = key_diff.count() == 0
    return all(checks.values()), checks

Sequence the migration by tier. Scheduled batch first because it is low risk and proves the toolchain. Then micro-batch, then complex transformations, then streaming last because it carries the highest blast radius.

This phased method is the core of how our cloud modernization practice runs Moonshot Migrations. The discipline is boring on purpose: parallel runs, reconciliation gates, tier-by-tier progression. Boring is what keeps a nine-figure data platform from going dark on a Tuesday.

Common Migration Traps and How to Avoid Them

Four traps kill these migrations. Name them before they find you.

The first is lift-and-shift of transformation logic. Teams try to translate a 200-step Informatica mapping into one giant Glue job line for line. That preserves every accumulated hack and misses the chance to simplify. Re-express the intent, not the implementation.

The second is ignoring lineage and governance. When one mapping becomes an event-driven mesh of Glue jobs and Lambdas, tracking where a data element came from gets harder, not easier. You need lineage capture from day one.

Distributed pipelines demand observability you never needed on a monolith. Instrument every Glue job with CloudWatch metrics, wire Step Functions execution history into your logging, and set alarms on reconciliation failures. Use the Glue Data Catalog and a lineage tool so an auditor can trace any field to its source.

The change-management cost is real. Your PowerCenter and DataStage experts are not obsolete, they are the people who know the business logic better than anyone. Pair them with cloud engineers during the pilot so knowledge transfers both directions. Budget for two to three months of overlap where the same people run both worlds.

Your Next 30 Days: A Concrete Starting Point

Here is the action for this week: pull your top 20 highest-cost pipelines by allocated compute and classify each into one of the four tiers. You do not need tooling. A spreadsheet with columns for latency SLA, volume, transformation complexity, dependency count, and target tier gets you 80 percent of the value in an afternoon.

Start tracking two metrics immediately: per-pipeline cost per run and change lead time. The first proves your business case. The second proves the operational win to the teams who have waited ten days for a field change.

FAQ

How long is the payback window? For mid-size estates, we typically see 6 to 12 months once migrated workloads are running on consumption pricing, driven mostly by eliminated idle compute and avoided license renewal.

Is running two systems in parallel risky? The opposite. Parallel operation with reconciliation gates is the risk mitigation. The legacy pipeline is your rollback plan until the new one passes multiple clean cycles.

What skills does the team need? PySpark for Glue, IAM and event-driven patterns for EventBridge and Step Functions, and comfort with infrastructure-as-code. Your existing ETL developers already understand the hard part, which is the business logic.

Do we have to migrate everything? No. Retire dead pipelines, consolidate duplicates, and migrate the rest by tier. Expect 10 to 20 percent of your inventory to never make the trip.

Go back to that renewal quote. The seven-figure number is not a bill you have to pay. It is a deadline that forces a decision. Classify your workloads this week, model the cost, and you will walk into that renewal meeting with the one thing that changes the conversation: a plan.