Migrating a transactional app from AWS to OCI: designing the isolated VCN (CIDR, NSGs, and São Paulo's quirks)

Phase 0: the network comes before the workload

In an AWS→OCI migration of a transactional app, the temptation is to start with the cluster. Wrong: the VCN (Virtual Cloud Network) is the foundation. CIDR, subnet, or NSG mistakes only surface once the app is running — and then the cost to fix is high. We designed the whole mesh first, isolated, with OKE/ARM from the start.

Decision 1: the CIDR that mustn't overlap

We chose 10.1.0.0/20 for the VCN — deliberately outside 10.0.0.0/16. If there's ever peering with another network (AWS during coexistence, or a future VCN), overlapping CIDRs make peering impossible. Choosing the range with the future in mind is free now and very expensive later.

CIDR is a one-way decision: changing it later means recreating subnets and re-addressing everything. Think about future peering before you type the first /20.

Decision 2: São Paulo has a single Availability Domain

OCI's SA-SAOPAULO-1 region has only 1 AD. Unlike regions with 3 ADs, you don't spread resilience across ADs — you spread it across Fault Domains within the single AD. Anyone designing for multi-AD gets frustrated; the HA pattern here is fault-domain-aware, not AD-aware.

Decision 3: the Service Gateway answers on an unexpected prefix

For internal traffic to reach OCI services (Object Storage, etc.) without going to the internet, you use the Service Gateway. The gotcha: in São Paulo, the service CIDR label is all-gru-services-in-oracle-services-network — not SA-SAOPAULO. Pointing the route rule at the wrong label makes traffic simply not route, with no obvious error.

# list available service CIDR labels in the region (find 'gru')
oci network service list --query "data[].{name:name,cidr:\"cidr-block\"}" --output table

# the Service Gateway route rule uses the 'all-gru-services...' label, not 'SA-SAOPAULO'
oci network route-table update --rt-id ocid1.routetable.oc1..aaaaEXAMPLE \
  --route-rules '[{"destination":"all-gru-services-in-oracle-services-network",
                   "destinationType":"SERVICE_CIDR_BLOCK",
                   "networkEntityId":"ocid1.servicegateway.oc1..aaaaEXAMPLE"}]'

The 4-NSG matrix

Instead of broad Security Lists, we used Network Security Groups per tier, with least-privilege — the database only accepts connections from the app's NSG:

nsg-lb    : ingress 80,443  from 0.0.0.0/0           (public load balancer)
nsg-app   : ingress 3000, 30000-32767 (NodePort),    from nsg-lb
            10250 (kubelet)            from within the VCN
nsg-wp    : ingress 2028 (custom SSH), 9100 (Prometheus) from ops IPs
nsg-db    : ingress 3306              ONLY from nsg-app/nsg-wp  (no 0.0.0.0/0)

NSG referencing NSG (source = another NSG, not a CIDR) is what makes the mesh change-proof: scaling the app doesn't require reopening the database to a new range.

Bonus: the amd64 image that won't run on the ARM node

The OKE cluster is ARM (Ampere). The image that ran on ECS was amd64-only (null runtimePlatform in the task definition). On the ARM node, the pod sits in CrashLoopBackOff with an exec format error. The fix is a multi-arch rebuild — a recurring topic when moving from x86 to Ampere:

docker buildx build --platform linux/arm64 -t registry.example.com/myapp/app:arm64 --push .

Lessons

1. Design the VCN before any workload. CIDR, subnets, and NSGs are foundation; fixing them with the app live is expensive.

2. CIDR with future peering in mind. Avoid overlap with 10.0.0.0/16; a /20 outside the range today saves peering tomorrow.

3. São Paulo = 1 AD, resilience via Fault Domain. Don't design for multi-AD; spread across fault domains.

4. Service Gateway uses the 'all-gru-services' label. Pointing at SA-SAOPAULO makes traffic silently fail to route.

Conclusion

A cloud migration that starts with the container starts with the roof. The right network — non-overlapping CIDR, fault-domain resilience, the Service Gateway on the right label, and 4 least-privilege NSGs — is what makes the rest of the migration predictable instead of a sequence of regional surprises.