I’ve been working recently on extending a client’s application to run across two Azure regions. That’s involved enabling geo-replication for our Azure Service Bus instance, which led us to discover an interesting gotcha that I’m documenting here for other people.
The service bus product team are aware of the issue and are investigating. I can’t comment on whether they are able to resolve the issue outside of the mitigation I describe here.
Scenario
Our application is already running in production. We are extending into a new region. Our service bus instance is already deployed into our primary region and is connected into a virtual network with a private endpoint for security.
We deployed new services into a second Azure region, and updated our service bus instance to add a replica in our second region. As per the documentation we then added a second private endpoint, connected to our virtual network in our second region.
We have a private DNS zone with a record. By default that record points at the IP of the private endpoint in our primary region.
Emergent connectivity issue
When performing failover testing we tried different scenarios:
- We run in our primary region with the DNS record using the IP of the private endpoint in our primary region. Result: Connectivity to service bus succeeds.
- We promote the service bus replica in our second region and update the DNS record to use the IP of the private endpoint in that region. Result: Connectivity to service bus succeeds.
- We promote the servce bus replica in our second region and leave the DNS record pointing at the IP of the private endpoint in our primary region. Result: Connectivity to service bus fails.
- We run in our primary region but the DNS record points at the IP of the private endpoint in the secondary region. Result: Connectivity to service bus succeeds.
This difference in behaviour led us to create a support ticket, and to liaise with the product team. There is a clear difference between the original private endpoint, created before we enabled geo-replication and added a replica, and the new one, created at the point of enabling replication.
Hypothesis: Private endpoint connection differs between regions
Our working theory was that somehow the private endpoints were connected to the service bus instance in different ways. The original endpoint was created before we enabled geo-replication; the secondary was created after. We wondered if the first was connected in such a way as to prevent us reaching the replica, whereas the second, because it was created after the replica was present, was somehow able to access the replicas in both regions.
Resolution: Recreate private endpoint in primary region
As an experiment, we deleted the original private endpoint and created a new one in our primary region. We wondered if this would establish connectivity in the same way as the secondary privte endpoint.
The connectivity behaviour now matched that of the private endpoint in our secondary region - both replicas were accessible through this private endpoint.
Recommendation: If adding geo-replication, recreate primary private endpoint
We don’t always know that we need geo-replication early in our application lifecycle. If you have deployed a service bus instance with private endpoints for virtual network connectivity and are now looking to enable geo-replication, plan to recreate the private endpoint in your primary region.
Obviously, this means creating the secondary endpoint, updating DNS to point at the secondary IP, and then removing and recreating the private endpoint. Be aware that your messaging traffic is going to cross into your secondary region during the operation - ensure your connectivity is working.
We believe that a new service bus instance, with geo-replication enabled from the point of initial creation and private endpoints added onace the service is in that state should work. We have not tested this scenario, however.
