Traffic Mirroring and Service Virtualization

Deploy Preference v2

We’ll use version 2 of preferences service. The only difference between v1 and V2 is that v2 prints its name in capital letters (PREFERENCE).

kubectl apply -f manifests/kubernetes/preference-v2.yml

You should have 2 pods for preference based on the steps above

kubectl get pods

NAME                                          READY     STATUS    RESTARTS   AGE
customer-755bbbdd44-t2vjq                     2/2       Running   1          6d
preference-v1-7c8b57fc49-kcsvc                2/2       Running   0          6d
preference-v2-f74bcc999-lwlbf                 2/2       Running   0          36s
recommendation-v1-798bf87d96-stzqt            2/2       Running   7          6d

Send several requests to the customer endpoint

while true; do curl $customer; sleep .5; done

In a new terminal, check the logs of recommendation-v2

kubectl logs -f `kubectl get pods|grep preference-v2|awk '{ print $1 }'` -c preference

The default Kubernetes behavior is to round-robin load-balance across all available pods behind a single Service.

So after running several times you’ll get some combinations of:

customer => preference => recommendation v1 from '798bf87d96-stzqt': 176
customer => PREFERENCE => recommendation v1 from '798bf87d96-stzqt': 177
customer => preference => recommendation v1 from '798bf87d96-stzqt': 178
customer => PREFERENCE => recommendation v1 from '798bf87d96-stzqt': 179

and see what happens when you run

kubectl apply -f manifests/istio/traffic-mirroring/preference-destination-rule-v1-v2.yml
kubectl apply -f manifests/istio/traffic-mirroring/preference-v1-mirror-v2-virtual-service.yml
curl $customer

The customer has stopped calling preference:v2

customer => preference => recommendation v1 from '798bf87d96-stzqt': 196
customer => preference => recommendation v1 from '798bf87d96-stzqt': 198
customer => preference => recommendation v1 from '798bf87d96-stzqt': 200
...

But preference:v2 still receiving requests:

preference v2 request from pod: f74bcc999-lwlbf
preference v2 request from pod: f74bcc999-lwlbf
preference v2 request from pod: f74bcc999-lwlbf

Clean up

kubectl delete -f manifests/istio/traffic-mirroring/preference-destination-rule-v1-v2.yml
kubectl delete -f manifests/istio/traffic-mirroring/preference-v1-mirror-v2-virtual-service.yml

Virtualizing Mirrored Service Dependencies

To avoid side-effects from the version 2 of preferences service, instead of communicating with recommendation service, we are going to make the preference:v2 communicate with a virtualized recommendation service.

Service virtualization can be understood as something similar as mocking where instead of mocking components (classes), you are mocking remote services.

For this concrete case, virtualized recommendation service will return recommendation v2 from 'virtualized': 2.

kubectl apply -f manifests/kubernetes/recommendation-virtualized.yml

Mirroring Traffic to Preference

After this step, you should have three versions of recommendation service (v1, v2 and virtualized).

NAME                                          READY     STATUS    RESTARTS   AGE
customer-755bbbdd44-t2vjq                     2/2       Running   1          6d
preference-v1-7c8b57fc49-kcsvc                2/2       Running   0          3h
preference-v2-f74bcc999-lwlbf                 2/2       Running   4          35m
recommendation-v1-798bf87d96-stzqt            2/2       Running   7          3h
recommendation-virtualized-5f9678d7d8-6lhx4   2/2       Running   0          30s

Send several requests to see their responses

while true; do curl $customer; sleep .5; done

The default Kubernetes behavior is to round-robin load-balance across all available pods behind a single Service.

So after running several times you’ll get some combinations of:

customer => PREFERENCE => recommendation v1 from '798bf87d96-stzqt': 233
customer => preference => recommendation v2 from 'virtualized': 2
customer => PREFERENCE => recommendation v2 from 'virtualized': 2
customer => preference => recommendation v1 from '798bf87d96-stzqt': 234
customer => PREFERENCE => recommendation v1 from '798bf87d96-stzqt': 235
customer => preference => recommendation v2 from 'virtualized': 2
customer => PREFERENCE => recommendation v2 from 'virtualized': 2

Notice that now the v2 reaches all recommendation services. Let’s avoid this by just sending traffic that comes from preference v2 service to the virtualized recommendation service.

kubectl apply -f manifests/istio/routing-simple/recommendation-destination-rule-v1.yml
kubectl apply -f manifests/istio/service-virtualization/recommendation-virtualized-virtual-service.yml

Then do again some requests and you’ll get something like:

customer => preference => recommendation v1 from '2409176097-kcjsr': 5
customer => PREFERENCE => recommendation v1 from 'virtualized': 2
customer => preference => recommendation v2 from '1275713543-2bs5k': 6
customer => PREFERENCE => recommendation v2 from 'virtualized': 2

We’re almost done. We will again make preference:v2 a mirror for v1.

kubectl apply -f manifests/istio/traffic-mirroring/preference-v1-mirror-v2-virtual-service.yml

Read the output of the curl command and the log of preference:v2, what do you observe ?

Now all requests that are from preference v2 are redirected to virtualized recommendation service. In this way when you deploy a new service, you can mirror the traffic without worrying about side-effects on other services, since the requests are redirected to a virtualized instance instead of a production one.

Next >