Skip to content

Microservices Canary Deployments

Step-01: Introduction

Usecase Description

  • User Management getNotificationAppInfo will call Notification service V1 and V2 versions.
  • We will distribute traffic between V1 and V2 versions of Notification service as per our choice based on Replicas
NS V1 Replicas NS V2 Replicas Traffic Distribution
4 0 100% traffic to NS V1 Version
3 1 25% traffic to NS V2 Version
2 2 50% traffic to NS V2 Version
1 3 75% traffic to NS V2 Version
0 4 100% traffic to NS V2 Version
  • In our demo, we are going to distribute 50% traffic to each version (V1 and V2).
  • NS V1 - 2 replicas and NS V2 - 2 replicas
  • We are going to depict one Microservice calling other Microservices with different versions in AWS X-Ray

List of Docker Images used in this section

Application Name Docker Image Name
User Management Microservice stacksimplify/kube-usermanagement-microservice:3.0.0-AWS-XRay-MySQLDB
Notifications Microservice V1 stacksimplify/kube-notifications-microservice:3.0.0-AWS-XRay
Notifications Microservice V2 stacksimplify/kube-notifications-microservice:4.0.0-AWS-XRay

Image

Step-02: Pre-requisite: AWS RDS Database, ALB Ingress Controller, External DNS & X-Ray Daemon

AWS RDS Database

  • We have created AWS RDS Database as part of section 06-EKS-Storage-with-RDS-Database
  • We even created a externalName service: 01-MySQL-externalName-Service.yml in our Kubernetes manifests to point to that RDS Database.

ALB Ingress Controller & External DNS

  • We are going to deploy a application which will also have a ALB Ingress Service and also will register its DNS name in Route53 using External DNS
  • Which means we should have both related pods running in our EKS cluster.
  • We have installed ALB Ingress Controller as part of section 08-01-ALB-Ingress-Install
  • We have installed External DNS as part of section 08-06-01-Deploy-ExternalDNS-on-EKS

XRay Daemon

  • We are going to view the application traces in AWS X-Ray.
  • We need XRay Daemon running as Daemonset for that.
    # Verify alb-ingress-controller pod running in namespace kube-system
    kubectl get pods -n kube-system
    
    # Verify external-dns & xray-daemon pod running in default namespace
    kubectl get pods
    

Kubernetes Manifests

#01-MySQL-externalName-Service.yml
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  type: ExternalName
  externalName: usermgmtdb.cxojydmxwly6.us-east-1.rds.amazonaws.com
#02-UserManagementMicroservice-Deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: usermgmt-microservice
  labels:
    app: usermgmt-restapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: usermgmt-restapp
  template:
    metadata:
      labels:
        app: usermgmt-restapp
    spec:
      initContainers:
        - name: init-db
          image: busybox:1.31
          command: ['sh', '-c', 'echo -e "Checking for the availability of MySQL Server deployment"; while ! nc -z mysql 3306; do sleep 1; printf "-"; done; echo -e "  >> MySQL DB Server has started";']
      containers:
        - name: usermgmt-restapp
          image: stacksimplify/kube-usermanagement-microservice:3.0.0-AWS-XRay-MySQLDB
          ports:
            - containerPort: 8095
          imagePullPolicy: Always            
          env:
            - name: DB_HOSTNAME
              value: "mysql"
            - name: DB_PORT
              value: "3306"
            - name: DB_NAME
              value: "usermgmt"
            - name: DB_USERNAME
              value: "dbadmin"
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-db-password
                  key: db-password
            - name: NOTIFICATION_SERVICE_HOST
              value: "notification-clusterip-service"
            - name: NOTIFICATION_SERVICE_PORT
              value: "8096"    
            - name: AWS_XRAY_TRACING_NAME 
              value: "User-Management-Microservice"                
            - name: AWS_XRAY_DAEMON_ADDRESS
              value: "xray-service.default:2000"    
            - name: AWS_XRAY_CONTEXT_MISSING 
              value: "LOG_ERROR"  # Log an error and continue, Ideally RUNTIME_ERROR – Throw a runtime exception which is default option if not configured                                            
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - nc -z localhost 8095
            initialDelaySeconds: 60
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /usermgmt/health-status
              port: 8095
            initialDelaySeconds: 60
            periodSeconds: 10          
---
# Kubernetes Secrets
apiVersion: v1
kind: Secret
metadata:
  name: mysql-db-password
#type: Opaque means that from kubernetes's point of view the contents of this Secret is unstructured, it can contain arbitrary key-value pairs. In contrast, there is the Secret storing ServiceAccount credentials, or the ones used as ImagePullSecret . These have a constrained contents.
type: Opaque
data:
  # Output of echo -n 'dbpassword11' | base64
  db-password: ZGJwYXNzd29yZDEx
#03-UserManagement-NodePort-Service.yml
apiVersion: v1
kind: Service
metadata:
  name: usermgmt-restapp-nodeport-service
  labels:
    app: usermgmt-restapp
  annotations:
  #Important Note:  Need to add health check path annotations in service level if we are planning to use multiple targets in a load balancer  
    alb.ingress.kubernetes.io/healthcheck-path: /usermgmt/health-status
spec:
  type: NodePort
  selector:
    app: usermgmt-restapp
  ports:
  - port: 8095
    targetPort: 8095
#04-NotificationMicroservice-Deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: v1-notification-microservice
  labels:
    app: notification-restapp
    track: stable
spec:
  replicas: 1
  selector:
    matchLabels:
      app: notification-restapp     
  template:
    metadata:
      labels:
        app: notification-restapp
        track: stable        
    spec:
      containers:
        - name: notification-service
          image: stacksimplify/kube-notifications-microservice:3.0.0-AWS-XRay
          ports:
            - containerPort: 8096
          imagePullPolicy: Always
          env:
            - name: AWS_MAIL_SERVER_HOST
              value: "smtp-service"
            - name: AWS_MAIL_SERVER_USERNAME
              value: "AKIASUF7HC7SQJ6BCLVS"
            - name: AWS_MAIL_SERVER_PASSWORD
              value: "BARcmLiC68wgmhTy/cQvz/E8vFzeizGqdeASNtCs6+Nv"
            - name: AWS_MAIL_SERVER_FROM_ADDRESS
              value: "stacksimplify@gmail.com"
            - name: AWS_XRAY_TRACING_NAME 
              value: "V1-Notification-Microservice"              
            - name: AWS_XRAY_DAEMON_ADDRESS
              value: "xray-service.default:2000"      
            - name: AWS_XRAY_CONTEXT_MISSING 
              value: "LOG_ERROR"  # Log an error and continue, Ideally RUNTIME_ERROR – Throw a runtime exception which is default option if not configured                                            
#05-NotificationMicroservice-SMTP-externalName-Service.yml
apiVersion: v1
kind: Service
metadata:
  name: smtp-service
spec:
  type: ExternalName
  externalName: email-smtp.us-east-1.amazonaws.com
#06-NotificationMicroservice-ClusterIP-Service.yml
apiVersion: v1
kind: Service
metadata:
  name: notification-clusterip-service
  labels:
    app: notification-restapp
spec:
  type: ClusterIP
  selector:
    app: notification-restapp
  ports:
  - port: 8096
    targetPort: 8096
#07-ALB-Ingress-SSL-Redirect-ExternalDNS.yml
# Annotations Reference:  https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/ingress/annotation/
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: eks-microservices-demo
  labels:
    app: usermgmt-restapp
  annotations:
    # Ingress Core Settings  
    kubernetes.io/ingress.class: "alb"
    alb.ingress.kubernetes.io/scheme: internet-facing
    # Health Check Settings
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTP 
    alb.ingress.kubernetes.io/healthcheck-port: traffic-port
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: '15'
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: '5'
    alb.ingress.kubernetes.io/success-codes: '200'
    alb.ingress.kubernetes.io/healthy-threshold-count: '2'
    alb.ingress.kubernetes.io/unhealthy-threshold-count: '2'
    ## SSL Settings
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:180789647333:certificate/9f042b5d-86fd-4fad-96d0-c81c5abc71e1
    #alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-1-2017-01 #Optional (Picks default if not used)    
    # SSL Redirect Setting
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'   
    # External DNS - For creating a Record Set in Route53
    external-dns.alpha.kubernetes.io/hostname: services.kubeoncloud.com, ums.kubeoncloud.com       
spec:
  rules:
    - http:
        paths:
          - path: /* # SSL Redirect Setting
            backend:
              serviceName: ssl-redirect
              servicePort: use-annotation            
          - path: /*
            backend:
              serviceName: usermgmt-restapp-nodeport-service
              servicePort: 8095                                   
# Important Note-1: In path based routing order is very important, if we are going to use  "/*", try to use it at the end of all rules.         
#08-V2-NotificationMicroservice-Deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: v2-notification-microservice
  labels:
    app: notification-restapp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: notification-restapp     
  template:
    metadata:
      labels:
        app: notification-restapp
    spec:
      containers:
        - name: notification-service
          image: stacksimplify/kube-notifications-microservice:4.0.0-AWS-XRay
          ports:
            - containerPort: 8096
          imagePullPolicy: Always
          env:
            - name: AWS_MAIL_SERVER_HOST
              value: "smtp-service"
            - name: AWS_MAIL_SERVER_USERNAME
              value: "AKIASUF7HC7SQJ6BCLVS"
            - name: AWS_MAIL_SERVER_PASSWORD
              value: "BARcmLiC68wgmhTy/cQvz/E8vFzeizGqdeASNtCs6+Nv"
            - name: AWS_MAIL_SERVER_FROM_ADDRESS
              value: "stacksimplify@gmail.com"
            - name: AWS_XRAY_TRACING_NAME 
              value: "V2-Notification-Microservice"              
            - name: AWS_XRAY_DAEMON_ADDRESS
              value: "xray-service.default:2000"      
            - name: AWS_XRAY_CONTEXT_MISSING 
              value: "LOG_ERROR"  # Log an error and continue, Ideally RUNTIME_ERROR – Throw a runtime exception which is default option if not configured                                            

Step-03: Review Deployment Manifest for V2 Notification Service

  • We are going to distribute 50% traffic to each of the V1 and V2 version of application
NS V1 Replicas NS V2 Replicas Traffic Distribution
2 2 50% traffic to NS V2 Version
  • 08-V2-NotificationMicroservice-Deployment.yml
    # Change-1: Image Tag is 4.0.0-AWS-XRay
        spec:
          containers:
            - name: notification-service
              image: stacksimplify/kube-notifications-microservice:4.0.0-AWS-XRay
    
    # Change-2: New Environment Variables related to AWS X-Ray
                - name: AWS_XRAY_TRACING_NAME 
                  value: "V2-Notification-Microservice"              
                - name: AWS_XRAY_DAEMON_ADDRESS
                  value: "xray-service.default:2000"      
                - name: AWS_XRAY_CONTEXT_MISSING 
                  value: "LOG_ERROR"  # Log an error and continue, Ideally RUNTIME_ERROR – Throw a runtime exception which is default option if not configured    
    

Step-04: Review Ingress Manifest

  • 07-ALB-Ingress-SSL-Redirect-ExternalDNS.yml
    # Change-1-For-You: Update with your SSL Cert ARN when using template
        alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:180789647333:certificate/9f042b5d-86fd-4fad-96d0-c81c5abc71e1
    
    # Change-2-For-You: Update with your "yourdomainname.com"
        # External DNS - For creating a Record Set in Route53
        external-dns.alpha.kubernetes.io/hostname: canarydemo.kubeoncloud.com
    

AWS EKS - Elastic Kubernetes Service - Masterclass

Image

Step-05: Deploy Manifests

# Deploy
kubectl apply -f kube-manifests/

# Verify
kubectl get deploy,svc,pod

Step-06: Test

# Test
https://canarydemo.kubeoncloud.com/usermgmt/notification-xray

# Your Domain Name
https://<Replace-your-domain-name>/usermgmt/notification-xray

Step-07: What is happening in the background?

  • As far as Notification Cluster IP Service selector label matches to Notificaiton V1 and V2 Deployment manifests selector.matchLables those respective pods are picked to send traffic.
    # Notification Cluster IP Service - Selector Label
      selector:
        app: notification-restapp
    
    # Notification V1 and V2 Deployment - Selector Match Labels
      selector:
        matchLabels:
          app: notification-restapp         
    

Step-08: Clean-Up

  • We are going to delete applications created as part of this section
    # Delete Apps
    kubectl delete -f kube-manifests/
    

Step-09: Downside of this approach

  • We will review the downside in a presentation

Step-10: Best ways for Canary Deployments

  • Istio (Open Source)
  • AWS AppMesh (AWS version of Istio)