IoT: From concept to daily must have

The Internet of things (IoT) made rapid progress from a pure academic concept to play a crucial role in our daily life. From my point of view, the IoT applications can be classified into three categories:

Must have ones which change our lifestyle and ways people are living. Smartphones, connected through 5G/4G LTE network with location detection and other devices connected to the smartphone with their respective applications, apple watch, bluetooth sensors, remote secure monitoring , remote device control applications are good examples of IoT ecosystem in this category.

Nice to have ones, which improve our daily living qualities. Such as wearable IoT devices, Wifi, BT connected fridge, stove, microwave, hue BT based smart lights ambiances bulbs all belong to this group. They are nice to have, but not absolutely necessary.

The last category is the innovation one. It does not exist in our daily life, but with the emergency of such devices, our living habits can be changed and new business revenue modeling is realized. AI enabled drone devices, automatic driving cars and even meta devices can be put into this category.

But, do not neglect one of the most important devices, which play a central and non replaceable role in this overall connected mesh device network: your GPON FTTH home wifi modem. It can not only provide reliable wifi connection to your home with gigabit level up and download speed, but also can be the central equipment to integrate all those individual IoT applications into a single system which can be accessed globally.

Here are two of those IoT device deployment and application examples:

Deployment of GPON FTTH Home Router

I just moved to a new property, there is no internet service yet. After I checked the wiring structure of the house, there is only a cable and phone outlet inside the house, no Ethernet outlet. I verified with the provider that FTTH service is available in my region. The big question in my mind: How will the FTTH be deployed and how can I get Ethernet service based on current wiring structure ? I actually noticed that there is one box beside the cable TV box outside the car garage and was assured this is the fiber cable box for the FTTH service. Here is how all those doubts are answered while I followed the whole FTTH service deployment process.

Connect the Fiber Box to the service center

The technician first connected the fiber box outside of the garage with the provider’s last mile terminator splitter and verified that the optical signal is good enough from the fiber box to the splitter with a optical signal monitor and loopback device.

Now, the physical optical connection from the central office to my house is done. Next step is to install the GPON terminal box with the SFP plugin. The techanin scans the barcode on the side of SFP device and add it to the provider CPE database to activate the device:

After all those, connect the TV cable compatible outlet from above network access hub to the TV cable box outside the garage. Oh !, suddenly I understood that the existing TV cable wiring structure will be used to provide the FTTH service to the house. SMART!

Now, go inside the house, pick up the TV cable outlet, which has been arranged to connect with above FTTH terminal box, finally connect the wireless modem with TV cable to the outlet:

After all those, you have WIFI service to the house with FTTH GPON at backend. Existing cable TV wiring structure is used for connection purposes. No Ethernet wiring is needed. On the back of the above wireless modem, there are a couple of Ethernet ports, which can be used to provide wired Ethernet connections to the devices like PC, smart TV and so on.

Then, the technician scans the above modem device’s barcode, adds it to the provider CPE database, verifies your account and you have both wifi and Ethernet service within the house. I also asked the technician to install a TV cable to Ethernet converter in the basement, so I can use the wired Ethernet function in the basement. Even without an Ethernet wiring structure, I also have an Ethernet outlet in the basement. Wow!

The service was down somehow the next day. I called the technical helpline, and the technician working with me did an online remote loopback testing to make sure everything is fine. The remote loopback test was passed and my modem can be remotely put into reset, reboot and whatever device testing from the provider’s remote central office.

Up to this moment, it has been an enjoyable experience. When worked on GPON FTTH technical project development in Nortel Networks about 10 years ago, I wrote the SFP driver, designed and coded the loopback testing procedures and implemented the service and device management system, such as put the device into service, out of service, monitoring the device with according alarms and so on, today I saw a ture, working deployment example of what we developed years ago. Existing !

Install and control multiple smart devices within home

With the gigabit upload and download WIFI and Ethernet service, different kinds of smart home devices are installed and put into service, including Amazon echo, smart doorbell, smart outlets, smart garage door opener, smart door lock, security camera and a lot of more devices and their according applications on my iPhone. Now, it is time to enjoy my new smart home environment. Here is a shot video I made to demo this kind of new smart home functions:

Demo of new smart home functions

I am sure we will see more and more innovative IoT appliance use cases emerging into our daily life in near future.

Keyuan Zhang

Access K8s API within Container

Modern applications are transformed to microservice based, API driven architecture. When the application is containerized, it needs to access other applications’ published APIs and K8s cluster API for its business logic.

In this article, an python application is containerized, its container image is built and pushed into docker register, an kubernetes manifest is created and deployed into K8s cluster and the containerized python application will access K8s cluster APIs to get cluster’s deployed pods’ configuration information. Each application running within containers of pods needs to be authenticated and authorized by K8s API server before each API access. When the application is containerized and deployed into Kubernetes cluster, each pod belongs to a specific namespace with its according service account (SA). Each service account has its own authentication secret, which can be used to authenticate with K8s API server. Role Based Access Control (RBAC) can then be used to control the authorization of associated service accounts. This service account secret is mounted to a pre-configured path, which can be accessed by the application running within the pod. The Kubernetes python package hides all those details from user python application code to make all those complicated processes easier to achieve.

Step One: Build the container image for the python application

Create a Dockerfile with following content :

controlplane $ cat Dockerfile 
FROM ubuntu
RUN apt-get update
FROM python:latest
RUN mkdir /src
COPY . /src
WORKDIR /src
RUN pip install -r requirements.txt
controlplane $ cat requirements.txt 
Kubernetes

Create following python program in the same directory of above Dockerfile based on this example:

from kubernetes import client, config

def main():
    config.load_incluster_config()

    v1 = client.CoreV1Api()
    print("Listing pods with their IPs:")
    ret = v1.list_pod_for_all_namespaces(watch=False)
    for i in ret.items:
        print("%s\t%s\t%s" %
              (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

if __name__ == '__main__':
    main()

Then, build the docker image and push it to the docker registry:

controlplane $ docker build -t podapiaccess:v1 .
controlplane $ docker tag podapiaccess:v1 twintiger/podapiaccess:v1
controlplane $ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: <Your Docker Registry User Name>
Password: <Your according password>

Login Succeeded
controlplane $ docker push twintiger/podapiaccess:v1
The push refers to repository [docker.io/twintiger/podapiaccess]
084f4d130a72: Pushed 
d32ebda4ee0b: Pushed 
db8d0fe6cf95: Mounted from library/python 
00901a4c6fc7: Mounted from library/python 
7e7decd61f68: Mounted from library/python 
aedcb370b058: Mounted from library/python 
c3a0d593ed24: Mounted from library/python 
26a504e63be4: Mounted from library/python 
8bf42db0de72: Mounted from library/python 
31892cc314cb: Mounted from library/python 
11936051f93b: Mounted from library/python 
v1: digest: sha256:795b7425e878f4478147a6c136eba6994fc0113cf8c7537fbe42954fb1257582 size: 2631
controlplane $

Step Two: Test the newly built docker image

Test the above docker image by following steps:

controlplane $ docker run twintiger/podapiaccess:v1
Unable to find image 'twintiger/podapiaccess:v1' locally
v1: Pulling from twintiger/podapiaccess
0e29546d541c: Pull complete 
9b829c73b52b: Pull complete 
cb5b7ae36172: Pull complete 
6494e4811622: Pull complete 
6f9f74896dfa: Pull complete 
fcb6d5f7c986: Pull complete 
290438add9da: Pull complete 
ab11df61f44a: Pull complete 
de4793a5fa46: Pull complete 
b7c69d9f5717: Pull complete 
4d8c3388b53a: Pull complete 
ceea8c40c3ea: Pull complete 
Digest: sha256:5c800bd260a1cea55d71c4454166c49597c8f35d3520eadf7e82e8643f198cbf
Status: Downloaded newer image for twintiger/podapiaccess:v1
Traceback (most recent call last):
  File "/src/podapiaccess.py", line 16, in <module>
    main()
  File "/src/podapiaccess.py", line 5, in main
    config.load_incluster_config()
  File "/usr/local/lib/python3.10/site-packages/kubernetes/config/incluster_config.py", line 121, in load_incluster_config
    try_refresh_token=try_refresh_token).load_and_set(client_configuration)
  File "/usr/local/lib/python3.10/site-packages/kubernetes/config/incluster_config.py", line 54, in load_and_set
    self._load_config()
  File "/usr/local/lib/python3.10/site-packages/kubernetes/config/incluster_config.py", line 62, in _load_config
    raise ConfigException("Service host/port is not set.")
kubernetes.config.config_exception.ConfigException: Service host/port is not set.
controlplane $

Based on the above testing, the python code failed. Why ? The authentication with K8s API server failed. There is no corresponding authentication secret which is needed for the python application code to be authenticated with K8s API server.

This piece of python container code must be deployed into Kubernetes cluster, running from pod to make authentication process success.

Step Three: Deploy the containerized python application inside Kubernetes cluster

Create following Kubernetes deployment manifest:

controlplane $ cat deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-k8s-api
spec:
  selector:
    matchLabels:
      app: hello-k8s-api
  replicas: 1 
  template:
    metadata:
      labels:
        app: hello-k8s-api
    spec:
      containers:
      - name: hello-api
        image: twintiger/podapiaccess:v1
        command: ["/bin/sleep", "3650d"]
        imagePullPolicy: IfNotPresent
      restartPolicy: Always

Deploy above manifest and verify its status by following procedures:

controlplane $ kubectl apply -f deployment.yaml 
deployment.apps/hello-k8s-api created

controlplane $ kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
hello-k8s-api-5769fb66fb-gtb2f   1/1     Running   0          29s

Test the python Kubernetes API access code by:

controlplane $ kubectl exec -it hello-k8s-api-5d5699b6b-b4v45 -- bash
root@hello-k8s-api-5d5699b6b-b4v45:/src# pwd
/src
root@hello-k8s-api-5d5699b6b-b4v45:/src# ls
Dockerfile  requirements.txt
podapiaccess.py

root@hello-k8s-api-5d5699b6b-b4v45:/src# python podapiaccess.py
Listing pods with their IPs:
10.32.0.193     default hello-k8s-api-5d5699b6b-b4v45
10.88.0.2       kube-system     coredns-fb8b8dccf-fmrd8
10.88.0.3       kube-system     coredns-fb8b8dccf-tzv4p
172.17.0.24     kube-system     etcd-controlplane
10.88.0.4       kube-system     katacoda-cloud-provider-776f5d7cb-g6r57
172.17.0.24     kube-system     kube-apiserver-controlplane
172.17.0.24     kube-system     kube-controller-manager-controlplane
172.17.0.26     kube-system     kube-keepalived-vip-r5xxf
172.17.0.26     kube-system     kube-proxy-5qdzn
172.17.0.24     kube-system     kube-proxy-wnzl6
172.17.0.24     kube-system     kube-scheduler-controlplane
172.17.0.26     kube-system     weave-net-9pjqm
172.17.0.24     kube-system     weave-net-wjt9x

The python K8s API access code works and the IP address of all pods are listed.

Now, let’s dig into the deployment and its according pod:

controlplane $ kubectl get deployments
NAME            READY   UP-TO-DATE   AVAILABLE   AGE
hello-k8s-api   1/1     1            1           5m44s
controlplane $ kubectl get pods
NAME                            READY   STATUS    RESTARTS   AGE
hello-k8s-api-5d5699b6b-b4v45   1/1     Running   0          5m50s
controlplane $ kubectl logs pod/hello-k8s-api-5d5699b6b-b4v45
controlplane $ kubectl describe pod/hello-k8s-api-5d5699b6b-b4v45
Name:               hello-k8s-api-5d5699b6b-b4v45
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               node01/172.17.0.26
Start Time:         Fri, 31 Dec 2021 19:12:46 +0000
Labels:             app=hello-k8s-api
                    pod-template-hash=5d5699b6b
Annotations:        <none>
Status:             Running
IP:                 10.32.0.193
Controlled By:      ReplicaSet/hello-k8s-api-5d5699b6b
Containers:
  hello-api:
    Container ID:  docker://750c20dd890cff333ccbb44de4b1be9ea59f73ffd7a9edf34474b30e6d4a8101
    Image:         twintiger/podapiaccess:v1
    Image ID:      docker-pullable://twintiger/podapiaccess@sha256:e809bd1c327941f9de5d4306c9cc9e507b84d0a3020fe8cd0c8dbe01736e058e
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sleep
      3650d
    State:          Running
      Started:      Fri, 31 Dec 2021 19:13:23 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-64mfb (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-64mfb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-64mfb
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  6m17s  default-scheduler  Successfully assigned default/hello-k8s-api-5d5699b6b-b4v45 to node01
  Normal  Pulling    6m16s  kubelet, node01    Pulling image "twintiger/podapiaccess:v3"
  Normal  Pulled     5m40s  kubelet, node01    Successfully pulled image "twintiger/podapiaccess:v1"
  Normal  Created    5m40s  kubelet, node01    Created container hello-api
  Normal  Started    5m40s  kubelet, node01    Started container hello-api
controlplane $

Pay special attention to the following information of the pod within the deployment: 

Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-64mfb (ro)

This secret is the service account’s secret within the namespace, which is used by pod to authenticate with K8s API server.

Keyuan Zhang

Secure Cloud Infrastructure and Deployed Applications through Multiple Principles

When microservice oriented applications are deployed into Kubernetes orchestrated cloud platform, the security model works well in traditional enterprise application deployment workflow will face a lot of new challenges and needs to be transformed and microservice friendly also.

To achieve end to end application level security needs coordination between different components within your cloud cluster: From platform software bootstrapping, operation system installation, system software patching and upgrade, user authentication and authorization, certificate management, network policy, application traffic rate limiting, policy agent, ingress and egress traffic control to name a few.

Start from platform software and keep it updated

Using technologies such as secure boot with signed operation system image, secured container images repository and signed docker application images to bootstrap your platforms within the cluster. Keep it secure and away from vulnerable security hacking by constantly updating and patching your whole cloud platform infrastructure. Deploy HA enhanced control plane to increase overall Kubernetes cluster availability. Use CEPH CSI backend to enhance data availability in your cluster.

Monitor your cluster system containers, application containers, storage, computing and networking resource utilization, performance with toolset such as ELK stack to warn, log and debug possible cluster system and application level issues.

Keep your certificate and secret safe

There are a lot of certificates used in the modern Kubernetes cluster: Kubernetes root CA, docker image repository CA, ETCD root CA, multiple system application root CA and so on.

Keep those CA and Kubernetes clusters secret safe using ways such as TPM hardware, Key Management System (KMS) to store the key remotely and encrypt your system disk, kubernetes system database content to avoid physically stolen hardware information leak.

Expire and renew those certificates periodically with the help of open source solutions such as certificate manager.

Deploy suitable user and account authentication and authorization mechanism

RBAC is the fundamental mechanism to achieve user, group and internal service account authentication and related authorization. You can define roles, cluster roles and bind those roles with certain users, groups and internal service accounts used by pods when those are deployed.

PSP (Pod Security Policy), its replacement: PSS(Pod Security Standard) and OPA (Open Policy Agent) can be used to enforce the security policies before a Pod can be allowed to be deployed into the cluster.

Control cluster network traffic with network policies

Besides the pod security policy, user and user group authentication and authorization, network policy can be used to control its network traffic behaviors: allow some kind of network TCP/UDP of certain ports, deny some of network traffic in both ingress, egress direction.

Decouple application with underlining network layer functions using service mesh

Service mesh can be used to decouple the application logics and its underline networking layer related control and monitor functions, such as rate limiting, service call trace, mTLS security realization.

Others

Besides above, other must-have mechanisms to secure your cluster includes using secured, trusted application container image repositories, signed container images, and highly available infrastructures to name a few.

Summary

There are multiple ways to secure your cluster. Balancing the security rigidness and ease of use is a real challenge in day by day cluster operation.

Achieve High Performance Cloud Computing using EPA

Keyuan Zhang

When you virtualized your infrastructure or applications in form of IaaS, PaaS and SaaS using virtual machine or containers, you gain a lot of benefits: easy to deploy, low CAPS, quick application deployment , load balanced based high availability, better resource utilization through sharing, open to 3rd party open sourced new technologies and so on. You also need to consider what the new environment means for your application: your application is sharing the underline cloud-based computing, networking and storage resources with an unknown number of other applications. The delay between client and server communication will be longer compared with the traditional way deployed through a dedicated hardware server.

EPA (Enhanced Platform Awareness) can help to improve the performance of crucial applications. It did this by allocating dedicated computing resources including cpu cores, memory page, cache memory, dedicated networking interface resource to the VM or container and giving the exclusive usage of those resources to those performance sensitive crucial applications. After those reservations, the rest of platform resources can be shared between other virtual machines and containers.

In modern Intel Xeon architecture, there are multiple CPU sockets, each with multiple cpu cores, each socket has its directly attached memory chips, PCI devices (PCI NIC card, GPU, co-processors). This is called NUMA(non-uniform Memory Architecture) in Intel’s term. EPA is realized to reserve a certain number of dedicated cpu cores to run your application and pin those cores to be co-located in the same socket where the memory, cache and network interface PCI card your application running from are all attached to the same cpu socket or NUMA node. This will give your application the best performance. Of course, applications can use resources such as memory, PCI devices attached to other CPU NUMA nodes. But, the side effect is that all data packets and memory access must go through the mesh connectors between different sockets: the QPI interface, which has capped bandwidth and this will become the performance bottom neck of your applications . The EPA principle can be illustrated by following diagrams:

By apply EPA principle, you assign dedicated CPU cores to virtual machine or container, assign dedicated memory, dedicated PCI device attached to the same cpu socket or NUNA node, passing through Ethernet card or its SR-IOV functions directly to those virtual machines or containers to guarantee the exclusive usage of those resources. You can also isolate VM or container cache allocation to avoid its content being evicted or inspected by another VM or container from both performance and security point of view.

How to achieve EPA ?

First of all, You can use following commands to find your platform’s CPU core, memory, network CPI device and its associated NUMA node information to gain a whole picture of your compute platform’s resources:

$cat /proc/cpuinfo
$lspci |grep Network
(Get Network NIC PCI Bus ID)
$cat /sys/bus/pci/devices/<NIC-PCI-Bus-ID>/numa_node

For allocating dedicated cores, memory and so on to OpenStack VM, flavor for virtual machines is used to define how many, which NUMA node the cores will come from, which cores on a specific CPU socket is used for reserved, dedicated purpose, specify the affinity requirement between virtual machine and NUMA node’s PCI devices. After that, launching a virtual machine using this flavor can guarantee those resource allocation for your virtual machine application. For example, set your OpenStack flavor as following:

openstack flavor set --property hw:cpu_policy=dedicated vm-flavor
openstack flavor set --property hw:numa_nodes=2 vm-flavor
openstack flavor set --property hw:numa_node.0=1 vm-flavor
openstack flavor set --property hw:numa_node.1=0 vm-flavor
openstack flavor set --property hw:numa_cpus.0=0 \  hw:numa_cpus.1=1,2 vm-flavor 
openstack flavor set --property hw:numa_mem.0=1048576 \  hw:numa_mem.1=1048576 vm-flavor 
openstack flavor set --property hw:mem_page_size=1024 vm-flavor
openstack flavor-key vm-flavor hw:pci_numa_affinity= strict
openstack flavor set vm-flavor --property "pci_passthrough:alias"="<PCI device alias name:Num of Device>"

The above flavor defined two virtual NUMA nodes. The virtual NUMA node 0 is mapped to physical NUMA node 1 and virtual NUMA node 1 is mapped to physical NUMA node 0. Virtual node 0 has one dedicated core coming from core number 0 from its underlying physical NUMA node. Virtual NUMA node 1 has two cpu cores mapped to core number 1 and 2 of the physical NUMA node. Each virtual NUMA node will have 1 Gb memory with a page size of 1024. The VM must be scheduled on NUMA cores with a direct PCI network device attached to it. The network PCI device is passed through to the VM directly when you launch the VM with command like: 

openstack server create --flavor vm-flavor --image <ID of Image> <Name of VM>

After cores are reserved for the virtual machine, the application running within it can check the available cores by routine like: sched_getaffinity() by code or look at the content of file: /dev/cpu and /proc/cpuinfo directory.

For containerized applications orchestrated by Kubernetes, achieving EPA is relatively easier due to the fact that Kubernetes uses a declarative way for deployment. You can use yaml file to define the desired state of your application, specify the replicate number of containers running your application, which image to run your application from, the number of cores, memory size required directly. For SR-IOV devices reserved for your application to use exclusively, you need to provision those resource first by using the “NetworkAttachmentDefinition” with a name and according parameters as following example:

apiVersion: "K8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: sriovi-vfio
  annotations:
    k8s.vl.cni.cncf.io/resourceName:  
      intel.com/pci_sriov_net_sriov_device
spec:
  config: '{
    "cniversion": "0.3.0",
    "type": "sriov",
    "vlan": 100,
    "link_state": "disable",
    "trust": "on"
}'

During the Pod definition stage, annotation metadata will be used to attach SR-IOV resource to the pod network namespace. Application pod’s core, memory request and SR-IOV network device reservation will be specified within container’s resource definition session per following code snippet:

apiversion: apps/v1 
kind: Deployment 
metadata:
  name: test-deployment
  labels:
    app: test-pod 
spec:
  replicas: 3
  selector:
    matchLabels: 
      app: test-pod
  template:
    metadata: 
      annotations:
        k8s.vi.cni.cncf.io/networks: '[
        { "name": "sriov-netdevice" }
        ]'
      labels: 
        app: test-pod
   spec:
     node Selector:
       kubernetes.io/hostname: SRIOV-host
     containers:
     - name: test-pod
       image: centos/tools
       imagePullPolicy: IfNotPresent
       command: [ "/bin/bash", "-c", "--"]
       args: [ "while true; do sleep 300000; done;" ]
       resources:
         requests:
           intel.com/pci_sriov_net_sriov_device: 'l'
           memory: 10Gi
           hugepages-1Gi: 4Gi
         limits:
           intel.com/pci_sriov_net_sriov_device: '1'
           memory: 10Gi
           hugepages-1Gi: 4Gi
       volumeMounts:
       – mountPath: /mnt/huge-1048576kB
         name: hugepage
       volumes:
       - emptyDir:
         medium: Huge Pages
         name: hugepage

The above declarative Kubernetes manifest deployed three copies of your application pod running from centos image with 10G memory, 4G of 1G size of huge page and one SR-IOV backed network interface. After the container is running, the code within the container can parse /sys/fs/cgroup/cpuset/cpuset.cpus for cpu related information and check huge page and network related information as normal.

What,Why and How: Infrastructure Orchestration

In the cloud computing field, there are already a lot of open sourced or private automation tools and infrastructure solutions such as Ansible, Chef, Puppet, OpenStack, Kubernetes. There are many on-premise or public cloud providers available for your infrastructure and applications. People asked why we need more orchestration solutions ?

What is Infrastructure Orchestructure ?

It is sometimes called orchestrator for orchestrator. True, you can all think OpenStack, Kubernetes, AWS, GKE, VMWare are all kinds of orchestrators and meet your cloud native application deployment requirements to some degree.

The Infrastructure Orchestructure is system wide orchestrator and using OpenStack, Kubernetes, VMWare, AWS and so on as underlining orchestrators to deploy system wide business applications across multiple cloud providers, multiple regions and heterogeneous underling technologies.

Why do we need it ?

Business cloud native applications need to deploy over hybrid clouds, over different underlying technologies. It needs an end to end, full scale Infrastructure as Code solution. That is what the Infrastructure Orchestrator provided.

How to do end to end Orchestration ?

Two types of solution exit for this requirement. One is a private solution such as Terraform from HashiCorp and the other one is open sourced, OASIS TOSCA (Topology and Orchestration Specification for Cloud Applications) template based solution by companies like Cloudify. All those realized the so-called Infrastructure as Code method.

Want to know each one’s strengths and best suitable use cases, here is quick discussion on this topic. Personally, I prefer the Cloudify solution for its open source characteristics, full scale capacity and more applicable to the VNF field.

Now, let’s take a little bit explanation of Cloudify solution architecture as illustrated by following diagram:

Source: Cloudify.co

Basically, you code your infrastructure as code in yaml TOSCA template. The Cloudify rendering engine will start from there to deploy your solution end to end crossing multiple cloud providers. It will also take care of the tasks such as healing, scale up and down your end to end cloud resources based on business requirements.

Demo

Let’s break the gap between theory and real use cases using a trivial demo. Here, we have a deployed, working OpenStack cloud and I will use Cloudify solution to deploy an application over it. In a more real deployment scenario, your application can deploy over hybrid cloud, different public cloud (AWS, GKE, VMWare . . .), mix of VM, Containers, Bare metal, SDN and so on, end to end, automatically using a TOSCA template. This saves the hassles of deploying business applications needed by individual cloud providers one by one and then connecting those segmented pieces of solution into one end to end solution manually. Here is the summary of demo steps:

  • Log into deployed, working OpenStack cloud through Horizon GUI interface to exame its deployed resources. The existing resources include private and public networks, Neutron router to connect those two network segments, VM image. But, there is NO VM deployed already.
  • Log into Cloudify GUI interface, load TOSCA OpenStack plugins. You can think of plugin as the abstraction layer between the TOSCA engine and underlying, deployed clouds. By this way, the same application can be deployed over any popular public or private cloud such as OpenStack, AWS, GKE and so on to achieve cloud agnostic.
  • The next step is to load the network blueprint and deploy it. Blueprint is the modeling template for your system, network and application. You can open the blueprint with any editor to view its content.
  • Cloudify displays the deployment model itself and creation progress in a graphic way.
  • Go back to OpenStack Horizon interface to check the new OpenStack resources created from network TOSCA template: new network segments, router are created. You can compare the network topology after and before the TOSCA template is applied there. There is no VM yet.
  • Load application blueprint after the success deployment of network blueprint. After that, the VM running your application and its corresponding floating IP address is successfully deployed.
  • You can then use the floating IP to access your application.

Here, we used the TOSCA modeling template, successfully deployed our application over OpenStack cloud.

Achieve CNCF WorkFlow using Azure Public Cloud

CNCF recommends four steps to achieve cloud native for your application:

  • Containerization of your application through Docker and push its image into registry
  • Setup CI/CD pipeline for DevOps automation
  • Using orchestrator such as Kubernetes to manage the full application life cycle
  • Add observability and analysis capacity to your deployed application
  • Add building block such as ingress, service mesh to enhance security, traffic control, tracing capacity to deployed application

Let’s start from scratch to achieve those goals using Microsoft Azure public cloud. At the end of this demo, you will have a deployed, containerized service running in a public cloud meeting above CNCF criteria.

Deploy a Kubernetes Cluster in Azure

This can be done either using Azure CLI or its GUI interface following a couple of easy steps. Here is the deployed Kubernetes cluster inside my Azure account using GUI interface from the quick start session:

Follow those GUI easy steps, select your cluster name, create a resource group, networking method and so on, you will have your Kubernetes cluster ready to use.

After your Kubernetes orchestored cluster is ready, go to its CLI to use your familiar command like kubectl to exam your cluster:

Deploy a containerized Application

Now, let’s deploy a containerized wordpress application inside the Kubernetes cluster following this procedure: wordpress application and it works without any issue. Here, containerized wordpress application, associated database, persistent volume are realized through Azure CLI command using kubectl. The service type used during this deployment is load balancer and you will get an routable IP address to access your service after those procedures.

Verify your wordpress application is up and running using the global IP address assigned to WordPress service:

Install Kubernetes dashboard to your cluster to add basic monitoring capacity by following step:

Get the global web address and according sign in token; then go to this web page, select sign in with token:

From there, you can do normal cluster administrative tasks through this GUI interface.

Add Cluster Monitoring and Visibility Facilities

There are built-in monitoring features inside Azure infrastructure such as Logs, Metrics, Insights, which can be accessed from your Azure portal:

Besides that, I tried to install and use two other Kubernetes cluster monitoring solutions in the above cluster: the managed solution from datadog and the open sourced one: prometheus with grafana. Those all rely on Kubelet cAdvisor, metrics server and kube-state-metrics component to scrape raw data from node, kubernetes itself, its deployed pods and applications running over it.

Helm is used to installed datadog agent inside the cluster by those steps and the containerized Datadog agent is deployed as a Kubernetes daemonset:

Verify datadog agent is successfully installed on your cluster from Azure CLI:

For quick testing purpose, disable TLS authentication with Kubernetes API server for Datadog by update Datadog helm chart’s values.yaml and update Datadog agent with helm upgrade command in order to make the Datadog agent to work:

Go to your datadog web portal, select your cluster, you should be able to see your cluster related metrics:

Let’s install the popular open source monitoring tool: Prometheus and its Grafana dashboard.

Source: https://prometheus.io/docs/introduction/overview/

Prometheus and Grafana Helm charts can be used to finish those two installations pretty easy and straightforward. Within helm git repo downloaded before, those Helm charts have been downloaded already. Follow those simple steps, you can install both tools from Azure command line:

kevin@Azure:~/charts/stable/prometheus$ helm dep update
kevin@Azure:~/charts/stable/prometheus$ kubectl ns create monitoring
kevin@Azure:~/charts/stable/prometheus$ helm install prometheus . --namespace monitoring --set rbac.create=true

Make sure to expose prometheus as a load balancer type of service to let Azure assign it a routable IP address. Then, you can access prometheus build in dashboard through that IP address with port 9090:

Use similar step to install Grafana dashboard from its helm chart, expose its service as load balancer type and you can access Grafana dashboard from Aure assigned routable IP address:

Add prometheus type data source to Grafana:

Go to Grafana web page to download a dashboard examples, for example, dashboard number 11802, import it into Grafana through GUI to start your cluster monitoring:

In order to let Prometheus to monitor your application besides the infrastructure components, you have to add prometheus data feeding capacity into your containerized application by using its SDK library, instrument your application with prometheus.io/scrape: ‘true’ and prometheus.io/port” ‘9090’ annotations. For example, by following yaml descriptive file, you can deploy a service with its according deployment with above annotations to monitor the performance specific to your deployed application.

Add Service Mesh and Ingress to your Cluster

Service mesh solutions, like Istio, can be used to provide an infrastructure layer for the deployed micro service applications, decouple the application function with underlying infrastructure functions, discover, monitor and track deployed application, secure and rate limiting communications between services,  through non instrumented sidecar mechanisms. Its installation on Azure is well documented by Azure AKS user guide. Same as installing an Ingress controller for your cluster, which can provide layer 7 application traffic routing and reverse proxy functions. I finished all those components’ installation without issues by following those installation guides: Istio service mesh inside Azure and Ingress Controller inside Azure. After the installation, you can play with some of the reverse proxy application routing examples. Here are some steps I captured during this process:

Use CI/CD Pipeline

Azure has built-in CI/CD solution by setting up a DevOps project following “Create a CI/CD Pipeline with Azure DevOps Projects” from its quick start session if you develop your application using .NET, Java, Python, PHP, Node.js. It has the basic features such as auto release building upon new code check in, auto testing and deliver. But, for more advanced features, you have to use open source tools like Jenkins/Jenkins x/Spinnaker and even bring in service mesh solutions for advanced canary release, rolling update of your microservice applications.

Summary

Through the above procedures, a full cycle of CNCF recommended work flow is achieved using Microsoft Azure public cloud.

Keyuan Zhang

Video:Utilize Multiple Technologies for Cloud Infrastructure

Multiple cloud Virtual Infrastructure Manager (VIM) exists for cloud infrastructure, such as OpenStack, Kubernetes, Kubeless, CloudStack to name a few. How to combine and utilize the best pieces from different solutions for the sake of faster cloud deployment, easy to manage, update and upgrade ? What’s the role of service mesh, API gateway will play for the overall cloud native application deployment architecture ? What are serverless functions ? How to achieve end to end visibility for your deployed cloud ?

For sure, no single solution can achieve all those goals. Stay tuned for this video I posted in Linkedin originally to understand technologies, solutions addressing those challenges.

Put the Best Aspects from Kubernetes and OpenStack into your Cloud

Two Key Takeaways from F5 Agility Conference

Digital transform will change the way of code development, service delivery, daily working style and more. How do we adapt to this change? We need to adopt a batch of new technologies, culture changes and promote new best practices.

From a technology point of view, a shift from monolithic way of development to cloud native, microservice based, containerized and orchestrated applications will be the first step. This transform will bring benefits with challenges: How to gain the end to end observability of tens to hundreds of, constantly migrating distributed services running from containers? What about everyday emerging new tools, open source solutions? How to handle more serious security issues with this new architecture? and lots of more questions.

I just attended the 2020 F5/Nginx virtual Agility conference and found those are common questions and concerns from the industry, technology community. F5/Nginx solutions have some very nice products and solutions for those concerns and let me share my comments on those.

Culture Change and Solution Architecture

The transformation starts from company culture change. F5 promoted 7 principles driven all their product development cycle, Which I can not agree more: Application first, multi cloud compatible / platform independent, API first, integrated with AI and end to end analytics and visibility, build in security, reusability with modular design and the last one: open source at core.

Here is the architecture diagram taken from the conference which best describes above culture change:

First of all, the code can be run in any cloud: public cloud like AWS, GCE or private cloud, communication between different services is using declarative APIs, security is built into every level of application from application level using WAF (Web Access Firewall), infrastructure level using ingress controller, service mesh, DDoS service. Telemetry is provided for all the components and full data path with integration of open source based toolsets. You get a single pair of glass views of your services from end to end with one integrated control plan. AI is used to enhance the security algorithm and service routing. With this architecture, DevOps,SecOps,AppDev and NetOps teams can do their jobs more efficiently within the same environment. That is pretty cool!

NFV/Telco Cloud needs Same Transformation

People usually think traditional IT service such as 3-tire based web application architecture is the best candidate for above digital transformation. Just like the message I got from the conference: The same transformation is coming for communication service providers. NFV is adopting containized implementation, moving close to the edge for 5G application latency concern, and migrating together with the application which consumes its collected data. This is illustrated by following diagrams:

The same 7 principles are carried out in Telco cloud with more emphasis to move containerized NFV to edge and migrate VNFs together with the applications using the data streams coming from it. Latency insensitive applications can still be hosted in traditional data centers while public cloud can be used to host other classic applications.

Actually, this already happened in the service provider world from core to edge NFV cloud. Project likes Airship from AT&T adopted container based, orchestrated architecture to take advantage of the same benefits from microservice infrastructure: agility, easy to deploy new release in CI/CD pipeline, auto scaling of services and resilience.

What I am still looking for?

All those services, solutions and architectures look pretty impressive. Things, I found missing to better facilitate this transformation, include:

  • Top to down telemetry integration: The above architecture provided end to end service application tracing, metrics, logs and so on. Most of those covers east to west traffic even with service mesh. API gateway North to South integration stops at infrastructure level. Customers still need to collect telemetry data from both the communication service provider and application provider to figure out the full North to South and vice versa picture. For example, using F5 tools, you found the packets are dropped in a queue and understood this was caused by underlying cloud infrastructure, there is no integration between those F5 toolsets with telco telemetry such as SDN opendaylight to dig out where the bottleneck is.
  • Still too many tools to choose from and master. F5 integrated a lot of open source tools and Nginx tools into their products. But, customers still have to use different tools from different products to fit for the best interestings of different application scenarios. If you are now using an open source or other 3rd party tool to do your daily business and those are not integrated, tested and verified with the F5 solution, we can only wish you best luck. It is not easy to DIY this without a steep learning curve and huge investment due to lack of expertise on those topics.

Overall, I am pretty excited about the solutions provided by F5/Nginx to help customers to speed up their journey to modernize their services to microservice based architecture with confidence and manageability.

Keyuan Zhang

Four Steps to achieve the ultimate CI/CD goals with Kubernetes

Zero downtime, Zero traffic performance penalty and metric impact are the ultimate goals of enterprise level CI/CD (Continuous Integration/Continue Delivering) workflow in cloud native environment.

This can not be achieved by a single piece of tools. We need multiple solutions working together. Kubernetes software release rollup and rollback with canary release support itself is not enough.

First: Use Kubernetes friendly CI toolset

For CI toolset, it should be well integrated with github, docker registry, kubernetes orchestrator and the toolset itself should be cloud native and easy to be deployed in public or private cloud. The tool will carry out daily kubernetes development tasks: containerized application, build and tag docker image, push it into the registry, detect the Kubernetes manifest change, deploy those Kubernetes objects, test and bug fix, push the source code ( Docker file, Application Source Code , Kubernetes Manifest File) into github automatically and smoothly.

Jenkins is the original, de facto standard in CI/CD and the Jenkins X version is its cloud native Jenkins implementation. If you think Jenkins is a little bit too complicated and heavy for your development cycle, the Forge project provides the starting point for this Kubernetes based CI workflow.

Second: Use Kubernetes Correctly

To best utilize Kubernetes canary release related features, following key points must be followed:

  • Understand when Kubernetes rolling update will happen: rolling update of deployment will be automatically triggered when you update the images, configuration, labels, annotations, resource limits and scale up your resource in the cluster. You can trigger an update rollout by updating the object’s spec: template.
  • Rolling updates are designed to update your workloads without downtime with following canvas:
    • No matter how you adjust the Kubernetes rolling update strategy parameters: maxSurge and maxUnavailable number, there is a service available gap. It is kube-proxy or ingress that detects Kubernetes object endpoints changes, updates node’s iptables and terminates old pods. This is an asynchronous process and may break the zero downtime rule.
    • Your containerized application should be designed to handle shutdown gracefully and may implement mechanisms to coordinate during the process of bring up of the new pod and killing of old pod. One way is to check the readiness of new pod before shutdown of the old pod and implement shutdown hookup handler to make sure kube-proxy and ingress have enough time to update its traffic routing tables.

Third: Use Service Mesh to control Traffic Flow

Using plain Kubernetes canary deployment method, the traffic load distribution is controlled by replica ratio. Large number of replica sets is needed to realize such as 1% of traffic going to canary deployment for example ( 100 replica sets are needed). To maintain the fixed traffic ratio will become more awkward when the Horizontal Pod Autoscaler (HPA) feature is enabled for traffic auto scaling. It also lacks the feature such as routing a specific sourced traffic to canary pods and so on.

Service mesh technology can precisely control the portion of traffic fed into newer versions of applications through the concept of traffic weight. It can implement traffic routing rules for the scenario we discussed before. Service mesh treats traffic routing and replica deployment completely independently. HPA can work independently with traffic routing rules.

Fourth: Coordinate with Metrics Measurement

During canary development, you should not only make sure new services are functional, but also need to guarantee there is no traffic performance impact during the whole process.

CD tool needs to work with the cluster metrics collection tool to monitor pod, node and whole cluster traffic metrics and feedback those to CD toolset. When there are any unexpected traffic performance impacts, such as traffic unexpected jitta, delay and loss, you can pause and rollback your deployment. For example, Spinnaker toolset coming from Netflix is continuous deployment centric. It is integrated with Prometheus and Datadog cluster metrics monitoring tools and can make decisions about deployments based on metrics. For canary deployment, if the canary deployment has caused any pertinent metrics degradation, you can roll it back. You can also integrate Nginx cloud application controllers, even some AI functions to feed back measured KPI results directly to the CD toolset and make roll back or roll up decisions automatically.

Achieving a successful CI/CD pipeline in a cloud native enterprise environment is a must do task. By adopting correct toolsets and methodology, you will reap the fruitful results it brings to your software development cycle.

Keyuan Zhang

Infrastructure Orchestrator for Cloud Computing

Cloud computing provides huge benefits for almost all existing business use cases. XaaS is a popular term in our daily life. Just like computers provide a huge productivity boost for our daily life, it comes with some kind of overhead. Software installation, hardware management, security, system upgrade and update, networking, all kinds of external devices and so on, are all tasks we have to do daily in order to maintain a working, high performance computer system, not to mention when its computing, storage and networking functions are moved into a cloud computing environment. There are tens, hundreds and even thousands of powerful servers involved to provide the so-called XaaS services to end customers.

There are different kinds of Virtual Infrastructure Manager (VIM), such as OpenStack and Kubernetes, to ease this task. They take care of orchestration of user applications’ deployment, monitoring and some degree of high availability, service scale up and down and so on for your applications running inside virtual machines or containers. But, the initial cluster installation, bootstrap and afterward infrastructure software update and upgrade, bug fix , patching are all manual processes which are time consuming, painful and error prone, plus infrastructure’s fault management, fault alarming, adding new compute nodes, cluster wide system resource usage monitoring. Those are values provided by, what is so called “Infrastructure Orchestrator”, for cloud computing. It fulfilled the task to take care of cloud computing infrastructure itself’s installation, monitoring, update and upgrade.

Open sourced edge computing project: StarlingX , seeded from Wind River Cloud Platform, is just such kind an infrastructure orchestrator. Its architecture is illustrated by following diagram:

The infrastructure orchestrators are those components within the red box of the above diagram. After the first node of the cluster, usually the system controller, is bootstrapped manually through installation media, it will be configured and put into service using open source technology such as Ansible YAML file descriptively. After this, all necessary cloud computing software components such as container runtime, Kubernetes, database, docker registry, calico networking plug in, Helm, Armada tools are all installed and bringed to the functional stage automatically. This brings the node into a fully functioned, Kubernetes managed platform. After that, adding new controller or compute nodes, configure those, install and update its system software, report any fault, raising any necessary alarm, entire cluster software update and upgrade, operating system patching and upgrade for server hardware, even software update of infrastructure orchestrator itself, are all be taken care by this infrastructure orchestrator in CI/CD way. All those day two cluster management tasks are becoming much easier with significant release cycle reduction.

Infrastructure orchestrator fills the gap between application container or virtual machine management done by Kubernetes or OpenStack and the underline cluster hardware and software management for cloud computing infrastructure.

Keyuan Zhang