High-availability cluster (5)-k8s deployment of microservices

High-availability cluster (5)-k8s deployment of microservices

k8s stateful service

What is a stateful service

  • Stateless service

    • 1. It means that the instance of the service running will not store the data that needs to be persisted locally, and the results of multiple instances for the same request response are exactly the same
    • 2. Multiple instances can share the same persistent data. For example: nginx instance, tomcat instance, etc.
    • 3. Related k8s resources include: ReplicaSet, ReplicationController, Deployment, etc. Because it is a stateless service, the pod sequence numbers created by these controllers are all random values. And when shrinking, it does not explicitly shrink a pod, but randomly, because all instances get the same return value, so any pod can be shrunk
  • Stateful service

    • 1, analog pets and cattle, cattle farmers can lose if the disease re-buy one, if the dead pet owners pet is not exactly find a pet; stateful service can be said that the service needs of data storage , Or refers to multi-threaded services, queues, etc. (mysql database, kafka, zookeeper, etc.)

    • 2. Each instance needs to have its own independent persistent storage, and in k8s it is defined by a declaration template; the persistent volume declaration template is created before the pod is created and bound to the pod, and multiple templates can be defined

      Note: Stateful pods are used to run stateful applications, so the data stored on the data volume is very important. It will be disastrous to delete this statement when Statefulset is scaled down, especially for Statefulset. It is as simple as reducing its replicas value. For this reason, when you need to release a specific persistent volume, you need to manually delete the corresponding persistent volume declaration

    • 3. The related k8s resource is: statefulSet. Because it is a stateful service, each pod has a specific name and network identifier. For example, the pod name is composed of statefulSet name + ordered numbers (0, 1, 2..)

    • 4. When performing scaling operations, you can clearly know which pod will be scaled, starting with the largest number. And Stat fulset is not allowed to perform shrinking operations when there are unhealthy instances.

k8s deploys MySQL

  • You can use kubesphere to quickly build a MySQL environment

    • Stateful service extraction configuration as ConfigMap

    • Stateful services must use pvc to persist data

    • Access to the service cluster uses the stable domain name provided by DNS

Create MySQL master-slave service

  • Step 1: Create storage volume pvc and configure configMap

    The content of configMap is my.cnf that was started by docker before

    #mysql master node [client] default-character-set=utf8 [mysql] default-character-set=utf8 [mysqld] init_connect='SET collation_connection=utf8_unicode_ci' init_connect='SET NAMES utf8' character-set-server=utf8 collation-server=utf8_unicode_ci skip-character-set-client-handshake skip-name-resolve server-id=1 log-bin=mysql-bin read-only=0 binlog-do-db=touch-air-mall-ums binlog-do-db=touch-air-mall-pms binlog-do-db=touch-air-mall-oms binlog-do-db=touch-air-mall-sms binlog-do-db=touch-air-mall-wms binlog-do-db=demo_ds_0 binlog-do-db=demo_ds_1 replicate-ignore-db=mysql replicate-ignore-db=sys replicate-ignore-db=information_schema replicate-ignore-db=performance_schema Copy code
  • Step 2: Create a stateful service

    • Set copy
    • Pull mysql mirror, modify memory in advanced settings, use default port
    • Set environment variables
    • Mount data, mount configuration

  • Step 3: Configure master and slave

    • Enter the master container, connect to mysql, authorize the account

      mysql -uroot -p GRANT REPLICATION SLAVE ON *.* TO'backup'@'%' IDENTIFIED BY '123456'; Copy code

    • Enter the slaver container, configure synchronization data

      # host uses the master's domain name (DNS) CHANGE MASTER TO MASTER_HOST='mysql-master.touch-air-mall', MASTER_USER='backup', MASTER_PASSWORD='123456', MASTER_LOG_FILE='mysql-bin.000005',MASTER_LOG_POS=439,master_port=3306; # Open from node start slave; # View slave status show slave status\G; Copy code

Test master-slave configuration

  • Create a database in the main library master

    CREATE DATABASE demo_ds_0; copy code
  • Create table

    SET NAMES utf8mb4; SET FOREIGN_KEY_CHECKS = 0 ; - ---------------------------- - Table structure for sys_user - ------------ ---------------- the DROP TABLE the IF EXISTS `user` ; the CREATE TABLE ` user` ( `id` VARCHAR ( 255 ) the CHARACTER the SET utf8mb4 the COLLATE utf8mb4_general_ci the NOT NULL the COMMENT 'master key' , ` been_deleted` int ( . 11 ) NULL the DEFAULT 0 the COMMENT 'represents a logic 1 logic 0 indicates deleted deleted deleted logical' , `create_by` VARCHAR( 255) The CHARACTER the SET utf8mb4 the COLLATE utf8mb4_general_ci NULL the DEFAULT NULL the COMMENT 'Creator' , `created` datetime ( 0 ) NULL the DEFAULT NULL the COMMENT 'Created' , ` remark` VARCHAR ( 255 ) the CHARACTER the SET utf8mb4 the COLLATE utf8mb4_general_ci NULL the DEFAULT NULL the COMMENT 'Remarks ' , `update_by` varchar ( 255 ) CHARACTER SETutf8mb4 the COLLATE utf8mb4_general_ci NULL the DEFAULT NULL the COMMENT 'updater' , `updated` datetime ( 0 ) NULL the DEFAULT NULL the COMMENT 'modified' , ` version` int ( . 11 ) NULL the DEFAULT NULL the COMMENT 'version number' , `last_logged_in` datetime ( 0 ) NULL the DEFAULT NULL the COMMENT 'last' , `nick_name` VARCHAR (255 ) CHARACTER the SET utf8mb4 the COLLATE utf8mb4_general_ci NULL the DEFAULT NULL the COMMENT 'Nickname' , `password` VARCHAR ( 255 ) the CHARACTER the SET utf8mb4 the COLLATE utf8mb4_general_ci NULL the DEFAULT NULL the COMMENT 'password' , ` phone` VARCHAR ( 255 ) the CHARACTER the SET utf8mb4 the COLLATE utf8mb4_general_ci NULL the DEFAULT NULL the COMMENT 'Mobile phone number' , first_sign_in` ` int ( . 11 ) NULL the DEFAULT NULL the COMMENT 'first log: 0 indicates unregistered, 1 represents first log, log 2 represents a non-first' , ` username` VARCHAR ( 255 ) the CHARACTER the SET utf8mb4 the COLLATE utf8mb4_general_ci NULL the DEFAULT NULL the COMMENT 'account ' , `email` VARCHAR ( 255 ) the CHARACTER the SET utf8mb4 the COLLATE utf8mb4_general_ci NULL the DEFAULT NULL the COMMENT ' mailbox ' , ` sys_dept_id` VARCHAR ( 255 )the CHARACTER the SET utf8mb4 the COLLATE utf8mb4_general_ci NULL the DEFAULT NULL the COMMENT 'sector ID' , `sys_dept_name` VARCHAR ( 255 ) the CHARACTER the SET utf8mb4 the COLLATE utf8mb4_general_ci NULL the DEFAULT NULL the COMMENT 'department name' , PRIMARY KEY ( `id` ) USING BTREE, INDEX `idx_username_password` ( `username` , `password` ) USING BTREE ) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci COMMENT = 'System user table' ROW_FORMAT = Dynamic; SET FOREIGN_KEY_CHECKS = 1 ; Copy code
  • Insert data

    INSERT INTO `demo_ds_0` . `user` ( `id` , `been_deleted` , `create_by` , `created` , `remark` , `update_by` , `updated` , `version` , `last_logged_in` , `nick_name` , `password` , `phone` , `first_sign_in` , `username` , `email` , `sys_dept_id` , `sys_dept_name` ) VALUES ( '402881f773b383ae0173b383d019001b' ,0 , NULL , '2020-08-03 16:50:27' , NULL , NULL , '2021-03-23 15:01:08' , 11007 , '2021-03-23 15:01:08' , ' administrators' , '$ $ 10 $ H57iIdDN5QigR7QOxjFtceRA1l4MSjPSJSm3t3AtyW9RaoIuc4m5y. 2A' , NULL , 2 , 'ADMIN' , NULL , NULL , NULL ); duplicated code

  • Confirm whether the slave library is synchronized

k8s deployment summary (*)

  • 1. Every MySQL and Redis must be a stateful service
  • 2. Each MySQL and Redis must mount its own configuration file (ConfigMap) and storage volume (PVC)
  • 3. When the docker mode is started, the IP address of the master in the configuration can be changed to the DNS domain name.

Due to the limited memory of the test machine, the following services are not started in the form of a cluster, but the steps are the same as the MySQL master-slave.

k8s deploy Redis

  • Step 1: Create redis configuration file configMap

    redis-conf: appendonly yes copy the code
  • Step 2: Create storage volume pvc

  • Step 3: Create a stateful service, pull redis image, set memory, mount data, mount configuration file

k8s deploys ElasticSearch&Kibana

Deploy ElasticSearch

  • Step 1: Create es configuration file configMap

  • Step 2: Create storage volume pvc for es

  • Step 3: Create a stateful service, pull the es mirror...

    • Test whether ES is created successfully

      • 1. Use the admin account to log in and test via the console
      • 2. Use the previously installed wordpress test
      #curl Access port 9200 of es, use domain name in k8s for access curl mall-es.touch-air-mall:9200 Copy code

Deploy kibana

  • kibana is a visualization service of ElasticSearch, there is no need to mount data and configuration, so only need to start the stateless service

    • Add environment variables

      SERVER_HOST ELASTICSEARCH_HOSTS http://mall-es.touch-air-mall:9200 Copy code

  • Browser access exposed port code

k8s deploys RabbitMQ

  • Same as above

  • The first step: create mq storage volume pvc

  • Create stateful services, pull images, mount storage volumes

k8s deploy Nacos

Two deployment methods for stateless services

  • 1. Create a stateless service directly

  • 2. Stateful service, delete the service without deleting the working copy, and then create the specified workload

  • Browser access test code

k8s deploy Zipkin

  • Create a stateless service

  • Access test code

k8s deploys Sentinel

  • Use open source mirroring

    bladex/sentinel-dashboard: 1.6.3 copy the code
  • Create a stateless service

  • Access test code

Create a service for the specified workload

  • Scenario: There is already a stateless sentinel service, and I want to create another stateful sentinel service

k8s deploys microservices

Deployment process

  • Deployment flowchart

  • Steps

    • Step 1: Prepare one for each project
      Follow this
      Make the project into a mirror
    • Step 2: Generate k8s deployment description file for each project
    • Connect the above operations in series: write it

Production environment configuration extraction

  • Based on the workload of NodePort, create services that can be accessed within the cluster

    mall-nacos-service mall-sentinel-service mall-zipkin-service Copy code

  • Add production environment configuration file

    Switch the host and port number information of all services previously debugged locally to a custom domain name in the cluster

    spring.rabbitmq.host=mall-rabbitmq.touch-air-mall spring.datasource.url=jdbc:mysql://mysql-master.touch-air-mall:3306/touch_air_mall_sms?serverTimezone=Asia/Shanghai spring.cloud.nacos.discovery.server-addr=mall-nacos-service.touch-air-mall:8848 spring.cloud.sentinel.transport.dashboard=mall-sentinel-service:8858 spring.zipkin.base-url=http://mall-zipkin.touch-air-mall:9411/ spring.redis.host=mall-redis.touch-air-mall Copy code

Make project mirror

  • renren-fast

    • 1. Generate


      mvn clean package -Dmaven.test.skip = true copy the code

    • 2. Will

      Package and
      Copy into the server where the docker service is located

      docker build -f Dockerfile -t docker.io/touch/admin:v1.0. Copy the code

Create Dockerfile

  • Create new in the root directory of each microservice


    FROM java:8 #8080 exposed in the container, all microservices are the same and do not affect each other EXPOSE 8080 VOLUME/tmp ADD target/*.jar/app.jar RUN bash -c'touch/app.jar' ENTRYPOINT ["java","-jar","/app.jar","--spring.profiles.active=prod"] Copy code

Create a microservice k8s deployment description file

  • reference document

  • deploy

    Description file

    kind: Deployment apiVersion: apps/v1 metadata: name: mall-auth-server namespace: touch-air-mall labels: app: mall-auth-server spec: replicas: 1 selector: matchLabels: app: mall-auth-server template : the Metadata: Labels: App: Mall-auth-Server spec: Containers: - name: Mall-auth-Server Image: $ REGISTRY/$ DOCKERHUB_NAMESPACE/$ APP_NAME: Latest the ports: - name: tcp-Mall-auth-Server containerPort: 8080 protocol: TCP resources: limits: cpu: 500m memory: 512Mi requests: cpu: 10m memory: 10Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File imagePullPolicy: IfNotPresent restartPolicy: Always terminationGracePeriodSeconds: 30 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 25 % maxSurge: 25 % revisionHistoryLimit: 10 progressDeadlineSeconds: 600 --- kind: Service apiVersion: v1 metadata: name: mall-auth-server namespace: touch-air-mall labels: app: mall-auth-server annotations: kubesphere.io/alias-name: authentication service kubesphere.io/serviceType: statelessservice spec: the ports: - name: HTTP Protocol: TCP Port: 8080 TARGETPORT: 8080 nodePort: 20001 Selector: App: Mall-auth-Server of the type: NodePort sessionAffinity: None copy the code

  • port
    : [ClusterIP:port], service is exposed in clusterIP

  • targerPort
    : [Container mapping port] is equivalent to the pod (Expose in Dockerfile)

  • nodePort
    [NodeIP:nodePort] provides an entrance for external traffic to access the service in the k8s cluster

apart from

Unavailable repeat,
Can be repeated

assembly line

  • Write Jenkinsfile

    • Step 1: Pull the code

    • Step 2: Parameterized construction & environment variables

    • The third step: sonar code quality analysis

    • Step 4: build & push (snapshot, latest version)

    • Step 5: Deploy to the dev environment

    • Step 6: TAG and release the version

    pipeline { agent { node { label'maven' } } environment { DOCKER_CREDENTIAL_ID ='dockerhub-id' GITHUB_CREDENTIAL_ID ='gitee-id' KUBECONFIG_CREDENTIAL_ID ='kubeconfig-id' REGISTRY ='docker.io' DOCKERHUB_NAMESPACE ='wlfctothemoon' GITHUB_ACCOUNT ='OK12138' SONAR_CREDENTIAL_ID='sonar-qube' BRANCH_NAME='main' } stages { stage('Pull code') { steps { git(url:'https://gitee.com/OK12138/touch-air-mall.git', credentialsId:'gitee-id', branch:'main', changelog: true, poll: false) sh'echo is building the $PROJECT_NAME version number: $PROJECT_VERSION will be submitted to the $REGISTRY mirror warehouse' container ('maven') { sh "mvn clean install -Dmaven.test.skip=true -gs `pwd`/mvn-settings.xml" } } } stage('sonar code quality analysis') { steps { container ('maven') { withCredentials([string(credentialsId: "$SONAR_CREDENTIAL_ID", variable:'SONAR_TOKEN')]) { withSonarQubeEnv('sonar') { sh "echo current directory `pwd`" sh "mvn sonar:sonar -gs `pwd`/mvn-settings.xml -Dsonar.login=$SONAR_TOKEN" } } } } } stage ('build & push build image and push') { steps { container ('maven') { sh'mvn -Dmaven.test.skip=true -gs `pwd`/mvn-settings.xml clean package' sh'cd $PROJECT_NAME && docker build -f Dockerfile -t $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER .' withCredentials([usernamePassword(passwordVariable:'DOCKER_PASSWORD' ,usernameVariable:'DOCKER_USERNAME' ,credentialsId: "$DOCKER_CREDENTIAL_ID" ,)]) { sh'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin' sh'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:latest ' sh'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:latest ' } } } } stage('Deploy to k8s') { steps { input(id: "deploy-to-dev-$PROJECT_NAME", message: "Do you want to deploy $PROJECT_NAME to the cluster?") kubernetesDeploy(configs: "$PROJECT_NAME/deploy/**", enableConfigSubstitution: true, kubeconfigId: "$KUBECONFIG_CREDENTIAL_ID") } } stage('push with tag TAG, release version'){ when{ expression{ return params.PROJECT_VERSION =~/v.*/ } } steps { container ('maven') { input(id:'release-image-with-tag', message:'Is the current version image released?') withCredentials([usernamePassword(credentialsId: "$GITHUB_CREDENTIAL_ID", passwordVariable:'GIT_PASSWORD', usernameVariable:'GIT_USERNAME')]) { sh'git config --global user.email "kubesphere@yunify.com" ' sh'git config --global user.name "kubesphere" ' sh'git tag -a $PROJECT_VERSION -m "$PROJECT_VERSION" ' sh'git push http://$GIT_USERNAME:$GIT_PASSWORD@gitee.com/$GITHUB_ACCOUNT/touch-air-mall.git --tags --ipv4' } sh'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:$PROJECT_VERSION ' sh'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:$PROJECT_VERSION ' } } } } } Copy code

Migrate the database

  • The MySQL master and slave nodes started before are all stateful services for access within the cluster, and ports are not exposed

    You can create a stateless MySQL service with a specified workload, and NodePort exposes access

    After the copy is completed, delete the exposure to ensure the security of the data in the cluster

Deployment effect

  • flow chart

Porting Nginx (Static resources cannot be accessed online)

  • Use Dockerfile to generate a custom nginx image

    • Prepare custom Dockerfile and files

    • Use Dockerfile to build an image

      docker build -t touch-air/mall-nginx:v1.2 .Copy code

  • It can be combined with the Alibaba Cloud image warehouse to deploy the local (docker image of the virtual machine) to the k8s cluster

    Push the image to dockerhub (Alibaba Cloud Image Warehouse)

    #Mark Mirror docker tag local-image:tag username/new-repo:tag #Upload mirror docker push username/new-repo:tag Copy code
  • Modify the upstream server of nginx to ensure that it is routed to the gateway in the k8s cluster

    • Back up the original

      , Then modify

Integrate Alibaba Cloud image warehouse
  • Log in to Alibaba Cloud Image Container Service, open and create a personal image warehouse

  • Push the image to Alibaba Cloud

    • 1. Log in to Alibaba Cloud, the password is the password used when opening the mirror warehouse

      docker login --username = xxx registry.cn-hangzhou.aliyuncs.com duplicated code
    • 2. Push mirror

      #Mark Mirror docker tag [ImageId] registry.cn-hangzhou.aliyuncs.com/tothemoon/mall-nginx:[Image version number] #Push docker push registry.cn-hangzhou.aliyuncs.com/tothemoon/mall-nginx:[Mirror version number] Copy code

Jenkins modifies Alibaba Cloud image warehouse
  • Modify warehouse voucher and warehouse address

Deploy all microservices in a pipeline

  • To

    Gateway service as an example

  • The above parameterized construction is very important, so that we deploy services, we can select one or more services we want to deploy

    When deploying different services, only need to modify

    Change to the name of the project you want to deploy (
    mall-gateway, mall-cart, mall-seckill

  • Run the pipeline repeatedly to build all mall microservices

    • The final effect is that the browser accesses the exposed port, and the standard appears

      The 404 interface of the project is enough; after the subsequent nginx service deployment, you can view the overall mall service

    • The nacos service registry in the k8s cluster

    • Sentinel traffic monitoring in k8s cluster

  • Modify the microservice permissions of Alibaba Cloud Warehouse to be public to facilitate k8s cluster access

  • Interface test

    # Copy code

Final deployment

Deploy nginx before

  • Use the nginx image that we previously packaged and pushed to Alibaba Cloud to deploy the nginx service in the k8s cluster

    • Create a personal warehouse address

    • Choose a personal nginx mirror on Alibaba Cloud

    • Test visit

    • The current access method using ip address + exposed port number obviously cannot meet our needs

Create gateway and application routing

Internet access gateway
  • Before creating an application route, you need to enable the external network access portal, that is, the gateway. This step is to create the corresponding application routing controller, which is responsible for forwarding the request to the corresponding back-end service

    Note: Since the use of Load Balancer requires the configuration and installation of the cloud-controller-manage plug-in that connects with the cloud service provider before installation, refer to Installing the Load Balancer Plug-in to install and use the load balancer plug-in (not shown for the time being)

  • As the project administrator, turn on application routing

    • create


Application routing
  • Create application route

Deploy the Vue project

  • Step 1: Modify the online gateway environment


    Api request address in

  • Step 2: Compile and package, generate dist directory

    npm run build copy the code

  • Step 3: Write Dockerfile

    FROM nginx MAINTAINER leifengyang ADD dist.tar.gz/usr/share/nginx/html EXPOSE 80 ENTRYPOINT nginx -g "daemon off;" Copy code
  • Step 4: Package into an image

    docker build -t touch-air-mall -vue-app:. v1.0 -f Dockerfile duplicated code

  • Upload the Alibaba Cloud image warehouse, kubesphere creates a stateless service

    • Upload

      #log in docker login --username=xxx registry.cn-hangzhou.aliyuncs.com #Mark Mirror docker tag [ImageId] registry.cn-hangzhou.aliyuncs.com/tothemoon/mall-nginx:[Image version number] #Push docker push registry.cn-hangzhou.aliyuncs.com/tothemoon/mall-nginx:[Mirror version number] Copy code

    • Create a stateless service