Skip to content

Union Manifest GitOps Solution

Union manifests are maintained using a GitOps solution now. We can deploy manifest as kubernetes custom resources and these are processed by a kubernetes custom operator that resides inside UnionAPI.

Custom Resource Definitions

We are using two types of custom resource definitions (CRDs) for this solution. The CRDs are packed inside the UnionAPI helm chart.

Product CRD

Product CRD is used to manage each product inside union. Each product can have one or more product environments.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: products.platformservices.spglobal.com
spec:
  group: platformservices.spglobal.com
  versions:
    - name: v1alpha1
      additionalPrinterColumns:
        - jsonPath: .spec.id
          name: Product Id
          type: string
        - jsonPath: .spec.key
          name: Product Key
          type: string
        - jsonPath: .spec.name
          name: Product Name
          type: string
        - jsonPath: .spec.color
          name: Color
          type: string
        - jsonPath: .spec.icon
          name: Icon
          type: string
      storage: true
      served: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                id:
                  type: string
                key:
                  type: string
                name:
                  type: string
                color:
                  type: string
                icon:
                  type: string
                isUnionSubHeader:
                  type: boolean
              required:
                - id
                - key
                - icon
  names:
    kind: Product
    singular: product
    plural: products
    shortNames:
    - prd
  scope: Namespaced

Product Environment CRD

Product Environment is used to manage each product environment inside union. Each product can have one or more product environments.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: productenvs.platformservices.spglobal.com
spec:
  group: platformservices.spglobal.com
  versions:
    - name: v1alpha1
      additionalPrinterColumns:
        - jsonPath: .spec.id
          name: Product Env Id
          type: string
        - jsonPath: .spec.productId
          name: Product Id
          type: string
        - jsonPath: .spec.name
          name: Env Name
          type: string 
        - jsonPath: .spec.appRootUrl
          name: App Root Url
          type: string 
        - jsonPath: .spec.apiEndpoints.routes
          name: Api Endpoints - Routes
          type: string
        - jsonPath: .spec.apiEndpoints.menu
          name: Api Endpoints - Menu
          type: string
      storage: true
      served: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                id: 
                  type: string
                productId:
                  type: string
                name: 
                  type: string
                isRestricted:
                  type: boolean
                appRootUrl:
                  type: string
                apiRootUrl:
                  type: string
                supportUrl:
                  type: string
                isInBeta:
                  type: boolean
                endBetaRedirectUrl:
                  type: string
                apiEndpoints:
                  type: object
                  properties:
                    config:
                      type: string
                    routes:
                      type: string
                    menu:
                      type: string   
                    widgets:
                      type: string  
                    feedback:
                      type: string
                    messages:
                      type: string
                    permissions:
                      type: string
                    logout:
                      type: string
                    logoutMethod:
                      type: string
                    updateToken:
                      type: string
                    version:
                      type: object
                      properties:
                        ui:
                          type: string
                        api:
                          type: string
                      required:
                        - ui
                  required:
                    - routes
                    - menu
                customHeaders:
                  type: object
                  properties:
                    clientId:
                      type: string
              required:
                - id
                - productId
                - name
                - appRootUrl
                - apiEndpoints
  names:
    kind: ProductEnv
    singular: productenv
    plural: productenvs
    shortNames:
    - prdenv    
  scope: Namespaced

Custom Manifest Resources

The Custom Manifest Resources are deployed for each union environment using flux and kustomization.

We currently have:

Custom Resource Examples

Product Resource

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
apiVersion: "platformservices.spglobal.com/v1alpha1"
kind: "Product"
metadata:
  name: "compliance"
  namespace: "uniondev"
spec:
  id: "compliance-b7c094ea-2ce2-4225-8d98-1ecce86b39e0"
  key: "compliance"
  name: "Compliance"
  color: "#569fa6"
  icon: "Compliance"

Product Environment Resource

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
apiVersion: "platformservices.spglobal.com/v1alpha1"
kind: "ProductEnv"
metadata:
  name: "compliance-dev"
  namespace: "uniondev"
spec:
  id: "compliance-dev-e40421a7-f1f2-4168-9a6e-9723f571c456"
  productId: "compliance-b7c094ea-2ce2-4225-8d98-1ecce86b39e0"
  name: "Dev"
  isRestricted: false
  appRootUrl: "https://compliance.dev.finapps.ihsmarkit.com"
  apiRootUrl: "https://api-compliance.dev.finapps.ihsmarkit.com"
  supportUrl: "https://supportlibrary.ihsmarkit.com/pages/viewpage.action?spaceKey=IMC&title=IHS+Markit+Compliance+for+CLO+Managers+Home"
  isInBeta: false
  apiEndpoints:
    routes: "https://compliance.dev.finapps.ihsmarkit.com/api/union/routes.json"
    menu: "https://compliance.dev.finapps.ihsmarkit.com/api/union/menu.json"
    permissions: "compliance/ui/permissions"
    logout: "compliance/ui/logout"
    logoutMethod: "GET"
    version:
      ui: "https://compliance.dev.finapps.ihsmarkit.com/assets/app.json"
      api: "https://api-compliance.dev.finapps.ihsmarkit.com/compliance/version"
  customHeaders:
    clientId: "system"

More examples can be found in Union DEV environment custom resources directory.

Manifest Custom Operator

The Manifest Custom Operator resides in UnionAPI. It gets packed and deployed along with UnionAPI. There is no separate CI/CD for this.

Manifest Custom Operator Configuration

The custom operator can be configured using config as specified below. It can be passed to the .NET app using appsettings.json or env variables. The env variables has higher precedence order. The configuration can also be overridden in helm release files.

1
2
3
4
5
6
"CustomManifestOperatorConfig": {
  "Enabled": true,
  "Group": "platformservices.spglobal.com",
  "Version": "v1alpha1",
  "Namespace": "uniondev"
}
  • Enabled: Operator is enabled when set to true
  • Group: The CRD group, can be set in CRDs if required.
  • Version: CRD version.
  • Namespace: Kubernetes namespace where custom resource are deployed.

Operator will use Group, Version and Namespace to locate the resources.

Integration

Union custom operator

Custom resource definations for manifest resources are build inside UnionAPI helm chart and are deployed inside the kubernetes environment using the same.

Once the CRDs are deployed, the environment is ready to accept the custom resources. These are deployed using flux/kustomization (GitOps approach).

The custom operator inside UnionAPI watches for the respective custom resources. As soon as the custom resources are available, it can read and cache these inside an in-memory storage. The watchers receives events for creation, updates and deletions. All subsequent changes to the custom resources will be processed and in-memory storage will be updated. The operator also takes care of any custom resources those were already present before watchers started running. This means, upon deployment of a newer version of UnionAPI or in an event of pod restarts, all the pre-deployed custom resources will be read and stored inside the in-memory storage.

UnionAPI exposes the manifest data via an API endpoint (GET /manifest). UnionUI uses this endpoint to get the manifest and renders UI. Whether or not use this endpoint can be controller using a flag in union UI.

1
2
3
4
5
{
  // ...
  "manifestFromApi": true,
  // ...
}

This flag can be set separately for each union ui environment. It resides inside the union-config files.

Here is an example of the flag for union DEV environment.

When flag is set to true, union ui will get manifest from the API, otherwise fallback to the older way of using manifest.json file.

How a product is matched to all of it’s environments?

As we deploy products and corresponding environments using separate custom resource (yaml) files, we need a mechanism to match a product to all of it’s environments.

The custom manifest operator uses the productId to match the product to all of it’s environments.

For a product resource it’s an id under the spec:

1
2
spec:
  id: "compliance-b7c094ea-2ce2-4225-8d98-1ecce86b39e0"

and then for all the corresponding product environments, we must set the same key as productId under the spec:

1
2
spec:
  productId: "compliance-b7c094ea-2ce2-4225-8d98-1ecce86b39e0"

product environments also have an id field. This should be set to a unique value for each environment. 

1
2
3
spec:
  id: "compliance-dev-e40421a7-f1f2-4168-9a6e-9723f571c456"
  productId: "compliance-b7c094ea-2ce2-4225-8d98-1ecce86b39e0"

All id fields are type string. A simple way to set these correctly is to use the as same as name under metadata like this:

1
2
3
4
5
6
7
# ...
metadata:
  name: "compliance-product"
  namespace: "uniondev"
spec:
  id: "compliance-product"
# ...

and then for the product env:

1
2
3
4
5
6
7
8
# ...
metadata:
  name: "compliance-product-env-dev"
  namespace: "uniondev"
spec:
  id: "compliance-product-env-dev"
  productId: "compliance-product"
# ...

C# code adding envs to products

1
allProductsEnvironments.Where(e => e.ProductId == product.Id)