Vstore

Informations

Supported operators for filter action type:

All SQL Data Types:
- numeric types, string types, date & time types and JSON...
All SQL Operators.

Policies Enforcement:

Based on tags and policies SQL Filter predicate will be build and pushdown to the underlying storage.

Data Masking policies will be process at VStore level in the data return processing phase.

Supported storages types

Vstore support currently the following storage for its namespaces:

TiDB
PostgreSQL
Clickhouse
- This storage specifically does not allow updating data once it has been written

Configurations

Configuration file

server:
  port: 9080
  addr: 0.0.0.0
  ## Allow models and namespaces loading
  ## from S3 buckets
  # resources:
  #   endpoint: rocky0:32082
  #   user: minioadmin
  #   password: minioadmin
  #   # # bucket where the models are stored
  #   # modelsPath: models
  #   # bucket where the namespaces are store
  #   namespacesPath: namespaces
  #   # bucket where the policies are stored
  #   policiesPath: policies
  maxReturnedEntities: 1000
  # poolSize: 10
  # # Optional, default algoritm is HS256
  # jwtAlgorithm: RS256
  # # Optional (if RS256)
  # # Optional (if RS256)
  # # This config and the publicKey/privateKey are mutually exclusive, with priority given to jwksUri
  # jwksUri: http://keycloak-cluster-service.kosmos-iam.svc.cluster.local./realms/kosmos/
  # # Optional (if RS256)
  # # This config & privateKey are mutually exclusive with jwksUri, with priority given to jwksUri
  # publicKey: |-
  #   -----BEGIN PUBLIC KEY-----
  #   xxx
  #   -----END PUBLIC KEY-----
  # # Optional (if RS256)
  # # This config & publicKey are mutually exclusive with jwksUri, with priority given to jwksUri
  # privateKey: |-
  #   -----BEGIN PRIVATE KEY-----
  #   xxx
  #   -----END PRIVATE KEY-----
store:
  # use only one namespace if you don't know how to use it
  # namespace must be unique
  - namespace: test
    user: root
    password: 8k+m3bdVj5^*u@791X
    addr: 192.168.100.129:4000

    ## Postgres store exemple
    # user: postgres
    # password: postgres
    # addr: 192.168.100.129:5432
    # impl: postgresql

    ## Clickhouse store exemple
    #   user: testuser
    #   password: testpassword
    #   addr: 127.0.0.1:9004
    #   impl: clickhouse

    ## maximum pool size used when connecting to the store
    # poolSize: 10

Define a Namespace/Store

A namespace/store requires :

a database name
the necessary information to reach the store in which the database is backed (address & credentials)
the store type when not using the default one

Usage

server:
  port: 9080
  addr: 0.0.0.0
  ## Allow models and namespaces loading from S3 buckets
  # resources:
  #   # s3 endpoint
  #   endpoint: rocky0:32082
  #   # s3 credentials
  #   user: minioadmin
  #   password: minioadmin
  #   # bucket where the models are stored
  #   modelsPath: models
  #   # bucket where the namespaces are store
  #   namespacesPath: namespaces
  #   # bucket where the policies are stored
  #   policiesPath: policies
  ## Default max number of entities returned by queries
  maxReturnedEntities: 1000
  # # Optional JWT algorithm, default algoritm is HS256
  # jwtAlgorithm: RS256
  # # Optional (if RS256)
  # # This config and the publicKey/privateKey are mutually exclusive, with priority given to jwksUri
  # jwksUri: http://keycloak-cluster-service.kosmos-iam.svc.cluster.local./realms/kosmos/
  # # Optional (if RS256)
  # # This config & privateKey are mutually exclusive with jwksUri, with priority given to jwksUri
  # publicKey: |-
  #   -----BEGIN PUBLIC KEY-----
  #   xxx
  #   -----END PUBLIC KEY-----
  # # Optional (if RS256)
  # # This config & publicKey are mutually exclusive with jwksUri, with priority given to jwksUri
  # privateKey: |-
  #   -----BEGIN PRIVATE KEY-----
  #   xxx
  #   -----END PRIVATE KEY-----
store:
  # use only one namespace if you don't know how to use it
  # namespace must be unique

  # default (TiDB) store exemple
  - namespace: test
    user: root
    password: 8k+m3bdVj5^*u@791X
    addr: 192.168.100.129:4000

    ## Postgres store exemple
    # user: postgres
    # password: postgres
    # addr: 192.168.100.129:5432
    # impl: postgresql

    ## Clickhouse store exemple
    #   user: testuser
    #   password: testpassword
    #   addr: 127.0.0.1:9004
    #   impl: clickhouse

      ## maximum pool size used when connecting to the store
    # poolSize: 10

In the config file

server:
    # Bind port
    port: 9080
    # Bind address
    addr: 0.0.0.0
    # limit to the number of entity returned by a search request without override.
    maxReturnedEntities: 1000
store:
    # use only one namespace if you don't know how to use it
    # namespace must be unique
    - namespace: test
      # Information to reach the store
      user: root
      password: 8k+m3bdVj5^*u@791X
      addr: 192.168.100.129:4000
      # Store type, if not provided, default to 'tidb'
      impl: tidb

      ## Postgres exemple
      # user: postgres
      # password: postgres
      # addr: 192.168.100.129:5432
      # impl: postgresql

      ## Clickhouse store exemple
      #   user: testuser
      #   password: testpassword
      #   addr: 127.0.0.1:9004
      #   impl: clickhouse

      ## maximum pool size used when connecting to the store
      # poolSize: 10

From a s3 bucket

Alternatively you can store your namespaces files on a s3 server (only the store part will be taken into account). Multiple files are allowed.

Add your config (same as above) in your s3 bucket: mybucketns

Set the resources part in your config file.

server:
    port: 9080
    addr: 0.0.0.0
    resources:
        endpoint: kube0:31318
        user: minioadmin
        password: minioadmin
        # bucket where the model is store
        namespacesPath: mybucketns

Define the Policy

A policy is a set of rules that applies to a certain subsets of resources in a namespace. These rules may filter out or mask (hiding the values) from a query result. This is a namespaced resource.

Usage

Define policies and tag entities:

Sample policy:

@data refer to data and @identity refer to entity tags.

id: intel_policy
namespace: test
actions:
  # filter out false predicate
  # true if data cls field <= identity tag 'icls' value
  - type: filter
    predicate: "@data.cls <= @identity.tags.cls"

  # true if data stmts json fields fully contained in the json value of identity tag 'cls' value
  - type: filter
    predicate: "@data.stmts IN @identity.tags.stmts"

  # mask attributes if predicate false
  # exclude 'exclude' fields from the masking process
  - type: masking
    # If set will mask the data with the provided value, else null
    # mask: M
    predicate: "@data.thematics IS NULL OR @identity.tags.thematics INTERSECT @data.thematics"
    # field to exclude from the masking process (-> will not be masked)
    exclude:
      - id
      - age

In the config file

---
id: intel_policy
namespace: test
# # will only apply to the provided entity if defined
# entitiesScope: [person]
# # will be exclude for the provided entity if defined
# entitiesExclude: [equipment]
actions:
  # # filter out false predicate
  # # true if data cls field <= identity tag 'cls' value
  # - type: filter
  #   predicate: "@data.cls <= @identity.tags.cls"
  # true if data stmts json fields fully contained in the json value of identity tag 'cls' value
  - type: filter
    predicate: "@data.stmts IN @identity.tags.stmts"
  # # true if data 'thematics' json has a non null intersection with the 'thematics value of the macthing identity'
  # # eq: at least one element of identity thematics is inside thematics of data or thematics of data is null
  # - type: filter
  #   predicate: "@data.thematics IS NULL OR @identity.tags.thematics INTERSECT @data.thematics"
  # mask attributes if predicate false
  # exclude 'exclude' fields from the masking process
  - type: masking
    # If set will mask the data with the provided value, else null
    mask: M
    predicate: "@data.thematics IS NULL OR @identity.tags.thematics INTERSECT @data.thematics"
    # field to exclude from the masking process (-> will not be masked)
    exclude:
      - id
      - age

From a s3 bucket

Alternatively you can store your policies files on a s3 server. Multiple files are allowed.

Add your policy (same as above) in your s3 bucket: mybucketpol

Set the resources part in your config file.

server:
  port: 9080
  addr: 0.0.0.0
  resources:
    endpoint: kube0:31318
    user: minioadmin
    password: minioadmin
    # bucket where the model is store
    policiesPath: mybucketpol
...

Define the identities

An identity is a user representation :

with a unique pair of id and tenantId
with a set of security tags that will be used in concordance with policies to define which query result they are able to see unfiltered/unmasked.

A user may be set as admin to avoid all policies by setting the profile key as admin.

Usage

Note that identities can be provided by configuration or dynamically see Dynamic Identities

id: eric
tags:
  - key: cls
    value: SECRET
  - key: stmts
    value: '{"SF":1,"ACSSI":1,"ECLAIR":1}'
  - key: thematics
    value: "NOTIN,PROLI"

In the config file

---
id: eric
tenantId: kosmos
tags:
  - key: cls
    value: TOP_SECRET
  - key: stmts
    value: '{"SF":1,"ACSSI":1,"ECLAIR":1}'
  - key: thematics
    value: "NOTIN,PROLI"
# profile: admin

Dynamic Identities

It is possible to provide identities dynamically from another vstore. To do so you will need to configure the identities part:

server:
  port: 9080
  addr: 0.0.0.0
  maxReturnedEntities: 1000
  [...]
  identities:
    # host to reach (can be another vstore)
    host: 127.0.0.1
    uri: /v0/test/entities/identity
    conditionalFilter: tenant_id = 'athea'
    # polling delay (default 3 sec)
    # delayMs: 5000
    # bearer token to use (user will need to have full access to all identities, admin, etc...)
    #pageSize: 5000 # number of user to return per request (if you have more users, multiple request will be made with pagination)
    token: |-
      eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJlcmljIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiZXJpYyIsInRlbmFudF9pZCI6ImF0aGVhIiwiaWF0IjoxNzI2NjQ4MDY3fQ.qrXHKy2pfXvCkeVPtdQXPfiX1dmjaytV8x9AxdRc6xU
  ...

The conditional filter will be added to the usual request and allow you to choose which user to retrieve from the identities source.

For the vstore owner of entities you will need the following model:

entities:
  - id: identity
    namespace: test
    primaryKeys:
      - id
    fields:
      - name: id
        type: VARCHAR(64)
      # The 'tags' should be a json array which contains the tags found in your identity in Json objet with the keys 'key' & 'value'
      # eg: if you only have a tag cls which is of type "ENUM('NP', 'DR', 'SECRET', 'TOP_SECRET')"
      #     an example of tags would be '[{"key":"cls","value":"TOP_SECRET"}]'
      - name: tags
        type: VARCHAR(256)
      # set on insert and update with millisec timestamp
      # for now this will be used to detect modification in the polling phase.
      - name: ts
        type: BIGINT
      - name: tenantId
        type: VARCHAR(64)

Define a model

A model is a series of entities, each of which has fields. This is a namespaced resource.

In the config file

---
version: 1
entities:
  - id: person

    # The namespace the entity belong to
    # use only one for all your entities if you don't know how to use it
    namespace: test

    # # will clear entity (use it only if you know what you are doing)
    # markForDeletion: true

    # Activate entity modification history
    history: true

    # # Optional list of mutations to apply to an existing entity to match itw new model
    # # This field is used to provide hints on how the previous model should be modified
    # # to match the new one.
    # # For more information see the 'Mutate a model' section
    mutations:
      add:
          indexes: []
          fields: []
      delete:
          indexes: []
          fields: []

    # optional indexes list on field 'testidx'
    # use only for high cardinality values and frequents queries on it.
    indexes:
      - name: testidx
        expression: (testidx)
    # for field UNIQUE constraint add UNIQUE keyword at the end of the type
    # eq: VARCHAR(128) -> VARCHAR(128) UNIQUE
    # Note that its useless for the PRIMARY KEYs field
    primaryKeys:
      - id
    fields:
      - name: id
        type: VARCHAR(128)
      - name: firstName
        type: VARCHAR(128)
      - name: lastName
        type: VARCHAR(128)
      - name: gender
        type: VARCHAR(128)
      - name: age
        type: SMALLINT
      - name: surname
        type: VARCHAR(128)
      - name: ref
        type: VARCHAR(128)
      - name: cls
        security: true
        # Hierarchical sort
        type: ENUM('NP', 'DR', 'SECRET', 'TOP_SECRET')
      - name: stmts
        type: JSON
        security: true
      - name: testidx
        type: VARCHAR(128)
      - name: ts
        type: BIGINT
      - name: thematics
        type: SET('PROLI','TARGET','NUCLEAR')
        security: true
      - name: testaddfield
        type: VARCHAR(128)
      - name: testbigjson
        type: JSON
      - name: testdouble
        type: DOUBLE
      - name: testfloat
        type: FLOAT

Mutate a model that already exists

Model mutation allow to add or delete fields and indexes but not to change their name, type or constraints.

When you want to change your model from a version n to n+1 you need to :

Update the version field to n+1.
Change the fields & indexes fields to match you new model.
Fill the mutations field with all the actions necessary to change the entity from version n to version n+1.

Note: Deleting and recreating the same field/index will not cancel each other.

As an exemple we will consider that the following model has been used to create an entity previously :

---
version: 1
entities:
  - id: person
    namespace: test
    history: true
    indexes:
      - name: firstName
        expression: (firstName)
    primaryKeys:
      - id
    fields:
      - name: id
        type: VARCHAR(128)
      - name: firstName
        type: VARCHAR(128)
      - name: testdouble
        type: DOUBLE
      - name: cls
        security: true
        # Hierarchical sort
        type: ENUM('NP', 'DR', 'SECRET', 'TOP_SECRET')

Let's say we want to :

remove the field 'testdouble'
remove the index 'firstName'
add a new field 'lastName' of type 'VARCHAR(128)'
add a new index 'lastName' on the new field 'lastName'

Our new model would look like this :

---
version: 2
entities:
  - id: person
    namespace: test
    history: true
    mutations:
      add:
        indexes:
          - name: lastName
            expression: (lastName)
        fields:
          - name: lastName
            type: VARCHAR(128)
      delete:
        indexes:
          - name: firstName
        fields:
          - name: testdouble
    indexes: # firstName should no longer exists, as such only the lastName index is present in here
      - name: lastName
        expression: (lastName)
    primaryKeys:
      - id
    fields:
      - name: id
        type: VARCHAR(128)
      - name: firstName
        type: VARCHAR(128)
      - name: cls
        security: true
        # Hierarchical sort
        type: ENUM('NP', 'DR', 'SECRET', 'TOP_SECRET')
      - name: lastName # testdouble has been removed and lastName has taken its place
        type: VARCHAR(128)

From a s3 bucket

Alternatively you can store your model files on a s3 server. Multiple files are allowed.

Add your model (with the same constraints and possibilities as above) in your s3 bucket, eg : mybucketmodel

Set the resources part in your config file.

server:
  port: 9080
  addr: 0.0.0.0
  resources:
    endpoint: kube0:31318
    user: minioadmin
    password: minioadmin
    # bucket where the model is stored
    modelsPath: mybucketmodel
[...]

Use the Change Data Capture (CDC) feature

CDC format is a lightweight version of the Canal JSON CDC format.

Exemple

Modify entity when history is activated:

HTTP PUT http://127.0.0.1:9080/v0/test/entities/person/937dd176-3548-4ddb-a3f3-a2b180286322
{
    "age" : 98
}

Read CDC entity

HTTP GET http://127.0.0.1:9080/v0/test/entities/person_history?filter=ref = '937dd176-3548-4ddb-a3f3-a2b180286322'

Return :

{
  "values" : [ {
    "ref" : "937dd176-3548-4ddb-a3f3-a2b180286322",
    "ts" : 1713873822105,
    "type" : "UPDATE",
    "identity" : "eric",
    "data" : "{\"age\": 98}",
    "old" : "{\"age\": 1}",
    "stmts" : "{\"SF\": 1, \"ACSSI\": 1, \"ECLAIR\": 1}",
    "thematics" : "PROLI,TARGET,NUCLEAR",
    "cls" : "DR"
  } ]
}

Dynamic Dataset/Views

Dynamic datset/views allow views creation with policy for database direct access with native driver.

Exemple

The following will create views based on provided model with the following format:

<entity_id_from_model>__<identity.id>__<identity.tenantId>

with the provided sample identity (see HTTP POST request):

<entity_id_from_model>__vuser__athea

HTTP POST http://127.0.0.1:9080/v0/<namespace>/vdataset
{
    "id": "vuser",
    "tenantId": "athea",
    "tags": [
      {
        "key": "cls",
        "value": "TOP_SECRET"
      },
      {
        "key": "stmts",
        "value": ["SF","ACSSI","ECLAIR"]
      },
      {
        "key": "thematics",
        "value": ["NOTIN","PROLI"]
      }
    ]
}

return:

{
    "status": 0,
    "token": "nY*66nNVcaN2"
}

The service account vuser__athea and token nY*66nNVcaN2 are now available for generic postgresql or mysql (TiDB/Clickhouse) driver usage.

API

Authentication

The authentication should be made with the Authorization header with a bearer token. This bearer token should contains these fields :

preferred_username
tenant_id

In order for the authorization to succeed, this pair should match a pair or id & tenantId in the entities known to vstore.

Request models

`GET /v0/models`

There are no parameters for this request.

Exemple

HTTP GET http://127.0.0.1:9080/v0/models

Return :

{
  "version" : 0,
  "entities" : [ {
    "id" : "person",
    "namespace" : "test",
    "history" : true,
    "mutation" : null,
    "indexes" : [ {
      "name" : "testidx",
      "expression" : "(testidx)"
    } ],
    "markForDeletion" : false,
    "fields" : [ {
      "name" : "id",
      "newName" : null,
      "type" : "VARCHAR(128) PRIMARY KEY",
      "security" : false
    }, {
      "name" : "firstName",
      "newName" : null,
      "type" : "VARCHAR(128)",
      "security" : false
    }, {
      "name" : "lastName",
      "newName" : null,
      "type" : "VARCHAR(128)",
      "security" : false
    }, {
      "name" : "gender",
      "newName" : null,
      "type" : "VARCHAR(128)",
      "security" : false
    }, {
      "name" : "age",
      "newName" : null,
      "type" : "SMALLINT",
      "security" : false
    }, {
      "name" : "surname",
      "newName" : null,
      "type" : "VARCHAR(128)",
      "security" : false
    }, {
      "name" : "ref",
      "newName" : null,
      "type" : "VARCHAR(128)",
      "security" : false
    }, {
      "name" : "cls",
      "newName" : null,
      "type" : "ENUM('NP', 'DR', 'SECRET', 'TOP_SECRET')",
      "security" : true
    }, {
      "name" : "stmts",
      "newName" : null,
      "type" : "JSON",
      "security" : true
    }, {
      "name" : "testidx",
      "newName" : null,
      "type" : "VARCHAR(128)",
      "security" : false
    }, {
      "name" : "ts",
      "newName" : null,
      "type" : "BIGINT",
      "security" : false
    }, {
      "name" : "thematics",
      "newName" : null,
      "type" : "SET('PROLI','TARGET','NUCLEAR')",
      "security" : true
    }, {
      "name" : "testaddfield",
      "newName" : null,
      "type" : "VARCHAR(128)",
      "security" : false
    }, {
      "name" : "testbigjson",
      "newName" : null,
      "type" : "JSON",
      "security" : false
    }, {
      "name" : "testdouble",
      "newName" : null,
      "type" : "DOUBLE",
      "security" : false
    }, {
      "name" : "testfloat",
      "newName" : null,
      "type" : "FLOAT",
      "security" : false
    }, {
      "name" : "count",
      "newName" : null,
      "type" : "BIGINT",
      "security" : false
    } ]
  } ]
}

Create element(s) in an entity

`POST /v0/<namespace>/entities/<entity>`

Path parameter	Description
`namespace`	Namespace of the entity in which you want to create elements
`entity`	Name of the entity in which you want to create elements

Exemple

HTTP POST http://127.0.0.1:9080/v0/test/entities/person
{
  "values": [
    {
      "id": "10009c11-ffb5-4bce-9806-fa4bffb22fb5",
      "firstName": "John10000",
      "cls": "SECRET",
      "count": 999
    },
    {
      "id": "10009c11-ffb5-4bce-9806-fa4bffb22fb6",
      "firstName": "John10001",
      "cls": "DR",
      "count": 1000
    }
  ]
}

Query element(s) of an entity

`GET /v0/<namespace>/entities/<entity>`

Path parameter	Description
`namespace`	Namespace of the entity in which you want to apply the SQL query
`entity`	Name of the entity in which you want to apply the SQL query

Query Parameter	Description
`actionstype`	Allow additional configuration on policies. The only supported is currently `masking=filter` which remove element with masked fields
`filter`	SQL like condition that should be used to limit the element that will be returned
`limit`	Maximum number of result to returns. Mutually exclusive with the body parameter of the same name
`offset`	Number of result that should be skipped in the initial result. Mutually exclusive with the body parameter of the same name
`sort`	SQL compliant mode of ordering results. Mutually exclusive with the body parameter of the same name

Body Parameter	Description
`limit`	Maximum number of result to returns. Mutually exclusive with the query parameter of the same name
`offset`	Number of result that should be skipped in the initial result. Mutually exclusive with the query parameter of the same name
`sort`	SQL compliant mode of ordering results. Mutually exclusive with the query parameter of the same name

The actionstype, limit, offset & sort parameters are optional.

`GET /v0/<namespace>/entities/<entity>/_query`

Path parameter	Description
`namespace`	Namespace of the entity in which you want to apply the SQL query
`entity`	Name of the entity in which you want to apply the SQL query

Query Parameter	Description
`actionstype`	Allow additional configuration on policies. The only supported is currently `masking=filter` which remove element with masked fields
`limit`	Maximum number of result to returns. Mutually exclusive with the body parameter of the same name
`offset`	Number of result that should be skipped in the initial result. Mutually exclusive with the body parameter of the same name
`orderby`	SQL compliant mode of ordering results. Mutually exclusive with the body parameter of the same name

Body Parameter	Description
`filter`	SQL like condition that should be used to limit the element that will be returned
`limit`	Maximum number of result to returns. Mutually exclusive with the query parameter of the same name
`offset`	Number of result that should be skipped in the initial result. Mutually exclusive with the query parameter of the same name
`orderby`	SQL compliant mode of ordering results. Mutually exclusive with the query parameter of the same name

The actionstype, limit, offset & orderby parameters are optional.

Exemple

Get all person with id = 5657de02-f9c9-448c-baa4-29a77ee2d540:

HTTP GET http://127.0.0.1:9080/v0/test/entities/person?filter=id = '5657de02-f9c9-448c-baa4-29a77ee2d540'

Same with a limit, offset and sort clause:

HTTP GET http://127.0.0.1:9080/v0/test/entities/person?filter=id = '5657de02-f9c9-448c-baa4-29a77ee2d540'&limit=10&offset=10&orderby=count asc

Or with a POST for too long url query:

HTTP POST http://127.0.0.1:9080/v0/test/entities/person/_query
{
  "filter": "id = '5657de02-f9c9-448c-baa4-29a77ee2d540'"
}

Same with a limit, offset and sort clause:

HTTP POST http://127.0.0.1:9080/v0/test/entities/person/_query
{
  "filter": "id = '5657de02-f9c9-448c-baa4-29a77ee2d540'",
  "limit": 10,
  "offset": 10,
  "orderby": "count asc"
}

Get all person with firstName = John4203:

HTTP GET http://127.0.0.1:9080/v0/test/entities/person?filter=firstName = 'John4203'

Return:

{
  "values" : [ {
    "id" : "1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2",
    "firstName" : "John4203",
    "lastName" : "Doe4203",
    "gender" : "F",
    "age" : 30,
    "surname" : "James5657de02-f9c9-448c-baa4-29a77ee2d540",
    "ref" : "5657de02-f9c9-448c-baa4-29a77ee2d540link",
    "cls" : "TOP_SECRET",
    "stmts" : "{\"ACSSI\": 1, \"ECLAIR\": 1, \"SF\": 1}",
    "thematics" : "{\"NUCLEAR\": 1, \"PROLI\": 1, \"TARGET\": 1}"
  } ]
}

If you need to transform on demand an actions type (see policies config) from masking to filter, use:

HTTP GET http://127.0.0.1:9080/v0/test/entities/person?actionstype=masking=filter

Updating element(s) of an entity

By primary key : `PUT /v0/<namespace>/entities/<entity>/<id>`

Path Parameter	Description
`namespace`	Namespace of the entity in which you want to update an element
`entity`	Name of the entity in which you want to update an element
`id`	Primary key of the element you wish to update

You must provide a JSON body with the fields you want to update and their new value.

Exemple

Update the field age of a person with id = 1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2 (id must be declared as a primary key):

HTTP PUT http://127.0.0.1:9080/v0/test/entities/person/1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2
{
    "age" : 34
}

Return (HTTP 200, status: 1 mean one element modified):

{
    "status": 1,
    "msg": null
}

By filter : `PUT /v0/<namespace>/entities/<entity>`

Updating by filter allow to provide a filter which will allow to select the element which will be updated. (Filtering through security attributes still applies) You should filter using query parameter as the preferred option and use the filter through body parameter only if your filter gets too long.

Filtering with query parameter

Path Parameter	Description
`namespace`	Namespace of the entity in which you want to update an element
`entity`	Name of the entity in which you want to update an element

Query Parameter	Description
`filter`	SQL like condition that should be used to limit the element that should receive the update

You must provide a JSON body with the fields you want to update and their new value.

Filtering with body parameter

Path Parameter	Description
`namespace`	Namespace of the entity in which you want to update an element
`entity`	Name of the entity in which you want to update an element

Body Parameter	Description
`filter`	SQL like condition that should be used to limit the element that should be updated
`update`	JSON object with the fields you want to update and their new value.

Exemple

HTTP PUT http://127.0.0.1:9080/v0/test/entities/person?filter= id = '1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2'
{
    "age" : 34
}

HTTP PUT http://127.0.0.1:9080/v0/test/entities/person
{
    "filter" : "id = '1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2'",
    "update": {
        "age" : 34
    }
}

Both return (HTTP 200, status: 1 mean one element modified):

{
    "status": 1,
    "msg": null
}

Delete element(s) of an entity

By primary key : `PUT /v0/<namespace>/entities/<entity>/<id>`

Path Parameter	Description
`namespace`	Namespace of the entity in which you want to delete an element
`entity`	Name of the entity in which you want to delete an element
`id`	Primary key of the element you wish to delete

Exemple

Delete a person with id = 1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2 (id must be declared as a primary key):

HTTP DELETE http://127.0.0.1:9080/v0/test/entities/person/1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2

Return (HTTP 204, status: 1 mean one element deleted):

{
    "status": 1,
    "msg": null
}

By filter : `DELETE /v0/<namespace>/entities/<entity>`

Deleting by filter allow to provide a filter which will allow to select the element which will be deleted. (Filtering through security attributes still applies) You should filter using query parameter as the preferred option and use the filter through body parameter only if your filter gets too long.

Filtering with query parameter

Path Parameter	Description
`namespace`	Namespace of the entity in which you want to delete an element
`entity`	Name of the entity in which you want to delete an element

Query Parameter	Description
`filter`	SQL like condition that should be used to limit the element that should be deleted

Filtering with body parameter

Path Parameter	Description
`namespace`	Namespace of the entity in which you want to delete an element
`entity`	Name of the entity in which you want to delete an element

Body Parameter	Description
`filter`	SQL like condition that should be used to limit the element that should be deleted

Exemple

HTTP DELETE http://127.0.0.1:9080/v0/test/entities/person?filter= id = '1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2'

HTTP PUT http://127.0.0.1:9080/v0/test/entities/person
{
    "filter" : "id = '1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2'"
}

Both return (HTTP 204, status: 1 mean one element deleted):

{
    "status": 1,
    "msg": null
}

Get an entity history

`GET /v0/<namespace>/entities/<entity>_history`

Path Parameter	Description
`namespace`	Namespace of the entity in which you want to update an element
`entity`	Name of the entity in which you want to update an element

Query Parameter	Description
`filter`	SQL like condition that should be used to limit the element that should be deleted

Do note that the history tables have predetermined fields, the ref field for exemple being the unique ID of the original element concerned by the history element.

Exemple

HTTP GET http://127.0.0.1:9080/v0/test/entities/person_history?filter=ref = '1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2'

Return:

{
  "values" : [ {
    "ref" : "1ddbf8b0-6065-4b56-a2fc-e1be3e21c3f2",
    "ts" : 1713873822105,
    "type" : "UPDATE",
    "identity" : "eric",
    "data" : "{\"age\": 34}",
    "old" : "{\"age\": 32}",
    "stmts" : "{\"SF\": 1, \"ACSSI\": 1, \"ECLAIR\": 1}",
    "thematics" : "PROLI,TARGET,NUCLEAR",
    "cls" : "DR"
  } ]
}

Using the DDL api

The DDL api allow the use of SQL on an entity.

`POST /v0/<namespace>/entities/<entity>/_ddl`

Path parameter	Description
`namespace`	Namespace of the entity in which you want to apply the SQL query
`entity`	Name of the entity in which you want to apply the SQL query

Body Parameter	Description
`alter`	SQL action that end with a `;`

Exemple

Add a column

POST http://127.0.0.1:9080/v0/iam/entities/person/_ddl
{
  "alter": "ADD COLUMN phone VARCHAR(128);"
}

Rename a column

POST http://127.0.0.1:9080/v0/iam/entities/person/_ddl
{
  "alter": "RENAME COLUMN phone TO phone2;"
}

Drop a column

POST http://127.0.0.1:9080/v0/iam/entities/person/_ddl
{
  "alter": "DROP COLUMN phone2;"
}

Informations​

Supported storages types​

Configurations​

Configuration file​

Define a Namespace/Store​

Usage​

In the config file​

From a s3 bucket​

Define the Policy​

Usage​

In the config file​

From a s3 bucket​

Define the identities​

Usage​

In the config file​

Dynamic Identities​

Define a model​

In the config file​

Mutate a model that already exists​

From a s3 bucket​

Use the Change Data Capture (CDC) feature​

Exemple​

Dynamic Dataset/Views​

Exemple​

API​

Authentication​

Request models​

GET /v0/models​

Exemple​

Create element(s) in an entity​

POST /v0/<namespace>/entities/<entity>​

Exemple​

Query element(s) of an entity​

GET /v0/<namespace>/entities/<entity>​

GET /v0/<namespace>/entities/<entity>/_query​

Exemple​

Updating element(s) of an entity​

By primary key : PUT /v0/<namespace>/entities/<entity>/<id>​

Exemple​

By filter : PUT /v0/<namespace>/entities/<entity>​

Filtering with query parameter​

Filtering with body parameter​

Exemple​

Delete element(s) of an entity​

By primary key : PUT /v0/<namespace>/entities/<entity>/<id>​

Exemple​

By filter : DELETE /v0/<namespace>/entities/<entity>​

Filtering with query parameter​

Filtering with body parameter​

Exemple​

Get an entity history​

GET /v0/<namespace>/entities/<entity>_history​

Exemple​

Using the DDL api​

POST /v0/<namespace>/entities/<entity>/_ddl​

Exemple​

Add a column​

Rename a column​

Drop a column​

Informations

Supported storages types

Configurations

Configuration file

Define a Namespace/Store

Usage

In the config file

From a s3 bucket

Define the Policy

Usage

In the config file

From a s3 bucket

Define the identities

Usage

In the config file

Dynamic Identities

Define a model

In the config file

Mutate a model that already exists

From a s3 bucket

Use the Change Data Capture (CDC) feature

Exemple

Dynamic Dataset/Views

Exemple

API

Authentication

Request models

`GET /v0/models`

Exemple

Create element(s) in an entity

`POST /v0/<namespace>/entities/<entity>`

Exemple

Query element(s) of an entity

`GET /v0/<namespace>/entities/<entity>`

`GET /v0/<namespace>/entities/<entity>/_query`

Exemple

Updating element(s) of an entity

By primary key : `PUT /v0/<namespace>/entities/<entity>/<id>`

Exemple

By filter : `PUT /v0/<namespace>/entities/<entity>`

Filtering with query parameter

Filtering with body parameter

Exemple

Delete element(s) of an entity

By primary key : `PUT /v0/<namespace>/entities/<entity>/<id>`

Exemple

By filter : `DELETE /v0/<namespace>/entities/<entity>`

Filtering with query parameter

Filtering with body parameter

Exemple

Get an entity history

`GET /v0/<namespace>/entities/<entity>_history`

Exemple

Using the DDL api

`POST /v0/<namespace>/entities/<entity>/_ddl`

Exemple

Add a column

Rename a column

Drop a column