SAS (Software Analytical Suite)
Overview
SAS is a standalone component that acts as the main entrypoint for data within kosmos platform solution.
It enables adhoc malware and virus detection of datas.
The component supports two type of Antivirus solution for advance threat and malware detection:
- ESET (Eset Endpoint Antivirus)
- ClamAV
Installation
SAS is composed of two helm charts
- core: threat detection
- ingest: pipelines
The ingest component requires eds, kflow and the core helm charts to be deployed.
Prerequisites
The complete list of dependencies for the SAS to work is as follows:
- keycloak (oss:deployed)
- natsjetstream (oss:deployed)
- postgresql (oss:deployed)
- kflow (kosmos:deployed)
- vstore (kosmos:deployed)
- eds (kosmos:deployed)
Deploying using Helmfile
cd helmfile
helmfile sync -f helmfile.yaml
Deploying using Helm
Namespace creation with PSP privileged
kubectl apply -f- <<EOF
apiVersion: v1
kind: Namespace
metadata:
name: kosmos-system-restricted
labels:
pod-security.kubernetes.io/enforce: privileged
EOF
Core with ESET/ClamAV as backend
helm install sas ../idu/helm_charts/iduv2 -n kosmos-system-restricted -f- <<EOF
replicaCount: 1 # does not support >1
imagePullSecrets: []
podAnnotations: {}
podLabels: {}
metrics:
enabled: true
namespaceSelector: default
admin:
initKeycloakIdp:
enabled: true
server: http://host.docker.internal:8888
realm: metier
clientId: sas
clientSecret: password
initPostgresDB:
enabled: true
addr: host.docker.internal:5432
username: postgres
password: password
image:
repository: TODO/iduv2-admintools
pullPolicy: Always
tag: "dev"
sas:
notificationCentre:
enabled: true
addr: host.docker.internal:4222
stream: notifs
subject: sas
defaultRoom: TODO
av: eset
# av: clamav
image:
repository: TODO/iduv2-eset
# repository: TODO/iduv2-clamav
pullPolicy: Always
tag: "dev"
engine:
numConcurrentScan: 100
maxScanFileBytesSize: 1073741824000044
resources:
runtime:
requests:
memory: 3Gi
logLevel: DEBUG # DEBUG, INFO, WARN, ERROR
nats:
addr: host.docker.internal:4222
stream: iduv2
consumer: iduv2scanner
vstore:
enabled: true
addr: http://host.docker.internal:9080
namespace: kosmos_back
entity: sas
jwt: TODO
store:
dst:
# createIfNotExists works only when static type is used
retention: "7d"
createIfNotExists: true
type: static
addr: http://host.docker.internal:9000
topics: clean
username: minioadmin
password: minioadmin
apiKey: "" # not required when using static type
src:
auditWebhook:
addr: http://host.docker.internal:8000
reset: true
ingress:
enabled: true
className: nginx
host: sas.kosmos.wip
hostAdmin: sas-admin.kosmos.wip
tlsApi: []
tlsUI: []
retention: "7d"
image:
repository: quay.io/minio/minio
pullPolicy: Always
tag: "latest"
resources:
disks:
storageClassName: local-path
resources:
requests:
storage: 22Gi
runtime:
requests:
memory: 3Gi
username: minioadmin
password: minioadmin
sshprivatekey: |
-----BEGIN RSA PRIVATE KEY-----
[TODO]
-----END RSA PRIVATE KEY-----
nodeSelector: {}
tolerations: []
affinity: {}
EOF
Ingest Pipelines
helm install sas ../idu/helm_charts/sasingest -n kosmos-data -f- <<EOF
image:
repository: TODO/iduv2-admintools
pullPolicy: Always
tag: "dev"
kflow:
parallelism: 3
namespace: default
addr: http://host.docker.internal:9080
forceDelete: false
nats:
addr: host.docker.internal:4222
list: # s3 src
addr: host.docker.internal:9000
topics: s3a://clean
user: "minioadmin"
password: "minioadmin"
checkpoint: false
sink: # s3 dst
image:
repository: TODO/basic-python
pullPolicy: Always
tag: "dev"
imagePullPolicy: Always
sap:
apiKey: TODO
addr: http://host.docker.internal:3080
EOF
Virus Definition Updates
Updating the virus definition for both of our supported antivirus engine is done at build time rather than at runtime.
In other words, updating an antivirus virus definition is done by producing a new image tag from source.
The generation of the image itself requires that services available in a non-locked down environment is made available on the build envrionment.
ClamAV / Online
# check current tag CVD
podman run -u 1000 -it --rm\
--entrypoint cat iduv2-clamav:dev /athea/clamav_cvd_version.txt
cat > Dockerfile.update <<EOF
FROM iduv2-clamav:dev
RUN freshclam &&\
freshclam --version > clamav_cvd_version.txt
EOF
# check new version
podman run -u 1000 -it --rm --entrypoint cat\
iduv2-clamav:dev-updated /athea/clamav_cvd_version.txt
podman build . -f Dockerfile.update -t "iduv2-clamav:dev-updated"
ClamAV / Offline
# check current tag CVD
podman run -u 1000 -it --rm\
--entrypoint cat iduv2-clamav:dev /athea/clamav_cvd_version.txt
# download CVD to local filesystem
# http://db.local.clamav.net/main.cvd
# http://db.local.clamav.net/daily.cvd
# rebuild image with new cvd
cat > Dockerfile.update <<EOF
FROM iduv2-clamav:dev
COPY daily.cvd /var/lib/clamav/daily.cvd
COPY main.cvd /var/lib/clamav/main.cvd
RUN freshclam --version > clamav_cvd_version.txt
EOF
# build
podman build . -f Dockerfile.update -t "iduv2-clamav:dev-updated"
# check new version
podman run -u 1000 -it --rm --entrypoint cat\
iduv2-clamav:dev-updated /athea/clamav_cvd_version.txt
ClamAV / Source
# build base image eset
podman build . -t base -f Dockerfile
# build from base image
podman build . -t iduv2-clamav:[new_tag]\
-f Dockerfile.engineclamav
ESET / Warning
Updating ESET virus definition revolves around generating a new image tag. The new image will contain statically all the latest virus definition avaiable at the date of the build.
Doing this action, requires a valid EEA license.
!!! warn "Expect long build time !"
Building the ESET container image from source may take severals minutes. This is due to license registration and the virus definition updates that is done at build time.
ESET / Online
export REGISTRY_URL=""
export TAG="dev"
export VALID_USR_PASS_ESET_LICENSE="eset_usr_pass_eea_license.lf"
export VALID_ESET_LICENSE="eset_eea_license.lf"
cat > Dockerfile <<EOF
FROM ${REGISTRY_URL}/iduv2-eset:${TAG}
COPY ${VALID_USR_PASS_ESET_LICENSE} eset_eea_license.lf
RUN eea-update-build
COPY ${VALID_ESET_LICENSE} eset_eea_license.lf
EOF
podman build . -t ${REGISTRY_URL}:[a_new_tag]
ESET / Offline
!!! note "You could use MirrorTool instead"
The only requirements is to generate the required signatures on local filesystem.
See documentation
Fetching updates using container approach on local filesystem
cat > Dockerfile.fetch <<EOF
FROM iduv2-eset:dev AS dl
RUN eea-update-build
FROM scratch AS sigs
COPY --from=dl /var/opt/eset/eea/lib/ .
EOF
docker buildx build\
--output=sigs\
--target=sigs . -f Dockerfile.fetch
See ESET EEA directory structure
# workdir
ls sigs/
> em000_64.dat em004_64.dat em022_64.dat em039_64.dat em061_64.dat
> em001_64.dat em005_64.dat em023_64.dat em042_64.dat modules_eea.tar
> em002_64.dat em017_64.dat em024_64.dat em045_64.dat
> em003_64.dat em019_64.dat em029_64.dat em052_64.dat
Update command:
cat > Dockerfile.update <<EOF
FROM iduv2-eset:dev
COPY sigs/ /var/opt/eset/eea/lib/
COPY eea-update-meta.sh eea-update-meta.sh
RUN ./eea-update-meta.sh
EOF
podman build . -f Dockerfile.update\
-t iduv2-eset:dev-updated
# check
podman run -it --rm --entrypoint cat\
iduv2-eset:dev-updated eset_vd_version.txt
Outputs:
EM000 1085 (20231103) Update module
EM001 1615.1 (20240815) Antivirus and antispyware scanner module
EM002 29900 (20240916) Detection engine
EM003 1352 (20240729) Archive support module
EM004 1229 (20240610) Advanced heuristics module
EM017 2019 (20240823) Translation support module
EM019 1475.4 (20240801) Internet protection module
EM022 1128 (20240814) Database module
EM023 24979 (20240916) Rapid Response module
EM024 1139 (20240603) Direct Cloud communication module
EM029 1049 (20231018) Mac/Linux support module
EM039 2138.2 (20240722) Configuration module
EM042 1695 (20231204) Network protection module
EM045 1088 (20240718) Cryptographic protocol support module
EM052 1024 (20240617) Cleaner module
EM061 1230 (20240910) Vulnerability & patch management module
########## Updated ##########
EM000 1085 (20231103) Update module
EM001 1615.1 (20240815) Antivirus and antispyware scanner module
EM002 29917 (20240919) Detection engine
EM003 1352 (20240729) Archive support module
EM004 1229 (20240610) Advanced heuristics module
EM017 2020 (20240828) Translation support module
EM019 1475.4 (20240801) Internet protection module
EM022 1128 (20240814) Database module
EM023 24996 (20240919) Rapid Response module
EM024 1139 (20240603) Direct Cloud communication module
EM029 1049 (20231018) Mac/Linux support module
EM039 2138.2 (20240722) Configuration module
EM042 1695 (20231204) Network protection module
EM045 1088 (20240718) Cryptographic protocol support module
EM052 1024 (20240617) Cleaner module
EM061 1231 (20240918) Vulnerability & patch management module
########## Updated ##########
ESET / Source
# build base image eset
podman build . -t base -f Dockerfile
# build from base image
podman build . -t iduv2-eset:[new_tag]\
-f Dockerfile.engineeset
ClamAV License Renewal
ClamAV is an opensource antivirus.
ESET License Renewal
ESET is a closed-source software and requires a license to be used in production/and for virus definitions updates.
EEA (ESET Endpoint Antivirus) is one of the supported antivirus backend of the SAS. To use EEA as an antivirus backend in a locked-down environment, you need an offline license. Offline license can be generated on ESET Portal.
Step 1: Generate License from ESET portal
Visit: ESET Portal
Step 2: Activation/Renewal with the new generated license
export REGISTRY_URL=""
export TAG="dev"
# path on your filesystem where the new license is located
export VALID_ESET_LICENSE="eset_eea_license.lf"
# build image
podman pull ${REGISTRY_URL}/iduv2-clam:${TAG}
cat > Dockerfile <<EOF
FROM ${REGISTRY_URL}/iduv2-clam:${TAG}
COPY ${VALID_ESET_LICENSE} eset_eea_license.lf
EOF
podman build . ${REGISTRY_URL}/iduv2-clam:${TAG}_[date_of_license]
podman push ${REGISTRY_URL}/iduv2-clam:${TAG}_[date_of_license]
# provision a SAS update by overriding image tag
export KUBECONFIG=$(readlink -f /sshkubectl/kubeconfig)
export ENV=prod.eset
helmfile apply\
-l av=eset\
--set sas.image.tag=${TAG}\
--namespace=esetnamespace