Data Engineering & AI

Data Engineering
& Artificial Intelligence

Secure ETL pipelines, SNDS extraction and linearisation, machine learning models and clinical natural language processing.

233
Selected SNDS variables
8
EDS pipelines
99.9%
Platform availability
15+
ML algorithms

A complete and high-performance data stack

We build robust, scalable and secure pipelines to extract, transform and analyse health data.

Victoria Pipeline (SNDS)

Proprietary ETL pipeline in Spark/Scala. 3 steps: SNDS to Parquet conversion, cleaning via WikiPipeline, entity extraction (CCAM, ICD-10, LPP, Drugs, Patients, Consultations, ALD, GHM). V2 medico-economic module for care pathway valuation.

Filters (Data Minimization)

11 specialised Spark/Scala jobs: care pathway variables, hospitalisations, drugs, DDD exposure, polypharmacy, resource consumption, sick leave and costs. 100% test coverage.

Machine Learning

Classification, regression and clustering algorithms. Automatic phenotyping of complex conditions using deep learning.

Automated quality control

Automatic anomaly detection in datasets. Business rule validation, temporal consistency and variable completeness checks.

Rosetta: cohort data processing

Quality pre-assessment

Automated quality analysis of registry data before processing.

Modular library

Common engine with core functions plus registry-specific functions. Developed in Python with test coverage.

Input/Output

Supports Excel files and specification files. Produces a normalised registry database (1 row per patient, 1 column per raw variable).

Deployment

Docker containerisation, GitLab versioning, continuous integration.

Deployment pipeline

Git (GitLab CI) Security scan Docker image
Kubernetes EDS

A secure and certified architecture

Our platform is built on an HDS-certified infrastructure, with isolated environments and end-to-end encryption.

Certified HDS hosting

Infrastructure compliant with health data security standards, annual audit and continuous certification.

Isolated environments

Dedicated workspaces per project, access segregation, complete audit logs for full traceability.

End-to-end encryption

Data encrypted in transit and at rest. Encryption key management, multi-factor authentication for all access.

Monitoring & alerts

Real-time pipeline monitoring, anomaly alerts, performance and uptime dashboards.

Analytical tools

R
R & RStudio
Data manipulation, visualisation, statistical models
Py
Python & Jupyter
Machine learning, deep learning, medical image processing

AI at the heart of all our processes

We integrate artificial intelligence across all our processes to improve production timelines and the quality of our deliverables.

Protocol writing

AI-assisted writing of study protocols, literature review and structuring of CESREES applications. Significant time savings on preparatory phases.

Development & code review

AI-assisted code generation and review for ETL pipelines, analysis scripts and phenotyping algorithms. Reduced errors and faster development.

Analysis & interpretation

AI assistance for interpreting statistical results, detecting anomalies in datasets and identifying patterns in patient care pathways.

Quality control

Automated consistency checking of results, outlier detection, cross-validation of outputs. AI reinforces every step of our production chain.

Accelerate your data project

Discover our technical infrastructure and data science capabilities.
We build robust and scalable pipelines for your health data.