Last updated: 2024-04-11 17:13

Introduction

This page explains the functioning of the CSVUploader feature. The CSVUploader feature is aimed to do a bulk upload of records: by filling a csv file, one record per row represents one submission so a user can fill as much records as needed.

Training

Below you can review the HD4D¨2 CSV Upload training organized by healthdata.be:

HD4DP2_CSV_Training_recording_part01 Download

HD4DP2_CSV_Training_recording_part02 Download

HD4DP2_CSV_Training_recording_part03 Download

Architecture

The CSVUploader is located under hd-connect/csvuploader. It uses both hd-connect-csvuploader and hd-connect-proxy modules.

The general architecture of the CSVUploader is explained in the sequence diagram below.

Third-party libraries and frameworks

Apache Camel: https://camel.apache.org/
Spring Boot

Testing and working

The CSVUploader creates a folder at root level (SFTP for end-user, or hd-all for developer) that contains a subfolder per existing organisation.
The CSVUploader will poll with a delay of 1 min, process the csv file and then create 3 folders:
- ARCHIVE folder: contains the source csv file.
- RESULTS folder: contains the results of processing the csv file. This file contains the specified data, and the final status of the processing: Success or Error. If an error occurred, the error message is displayed. For multiple uploads, the result is added to the end of the result file each time.
- ERROR folder: this folder is created if the csv test file was not processed, due to an I/O error (file corrupted, not found etc.). So for now, only technical errors are caught and the source csv file is moved to that folder instead of the ARCHIVE folder. In principle, this folder should contain any result that is an error. The RESULT folder should only contain results that end with a SUCCESS status.

Formats

Some formats are specific:

Dates: should be dd/mm/yyyy
Boolean: true / false
Codes: the value of the code (not the translation)
Multi codes: there is only one column per field. So when a select box is set as multiple, values have to be separated by a "|". e.g.: 68452|68453|68454
Repeatable blocks: in some DCDs, a complete block of fields is repeatable. In that case, value have to be separated by a ";"
- e.g.: A block is containing 3 fields: A (Lob), B (Type klep) and C (Aantal kleppen)

Examples:

In case of a multiple choice:

Example of a multiple choice as presented in the DCD:

As presented in the form:

For reporting fields with a multiple choice formatting, the selected answers can be reported separated by the pipe symbol (|).

Example of a multiple field reporting:

cftr_modulating_therapy_1;

1|2|3|4|5|6;

In case of a repeatable block

Example of a repeatable block as presented in the DCD:

As presented in the form:

Manually enter the repeatable fields as shown in the following template: field_category|<index>|field.

Example of a repeatable block reporting:

Repeatable 1:

Transplants

transplant_status

Repeatable 2: