CSV Upload for BSACC-ME for Medical Follow-Up

Last updated: 2024-02-22 22:41

Documentation for CSV Upload on Architecture 2.0

Description of the service

The CSV Upload functionality gives the possibility to import multiple parameters from a set of patients in one time into HD4DP 2.0. The csv file is based on an extract of the electronic patient files and/or other local databases.

Currently there is no user interface in HD4DP 2.0 for uploading csv files. If a data provider wants to upload a csv file, the file has to be dropped at a specific location. These files will be picked up and processed periodically. The files should be “final”, meaning that no application is writing to them. The pick-up location will be identical for all registries.
Pre-registry handling will be based on a naming-convention of the csv file.

HOW TO: Upload data using CSV Upload

Steps To Upload data

1. Prepare the csv file (example file in this section)

  • Extract the csv file from the electronic patient files and/or other local databases.
  • Author group, Author and Coauthor:
    • When the Author Group, Author and Coauthor have been left out in the csv file, the default Author group, Author and Coauthor will be used automatically.
    • When the desired Author Group, Author and Coauthor are specified in the csv file, the following headers TX_AUTHOR_GR, TX_AUTHOR and TX_COAUTHOR must be added to the csv file with their values respectively.

      Example:
TX_AUTHOR_GR;TX_AUTHOR;TX_COAUTHOR
Test group;test@sciensano.be;test@sciensano.be

Note:
The Author group, Author and Coauthor must exist and are well configured at the back-end of the system. TX_AUTHOR_GR can be a string that identifies the Author group to which this Author belongs. Commonly, the first name and last name are used to identify the TX_AUTHOR_GROUP. Be sure to avoid leading and trailing spaces when entering the Author group value.

  • Similarly, add the field name 'STATUS' in capitals in an additional column. Add the value 'draft' in case a manual submission of the record is requested.
    If not, the record will be submitted without manual intervention.
  • Adding separators to a NISS number:
    It is not necessary to add separators in a NISS number when uploading a file using CSV Upload. You can fill out the NISS number both with or without separators. E.g.: 85.04.02-169.32 or 85040216932.

    Example:
  • Make sure the name of the csv file has the correct format:
    HD_DCD_submcsv_HDBPnumber_HDBPabbreviation_versionnumber_versionreleasedate

So for BSACC-ME - Medical Follow-up the format would be:
HD_DCD_submcsv_HDBP0242_BsaccMeBsaccFuMed_01_20221220.csv

EXAMPLES:

Disclaimer: The example files above are only provided as a guideline and do not contain real life data.

2. Uploading the csv file

Step 1: Open the sftp tool like WinSCP

Step 2: Get the credentials (Host name, Port number, User name and Password) from the IT department of the Data Provider, to log on to the sftp server located on the Data Provider side. See section Data transfer via an SFTP client in the documentation of the Online Acceptance Environment.

Step 3: Fill in the credentials into the Login screen and click on Login to be able to access the different upload folders:

Note: a warning might be given, just click on Update

Now the CSV Upload folder structure is displayed on the right-hand side panel. You will notice draftcsv folders and submcsv folders for the diverse HD business projects. Historically, a directly submitted csv file would end up in the submitted folder, whereas a csv file featuring ‘draft’ in the status field would have ended up in the draft folder. At this stage there is no real distinction between both folders, so you can save the csv file in either of them.
Attention: If you wish to create a draft to work on at a later stage, don’t forget to add ‘draft’ in the status column of the csv file.

Step 4: Select the project folder BSACC:MonitoringandEvaluation-19 and open it by double-clicking on it:

Step 5: Double-click on the DCD folder to open it:

Step 6: Now go to the folder on the left-hand side panel where the CSV file to be uploaded is located:

Step 7: Drag the CSV file to be uploaded from the left-hand side panel into the folder on the right-hand side panel:

Step 8: Wait until the polling system of the CSV Uploader has picked up the CSV file and has processed it.
Once the CSV file has been processed it will disappear from the folder, when we refresh the page manually!

3. Validate csv upload

Once the csv file has been processed 3 folders will be created (if they haven't been created already) in the DCD folder:

  • ARCHIVE (after a csv file has been processed, the original csv file will be saved in this folder)
  • RESULT (when the csv file is placed in this folder, it means that the csv file has been processed, a file will be created or (or appended, if the file already existed) with the result of the upload of the csv file).
    All the errors that are described in this file are business related, which means that they are technically correct, but in violation with the business rules or contain wrong values for that field.
  • ERROR (when the csv file is placed in this folder, it means that a technical error has occurred like the csv file contained erroneous formatting. The csv file won't get processed and an error file will be created with the errors and reason why the csv file couldn't be processed)

3.1 Validation of the csv upload via sftp tool:

Step 1: Double-click on the ERROR folder to open it, click on the refresh button and verify that there is no error file present.

Step 2: Return to the DCD folder. Now double-click on the RESULT folder to open it, click on the refresh button and verify that the result file is present.

Step 3: Double-click on the result file to open it.

Step 4: If there are multiple records in the result file, scroll to the entry of the current csv upload by looking at the upload date (Started at dd/mm/yyyy hh:mm).
Verify the result file that the upload was successful by searching for the word SUCCESS and having a look at the Status. This Status must contain: Success;Success Count:1;Error Count:0

3.2 Validation of the csv upload via HD4DP 2.0:

Step 1: Open the web application HD4DP 2.0.

Step 2: Select the concerned organization in the dropdown list and click on Volgende (Next)

Step 3: Fill in the username and password, that has been provided by your IT Department or Healthdata team, and click on Log in to access the HD4DP 2.0 application.

Step 4: Navigate in the menu on the left-hand side panel to the desired study program:

Step 5: Check that the uploaded DCD(s) is/are displayed in the overview table.