Documentation for CSV Upload on Architecture 2.0
Description of the service
The CSV Upload functionality gives the possibility to import multiple parameters from a set of patients in one time into HD4DP 2.0. The csv file is based on an extract of the electronic patient files and/or other local databases.
Currently there is no user interface in HD4DP 2.0 for uploading csv files. If a data provider wants to upload a csv file, the file has to be dropped at a specific location. These files will be picked up and processed periodically. The files should be “final”, meaning that no application is writing to them. The pick-up location will be identical for all registries.
Pre-registry handling will be based on a naming-convention of the csv file.
HOW TO: Upload data using CSV Upload
Steps To Upload data
1. Prepare the csv file (example file in this section)
- Extract the csv file from the electronic patient files and/or other local databases.
- Author group, Author and Coauthor:
- When the Author Group, Author and Coauthor have been left out in the csv file, the default Author group, Author and Coauthor will be used automatically.
- When the desired Author Group, Author and Coauthor are specified in the csv file, the following headers TX_AUTHOR_GR, TX_AUTHOR and TX_COAUTHOR must be added to the csv file with their values respectively.
TX_AUTHOR_GR;TX_AUTHOR;TX_COAUTHOR Test group;firstname.lastname@example.org;email@example.com
The Author group, Author and Coauthor must exist and are well configured at the back-end of the system. TX_AUTHOR_GR can be a string that identifies the Author group to which this Author belongs. Commonly, the first name and last name are used to identify the TX_AUTHOR_GROUP. Be sure to avoid leading and trailing spaces when entering the Author group value.
- Similarly, add the filename 'STATUS' in an additional column. Add the value 'draft' in case a manual submission of the record is requested.
If not, the record will be submitted without manual intervention.
- Make sure the name of the CSV file has the correct format:
So for BeCPR - Main registration the format would be:
- Please find below an example file for BeCPR - Main registration:
Disclaimer: The example files above are only provided as a guideline and do not contain real life data.
2. Uploading the csv file
Step 1: Open the sftp tool like WinSCP
Step 2: Get the credentials (Host name, Port number, User name and Password) from the IT department of the Data Provider, to log on to the sftp server located on the Data Provider side. See section Data transfer via an SFTP client in the documentation of the Online Acceptance Environment.
Step 3: Fill in the credentials into the Login screen and click on Login to be able to access the different upload folders:
Note: a warning might be given, just click on Update
Now the CSV Upload folder structure is displayed on the right-hand side panel:
Step 4: Select the project folder <project folder name> and open it by double-clicking on it:
<Screenshot of Project with DCD folders>
Step 5: Double-click on the DCD folder to open it:
<Screenshot of DCD folders>
Step 6: Now go to the folder on the left-hand side panel where the csv file to be uploaded is located:
<Screenshot of DCD folders>
Step 7: Drag the csv file to be uploaded from the left-hand side panel into the folder on the right-hand side panel:
<Screenshot of file on right-hand side panel>
Step 8: Wait until the polling system of the CSV Uploader has picked up the CSV file and has processed it.
Once the csv file has been processed it will disappear from the folder, when we refresh the page manually!
<Screenshot of DCD folder where the file has disappeared after a refresh>
3. Validate csv upload
Once the csv file has been processed 3 folders will be created (if they haven't been created already) in the DCD folder:
- ARCHIVE (after a csv file has been processed, the original csv file will be saved in this folder)
- RESULT (when the csv file is placed in this folder, it means that the csv file has been processed, a file will be created or (or appended, if the file already existed) with the result of the upload of the csv file).
All the errors that are described in this file are business related, which means that they are technically correct, but in violation with the business rules or contain wrong values for that field.
- ERROR (when the csv file is placed in this folder, it means that a technical error has occurred like the csv file contained erroneous formatting. The csv file won't get processed and an error file will be created with the errors and reason why the csv file couldn't be processed)
3.1 Validation of the csv upload via sftp tool:
Step 1: Double-click on the ERROR folder to open it, click on the refresh button and verify that there is no error file present.
Step 2: Return to the DCD folder. Now double-click on the RESULT folder to open it, click on the refresh button and verify that the result file is present.
Step 3: Double-click on the result file to open it.
<Screenshot of result file of the DCD>
Step 4: If there are multiple records in the result file, scroll to the entry of the current csv upload by looking at the upload date (Started at dd/mm/yyyy hh:mm).
Verify the result file that the upload was successful by searching for the word SUCCESS and having a look at the Status. This Status must contain: Success;Success Count:1;Error Count:0
<Screenshot of result file of the DCD>
3.2 Validation of the csv upload via HD4DP 2.0:
Step 1: Open the web application HD4DP 2.0.
Step 2: Select the concerned organization in the dropdown list and click on Volgende (Next)
Step 3: Fill in the username and password, that has been provided by your IT Department or Healthdata team, and click on Log in to access the HD4DP 2.0 application.
Step 4: Navigate in the menu on the left-hand side panel to the desired study program:
<Screenshot of menu navigation to the DCD>
Step 5: Check that the uploaded registration(s) is/are displayed in the overview table.
<Screenshot of DCD Overview screen>
This documentation is being updated regularly. We try to provide as correct, complete and clear as possible information on these pages. Nevertheless, if you see anything in the documentation that is not correct, does not match your experience or requires further clarification, please create a request (type : request for information) via our portal (https://sciensano.service-now.com/sp) or send us an e-mail via firstname.lastname@example.org to report this documentation issue. Please, do not forget to mention the URL or web address of the page with the documentation issue. We will then adjust the documentation as soon as possible. Thank you!