docs.healthdata.beDocumentation for CSV Upload on Architecture 2.0
Description of the service
The CSV upload functionality gives the possibility to import multiple parameters from a set of patients in one time into HD4DP2.0. The csv-file is based on an extract of the electronic patient files and/or other local databases.
Currently there is no user-interface in HD4DP2.0 for uploading csv-files. If a data provider wants to upload a csv-file, the file has to be dropped at a specific location. These files will be picked up and processed periodically. The files should be “final”, meaning that no application is writing to them. The pickup-location will be identical for all registries.
Pre-registry handling will be based on a naming-convention of the csv file.
HOW TO: Upload data using CSVUpload
Steps To Upload data
1. Prepare the CSV file (example file in this section)
- Extract the CSV file from the electronic patient files and/or other local databases.
- Make sure the name of the CSV file has the correct format: HD_DCD_submcsv_HDBPnumber_HDBPabbreviation_versionnumber_versionreleasedate
So for Endobronchial valve - Primo-implantation the format would be : HD_DCD_submcsv_HD0231_PNEUMOLOGY_ENDOBRONCHIAL_VALVE_PRIMO_IMPLANTATION_Pneumology_Endobronchial_valve_Primo_implantation_01_06262022.csv
Please find here an example file for Endobronchial valve - Primo-implantation:
Created,Modified,TX_COAUTHOR,TX_REGN_CD,CD_NIC_TPE,CD_BILL,IDC_PAT,FL_IDC_PAT_GENER,TX_IDC_PAT_INT,D_PAT_DOB,CD_PAT_SEX,CD_PAT_DECEA,D_PAT_DOD,TX_PAT_FIRST_NAM,TX_PAT_LAST_NAM,CD_PAT_PLC_RESDC,CD_DATA_PROV,CD_HPIN,MS_PAT_WGHT,D_PAT_WGHT,MS_PAT_HGHT,D_PAT_HGHT,MS_PAT_BMI,D_SMOK_STP,MS_SMOK_YRS,MS_EXCB,MS_EXCB_HOSPZ,CD_LABA,CD_LAMA,CD_SABA,CD_SAMA,CD_INHAL_CORTIC,CD_SYSTMC_STERD,CD_AZI,CD_CHRON_O2,MS_FVC_ABSO,MS_FVC_PERC,MS_FEV1_ABSO,MS_FEV1_PERC,MS_RV_ABSO,MS_RV_PERC,MS_TLC_ABSO,MS_TLC_PERC,MS_TLCO_ABSO,MS_TLCO_PERC,MS_6MWT,MS_MMRC,MS_CATSCORE,MS_BODE,CD_ANATCL_LOC,CD_EMPHYS_LTRLTY,CD_EMPHYS_HOMO_LL,CD_EMPHYS_HETERO_LL,CD_EMPHYS_DUL_LL,CD_EMPHYS_HOMO_RL,CD_EMPHYS_HETERO_RL,CD_EMPHYS_DUL_RL,CD_ANAL_TPE_FISSINT,CD_VISANAL_HORZNTL_RL,CD_VISANAL_DIAGNL_RL,CD_VISANAL_DIAGNL_LL,CD_QUANTANAL_HORIZNTL_RL,CD_QUANTANAL_DIAGNL_RL,CD_QUANTANAL_DIAGNL_LL,D_PROC_EBV,CD_PROC_TPE,CD_PROC_ANATCL_LOC,CD_QUANTANAL_COLLATVENTI,CD_EBV_MSDEVICE_TPE,CD_EBV_MSDEVICE_VFROM,CD_EBV_MSDEVICE_VTO,CD_OUTCM_LUL,CD_OUTCM_LLL,CD_OUTCM_RUL,CD_OUTCM_RML,CD_OUTCM_RLL,CD_LOBE,CD_EBV_TPE,CD_EBV_VFROM,CD_EBV_VTO,MS_EBV_AMNT,CD_CONT_TPE,D_ADMI,D_DISCH,MS_COMPLIC_ICU_HRS,CD_COMPLIC_PNEUTH,CD_PROB_TPE_PNEUTH,D_PNEUTH_START,D_PNEUTH_STP,CD_PNEUTH_TRTMNT,CD_COMPLIC_INFCT,CD_PROB_TPE_INFCT,D_INFCT_START,D_INFCT_STP,CD_COMPLIC_ACUT_EXCB,CD_PROB_TPE_ACUT_EXCB,D_ACUT_EXCB_START,D_ACUT_EXCB_STP,CD_COMPLIC_READM,CD_PROB_TPE_READM,D_READM_START,D_READM_STP,MS_MIGR_VALVE,MS_REM_VALVE,CD_REM_VALVE_REAS,CD_STATUS_REC,TX_LANG,projectId,CD_VISANAL_HORIZNTL_RL,CD_EBV_TPE,CD_LOBE,MS_EBV_AMNT
31/03/2021,31/03/2021,frank@example.com,,67809,67826,,TRUE,sdsd,31/03/2021,4522,4520,,Gaurav,Bhoyar,3,196,7381,1,,20,,1,31/03/2021,1,1,1,4516,4515,4515,4515,4515,4515,67760,67745,1,1,1,1,1,1,1,1,1,1,1,1,1,1,67805,67770,4515,4515,4515,,,,67741,,67739,67739,,,,31/03/2021,67827,67803,67827,,,,,,,,,67777,67841,,,1,67815,31/03/2021,31/03/2021,1,67747,,,,,67762,,,,67769,,,,4515,,,,1,1,67788,status_submitted,en,4,67739,67841,67777,1
Disclaimer: The example files above are only provided as a guideline and do not contain real life data.
2. Uploading the CSV File
Step 1: Open the sftp tool like WinSCP
Step 2: Get the credentials (Host name, Port number, User name and Password) from the IT department of the Data Provider, to log on to the sftp server located on the Data Provider side.
Step 3: Fill in the credentials into the Login screen and click on Login to be able to access the different upload folders:
Note: a warning might be given, just click on Update
Now the CSVUpload folder structure is displayed on the right-hand side panel:
Step 4: Select the project folder Endobronchialvalve-4 and open it by double-clicking on it:
Step 5: Double-click on the DCD folder to open it:
Step 6: Now go to the folder on the left-hand side panel where the CSV file to be uploaded is located:
Step 7: Drag the CSV file to be uploaded from the left-hand side panel into the folder on the right-hand side panel:
Step 8: Wait until the polling system of the CSV Uploader has picked up the CSV file and has processed it.
Once the CSV file has been processed it will disappear from the folder, when we refresh the page manually!
3. Validate CSV Upload
Once the CSV file has been processed 3 folders will be created (if they haven't been created already) in the DCD folder:
- ARCHIVE (after a csv file has been processed, the original csv file will be saved in this folder)
- RESULT (when the csv file is placed in this folder, it means that the csv file has been processed, a file will be created or (or appended, if the file already existed) with the result of the upload of the csv file).
All the errors that are described in th this file are business related, which means that they are technically correct, but in violation with the business rules or contains wrong values for that field. - ERROR (when the csv file is placed in this folder, it means that a technical error has occurred like the csv file contained erroneous formatting. The csv file won't get processed and an error file will be created with the errors and reason why the csv file couldn't be processed)
3.1 Validation of the CSV Upload via sftp tool:
Step 1: Double-click on the ERROR folder to open it, click on the refresh button and verify that there is no error file present.
Step 2: Return to the DCD folder. Now double-click on the RESULT folder to open it, click on the refresh button and verify that the result file is present.
Step 3: Double-click on the result file to open it.
Step 4: If there are multiple records in the result file, scroll to the entry of the current csv upload by looking at the upload date (Started at dd/mm/yyyy hh:mm).
Verify the result file that the upload was successful by searching for the word SUCCESS and having a look at the Status. This Status must contain: Success;Success Count:1;Error Count:0
3.2 Validation of the CSV Upload via HD4DP 2.0:
Step 1: Open the web application HD4DP 2.0.
Step 2: Select the concerned organization in the dropdown list and click on Volgende (Next)
Step 3: Fill in the username and password, that has been provided by your IT Department or Healthdata team, and click on Log in to access the HD4DP 2.0 application.
Step 4: Navigate in the menu on the left-hand side panel to the desired study program:
Step 5: Check that the uploaded registration(s) is/are displayed in the overview table.
