Process

Process manager Thu, 11/11/2021 - 17:04

Create and submit in HD4DP v1

Create and submit in HD4DP v1

The Data Collection view leads a user to the existing registers. A user can start collecting data from this view.

Creating and submitting a registration consists of 4 steps:

  • Select the register for creating a registration.
  • Click on "New registration" button.
  • Fill in the registration form and save or submit the registration
  • Save the registration temporarily if needed. The status of the record will be "Open" for saved registrations.​
  • Submit the registration if  no validation errors present. The status of the registration will change to "Sending".
  • When the record has been processed, the status will change to "Submitted".

As soon as a registration is sent, the "status confirmation" column will show "Pending".

  • ​It shows "OK" once the registration has arrived, and "NOK" if the registration did not arrive within 48 hours. If the status is "NOK", the software will automatically try to resend the registration up to 10 days after the initial submission.
  • For the statuses "Sending","Corrections needed" and "Approved" the status confirmation will be empty because the registration has not yet been submitted.
  • A registration can be "Reopened" if needed as long as the status of the registration is "Submitted".
manager Tue, 04/12/2022 - 11:54

Change or complete registrations

Change or complete registrations

A user can modify and complete a registration in four steps:

  • Select the register from which you wish to modify one or more registration(s).
  • Click on the registration to be changed, and complete the form in full. The status of the record is then:
    • Open for registrations saved manually or by uploading a csv file
    • Corrections needed for registrations with errors
  • Save registration temporarily if necessary. Record status becomes Open for stored registrations.
  • Send the registration if there are no more validation errors.
    • Registration status changes to Sending
  • When the record is processed, the status changes to Submitted.

Once a registration is sent, the message Pending appears in the status confirmation column.

  • When the registration is received, the message OK appears; if the registration is not received within 48 hours, the message NOK appears. With the status "NOK", the software will automatically try to send the registration up to 10 days after the initial submission.
  • For the statuses Sending, Corrections needed and Approved, the status confirmation is empty because the registration has not yet been sent.
  • A registration can be Reopened if necessary, as long as the status of the registration is "Submitted".
manager Thu, 12/08/2022 - 15:53

Registrations in status 'Corrections needed'

Registrations in status 'Corrections needed'

Registrations with status 'Corrections needed' need to be revised. The researcher has comments that require an update or revision.

  • Select the record with status 'Corrections needed'
  • Verify and react to the comments made by the researcher by selecting the ‘comments’ tab
  • Reply to the comments and modify the registration and save the comment
  • Click 'Re-submit' to send the corrections to the researcher. Alle validation errors need to be solved before the 'Re-submit' button becomes available
manager Thu, 11/11/2021 - 17:04

Registrations with status 'Follow-up needed'

Registrations with status 'Follow-up needed'

Some registrations need follow-up e.g. to monitor multiple consultations of a patient. Therefore additional information will be asked at predefined milestones, which are defined in the set-up phase of the registry. 
Follow-up registrations can be triggered by:

  • Timing e.g. x time after the initial registration
  • Answers in the initial registration e.g. only if the patient has taken medication X
  • A combination of both e.g. if the patient has taken medication X, ask follow-up information 3 months later


If the Status column shows 'Follow-up needed' and the 'Follow up' column is active, the user can add the extra information needed in the details of the registration.

Note: the process of validation and resending the registration(s) back for the follow-up data takes minimum 4 hours.

Select 'Follow up' in the settings of the view:

The Follow-up column is now shown in the view:

When follow-up is needed, the column 'Follow up' in the list view indicates if follow-up is needed by indicating:

  • Planned:  follow-up will be needed in the future, indicated by the date indication
    • 'Status Column' shows 'Submitted', no action is needed at this moment 
  • Active: the questions for this follow up are now visible and can be filled
    • 'Status Column' shows 'Follow-up needed', action is needed
  • Submitted:  the follow-up column will change to submitted after the additional information is given
    • 'Status Column' shows 'Submitted', no more action is needed


Depending on how many times follow-up is needed, a row is added to the column. The milestones are indicated by t1, t2 etc. 

manager Thu, 11/11/2021 - 17:04

Delete registrations

Delete registrations

Only registrations with status Open or Corrections needed can be deleted.

The following steps are required to delete the registration:

  • Select the record and version for which you wish to delete a registration
  • Select the registration(s) you wish to delete
  • Select the Actions button and select the Delete registrations option
  • Select Yes to confirm
manager Tue, 04/12/2022 - 12:30

Finalize the data collection

Finalize the data collection

The researcher needs to be notified if the data collection is ready. 

There are 2 ways to finalize the data collection:

1. By using the link in the Data Collection tab:

2. By using the button in the Participation tab:

  • Confirm finalization of the data collection:

Sometimes additional data is requested by the researcher. This must be completed first before using the finalize button or link.

After you select the finalize button, the researcher will be notified

manager Thu, 11/11/2021 - 17:05

Submit registrations in bulk

Submit registrations in bulk
  • The registrations that need to be submitted, have to be checked to proceed the registration in bulk
  • Click on the button 'Actions' and select 'Submit registrations'. The number of selected records is displayed between brackets​​
  • Confirm the submission on the window that appears
  • summary of the registrations that are submitted and failed is shown. The records that contain errors will not be submitted
manager Thu, 11/11/2021 - 17:05

Using the upload center

Using the upload center

The upload center is introduced to make the upload of multiple registrations more performant and user friendly.​

The link to the upload center can be reached by the main menu:

If uploads are processed, an overview with the name of the uploade file, the status and a visual representation of the status, the number of records that were processed, the way of uploading, the user and the upload date is saved in the main screen of the upload center is given to the user a quick and visual impression of the data uploaded in bulk by using the upload center:

In the right upper corner two buttons are shown.

  • Configuration: the default settings for a register can be set by any user that is authorized to participate in a register. 

Following options are available:

  • Conflict mode: what need to be done in case of conflict
  • Conflict master: is the data from the new registration of from the record in the database saved in case of conflict
  • Ignore duplicate records
  • Autosubmit: send the data automatically for processing in HD4RES
  • Validation: what need to be done after validation the registration:
    • None: only the registrations without errors will be saved
    • Save on validations: Save the registrations when there are validation errors upon uploading
    • Validate all: Only perform commit if non of its records have (errors or) validation errors

4 steps need to be executed for a new upload. When the titel bar of a separate step turns green, orange or red and the icon of the step in the overview, the step has respectively ended with success, with warnings or errors.

Step 1 - Select file

The first step is to select the CSV file with the data to be uploaded. The default settings for a register can be found in the Configurations on the main screen of the upload center. 

Be sure to deselect the "Use default configurations" when changing the default settings: 

For the BCR register, the possibility exists to upload two files and link them during the process:

  • Both documents will be validated separately. By using the link "Click here to see the linked upload" the user can switch between the linked documents. Refresh of push F5 on you to see the status of the upload.

After uploading succesfully, the title bar of this step will turn green.

Step 2 - Validity Check

Validation checks are executed for every record. 

Step 3 - Upload

The file is being uploaded and the result is shown in this step per line of data added:

Step 4 - Finalize 

The result is shown visually for all registrations and for all registrations in a CSV file in reports after finalization:

Succes report is a CSV file with all registrations who were processed succesfully.

In case of errors, a link to a detailed error report is shown too:

Please find below the movie that guides you through the functionalities of the Upload Center:

manager Tue, 04/12/2022 - 12:54

Create a CSV file

Create a CSV file

Processing registrations in bulk is done by uploading CSV files. These files are plain text files which can contain multiple registrations at once and are extracted from your primary system. 

The first step is to create the file correctly.

Using a CSV editor

While Excel is a fine tool to view CSV’s, we do not recommend it to edit CSVs. Instead use notepad++ or any other text editor. Here are a few risks you should be aware of when editing a CSV in Excel. Excel will interpret the content, which may lead to changes:

  • Leading zeros disappear in fields that are recognized as numeric fields
  • Entries like 3-9 can become March-9
  • ​The only accepted date format DD/MM/YYYY can be modified (e.g. To DD/MM/YY)
  • The decimal separator can differ from that in HD4DP, a semicolon wil lead to a correct upload
  • When saving a file as .csv, Excel uses the default field separator. HD4DP only accepts CSV with a semicolon as separator. This default setting can be adapted in the properties of your computer.
  • CSV encoding must be in UTF-8.

Setting up the document 

Every column in the CSV file needs to be recognized as a field of the register by the HD4DP application. Therefor each column in the file must be identical by the technical name of the field in the register.

Tip:
 Downloading (manually) entered data from HD4DP will guide you in formatting a CSV file and may help during the development of the CSV extraction from the primary systems.

The Data Collection Definition (DCD) specifications for a register and its fields are defined and documented on this documentation portal.

Each field in the form can be completed through a value in a CSV file.
An example field is “Date of last follow-up”, shown in the screenshot below.

This field is of the type “date” and is required (*). Within the technical documentation of this data collection, this is shown as follows:

To include this field in a CSV upload file, it is sufficient to create a column with the name “date_of_last_followup” and populate it with the appropriate data i.e. a date in the format dd/mm/yyyy.

Fields can be required, read-only and computed (automatically calculated). Fields can also have a default value.

This information is present in the detailed technical description for each data collection.

General requirements

  • The column separator is the semicolon (;)
  • The decimal separator for numbers is the comma (,)
  • The date format is dd/mm/yyyy

Basic content types

Depending on the type of the field, a different representation of the data is expected. The table below describes the different basic types and the rules on how to provide the content for these types.

Content typeExpected format/content
booleanTRUE, FALSE
datedd/mm/yyyy
choicecode from choice list
listoption1_label|option2_label|etc.
multilinefree text
numbernumber (decimal separator = ,)
patientIDSSIN number. If the person does not have a SSIN, leave this field blank.
questionnairecode from questionnaire answer list
text●   free text
●   if a binding reference list is used: a code from the reference list
●   if a non-binding reference list is used: a code from the reference list or free text
attachment●   expected format/content: Name of the file that must be attached (e.g. protocol.txt).
●   expected extension: .txt
●   file must be stored in the same folder as the folder that is used for the CSV-upload

Advanced content types

Other than these simple types, more complex data structures can be used, as shown in the table below. Each of these types is explained in more detail below the table.

Content typeCSV column nameExpected format
fields within fieldsetfieldset_label|field_labeldepending on the field type
list (1 field)list_label|fieldvalue1|value2|etc
list (block of fields)list_label|0|field1
list_label|0|field2
list_label|1|field1
list_label|1|field2
etc.
depending on the field type
nested fields below choice or multichoicechoice_label|nested_itemdepending on the field type

Fieldset

A fieldset is a collection of fields, as shown in the image below:

Anthropometry is the title of the fieldset, and this fieldset contains two fields, weight and height. Fieldsets fieldsets do not have a number cfr. image below - Anthropometry.

Sections do not have an impact on the CSV file, whereas fieldsets do. The title of the fieldset must be included in the field name column as follows: fieldset_label|field_label.

E.g. for the two Anthropometry fieldset fields weight and height below, the correct CSV column headers are: anthropometry|weight en anthropometry|height.

Lists

A list is also a collection of fields, like a fieldset, but with the additional property that the collection of fields can be repeated.

An example is shown in the image below: “Birthdays of the biological children for this patient” is a list. One list item consists of two fields, “Child birth month” and “Child birth year”. For each child, a list item can be added.

The CSV column names consist of the list header label and the field label (as for fieldsets), together with a counter to distinguish the different list items. The correct CSV column names for the two list items below are:

  • birthdays_of_the_biological_children_for_this_patient|00|child_birth_month
  • birthdays_of_the_biological_children_for_this_patient|00|child_birth_month
  • birthdays_of_the_biological_children_for_this_patient|01|child_birth_month
  • birthdays_of_the_biological_children_for_this_patient|01|child_birth_year

Please note that for every line, the numbers should increment, starting from 0 (|00|,|01|, .. is ok, |01|, |03|, ... is not). You can't have blank values for |00| and filled values for a higher number.

Please note that the numbering requires a stable format, meaning the number of characters used by the number has to be constant. You can't have one record using |00| and another using |0|. Generally we advice to use a string length of 2 digits.

For lists consisting of 1 field a simplified implementation is possible. The CSV column header only consists of the list header label and multiple values are provided in the one column, separated by a pipe (|).

E.g. for the list in the image below, the CSV column header is diagnosis_orphacode and the content of the column is 562|702. This is the example of a text field with a reference list: providing the codes of the reference list is sufficient.

Nested fields

Nested fields are fields that only become available when specific options are selected in the form. An example is shown below: the field “Specify” only becomes available if the checkbox “Other” is marked. These fields also have a combined CSV column header, consisting of the choice list label and the field label. For the example below, the correct CSV column header is hence base_of_diagnosis|specify.

Attachments

When a CSV is prepared and put in the provisioning folder, it can contain references to attachments for data collections that specifically allow this.

These references are relative paths to the file location. If such a reference is present in the CSV file, the attachment content is uploaded and linked to the created registration. The attachment is then available in the HD4DP client as well as in the HD4RES client when the registration is submitted.

The maximum file size for attachments is 6 MB

If a data collection permits you to send attachments you should have the column name to use in the CSV. If not, you should be able to find it at https://www.healthdata.be/dcd/#/collections or you can contact Support.Healthdata@sciensano.be.

Add the column name to the header of the CSV and add the file names as values in the column.
Example: “picture.png”

Put the CSV file in the correct provisioning folder (organization sub folder, then in the register sub folder), along with a “picture.png” file of your choice. The application picks the CSV file and creates a new registration.

Open the registration and verify the attachment has correctly been uploaded.

manager Tue, 04/12/2022 - 13:32

Upload a CSV file

Upload a CSV file

There are 3 ways to upload CSV files with data into HD4DP v1:

  • Manually
    • In de HD4DP application
    • In the specific folders on the server (provisioning)
  • Automatically
    • in the specific folders on the server (fasttrack)

The file can be maximum 6 MB

When using the manual upload by using te application or putting files on the server (provisioning), the registrations still need to be manually submitted.

When using the fasttrack methodsubmission is done automatically.

Manually in the HD4DP application

Data can be uploaded by using the same approach as we use for the stable data.

More information can be found in the article "CSV download and upload for stable data".

Manually in the application folders

HD4DP allows to import files from the folders used by the application.

Save the created CSV file in the correct register folder of your organization or sub-organisation. The folders of the sub-organizations are to be find in the subfolders of the main organization.

When uploading a CSV, a check is performed for each record if any corresponding record is already available in the application. The check for corresponding data is based on the unique id. If a record is already available in the application a merge takes place between the existing record and the new CSV. In case of conflicts, the values of the new CSV are implemented.

The uploaded registrations will be automaticallyuploaded in the HD4DP application.

They will need to be manually validated and submitted in the application.

Automatically in the application folders

The goal of the CSV fast track is to automatically transmit complete valid CSV files to healthdata without human intervention. The following topics must be taken into account when using the CSV fast track. HD4DP allows to automatically import files from dedicated folders. On the server where HD4DP is installed, create a new folder next to the provisioning and stable data folders.

After creating the fast track folder, you must configure the location of the dedicated folder in HD4DP by adapting the configuration parameter "FAST_TRACK_DIRECTORY" in the admin configuration panel

Subfolders contributing to the organization and data collections, will  automatically be created according to a specific folder structure. 

In this video a more profound explanation about how to set up the fast-track is given:

When a CSV file is placed in the folder of a data collection, several actions can occur: 

  • If the file could not be manipulated (e.g. an invalid CSV or a data collection that has not yet started) the file is moved to ARCHIVE_ERROR
  • If the file was successfully read but only partially valid, the file will be split accordingly.
    • e.g: line1 is valid, line2 ignored because of duplicate record, line3 contains an error
      • a CSV file containing line1 will be placed in SUCCESS
      • a CSV file containing line2 will be placed in IGNORE
      • a CSV file containing line3 will be placed in ERROR and an extra column is added which explains why the CSVs raised an error.
      • the original CSV will be moved to ARCHIVE
  • If the file is successfully read and validated, it is moved entirely to both the SUCCESS folder and the ARCHIVE folder.
manager Thu, 11/11/2021 - 17:06

Using a CSV file : tips and tricks

Using a CSV file : tips and tricks

This document goes through a list of issues which users could encounter when uploading a CSV file in HD4DP v1. The article provides by tips and tricks to overcome these issues.

How should I create a CSV file?

Several methods can be used for creating a .csv file :

  • Download the .csv file for your data collection on https://www.healthdata.be/dcd/#/collections
  • Consult this article first
  • Use a spreadsheet  (e.g. Microsoft Excel, OpenCalc)
  • Save the document using one of the 2 following methods :
    • Use the spreadsheet and make sure the file saved use semi-column (;) as separator to fitting the standard of healthdata.be. Depending of the spreadsheet used, the creation of the .csv file containing semi-columns as file separators, could be carried out in 2 steps :
      • save the file as .csv file containing commas as column separators
      • edit the .csv file with a text editor and replace the commas by semi-column
    • Copy the data in a note editor and save it as .csv file
      • on Microsoft : notepad, notepad++
      • on Macintosh : textedit;app

How to find the technical requirements for a field in the registration?

  • Consult the registration requirements for each field in the data collection definition, as shown below :
    • Field Type e.g. text, boolean, choice, date, patient
    • CSV column_name 
    • Required : 'true' means that is a mandatory field and it has be present within the .csv extract
  • Postal code missing : use the value 999
  • Postal code for people living abroad : 9999

How to perform a massive deletion of records with HD4DP v1?

  • If your HD4DP version is not yet version 1.10.3 (consult this website for getting informed with HD4DP version), contact support.healthdata.be@sciensano.be
  • For future releases, the functionality is available

How should I treat NISS code with leading zeros when creating my csv?

 For treating leading 0s of a national registry number (NISS), you can fill in the field 'patient_id' by adding a simple quote (')  before the leading zeros as follows:

  • as is : 0007788..
  • to be: '0007788...

We advice the use of Excel TEXT function for formatting your NISS field for adding leading zeros. You can consult the following article for more information.

Can I use both Windows and Macintosh file for creating a csv file?

  • From the HD4DP version 1.10.3, you create your CSV file can either on Windows or Macintosh
  • For older versions, you can create a CSV on Macintosh buy you have to save it on a Windows environment before uploading it in HD4DP

How should I format the date fields within my csv extract?

The date format has to be dd/mm/yyyy, therefore  make sure the date format is correct by consulting your .csv file before uploading. Open the file with a text editor and not a spreadsheet and check the format date.

How should I proceed with deceased patient?

  • Put a value in the field 'patient | deceased' 
    • TRUE if patient deceased
    • FALSE if patient alive
  • Put the date of death in the field 'patient_id | date_of_death'  if the patient is deceased 

How can I fix errors with the automatic generation of Patient_ID?

  • If you do not dispose of a national number of the patient, a patient_id will be generated automatically. This is called a 'codepat' and you can find more information in the article "What is a codepat and how is it calculated?
  • If you use a codepat, you should use the substrings function for extracting the name and first name from the codepat field to fill in the strings from the 'CODEPAT': 
    • patient_id | name
    • patient_id | first_name
  • Below, an example of use of substring functions directly applied on the 'CODEPAT' for carrying this information extraction
    • patient_id | name  = MID(cell,9,2)   
    • patient_id | first_name =MID(cell,11,2)
    • patient_id | generated has to set to 'TRUE'
    • patient_id | sex has to be set to 'M' (Male) or 'F'(Female)
  • Test this procedure using the following csv extract, you should get the 'patient_id' filled in automatically

​Example:CODEPAT;patient_id;patient_id|generated;patient_id|internal_patient_id;patient_id|
name;patient_id|first_name;patient_id|date_of_birth;patient_id|sex;patient_id|deceased;
patient_id|place_of_residence;HIVType;RNA_DATE;RNA_VALUE;RNA_SPECIMEN;
RNA_ASSAY;DNA_PCR_DATE;DNA_PCR_VALUE19711125JADAM;71112535971;;;;;25/11/1971;M;;2000;
HIV1;12/12/2016;25;B;99;;19711125JADAM;19711125JADAM;TRUE;;JA;DA;25/11/1971;M;;2000;
HIV1;05/10/2016;82;B;90;;

  • Pay attention cases when the substring function returned values as DE, DU IN, LA, LE, OP in the field 'patient_id | name', you should add one characteristic 

e.g. LE could become LEE or DE could become DEE 

Which norm of country code should I use in my csv extract?

Fill in the field 'NATION' using the the 2-digits ISO standard 3166alpha2 format.

How can I make sure that there are no spaces in cells left empty?

Make sure there is no spaces in cells left empty by searching the 'SPACE' characters (e.g. CTRL-F and push on the SPACE touch).
Remove all the spaces in cells left empty by replacing all the 'SPACE' characters by an empty string '' (e.g. CTRL-H and replace the 'SPACE' characters by an empty field '').

How can I treat the numbers with decimals?

The decimal separator must be the comma (e.g. 1,78 ; 1,89)

What are the not supported postal codes?

Postal CodeLocality
1007Assemblée de la Commission Communautaire Française
1005Assemblée Réunie de la Commission Communautaire
1099Bruxelles X
1008Chambre des Représentants
1046European External Action Service
1041International Press Center
1035Ministère de la Région de Bruxelles Capitale
1031Organisations Sociales Chrétiennes
1110OTAN
1012Parlement de la Communauté française
1047Parlement Européen
1100Postcheque
1006Raad van de Vlaamse Gemeenschapscommissie 
1044RTBF
1033RTL-TVI
612Saint-Nicolas
1101Scanning
1009Senat de Belgique
1105SOC
1048Union Européenne - Conseil
1011Vlaams parlement
1043VRT

What values to use when postal code is missing or for patient living abroad?

Missing postal code : 999
Patient living abroad : 9999

manager Thu, 11/11/2021 - 17:06

Reuse 'stable' data

Reuse 'stable' data

Stable data are data that remains the same or ‘stable’ over time e.g. birht data.
Since one of the main objectives of Healthdata is to reduce the registration effort, we want to avoid that this stable data has to be entered manually over and over again.

Stable data can be reused in 4 ways:

  • At first data collection via HD4DP
  • After 1st data collection period via HD4DP
  • Across registers
  • By recoding registers

At first data collection via HD4DP v1

  • Only for registers that used eHealth coded NISS before healthdata (BCFR, BNMDR, IQED, IQECAD) 
  • For these registers, a recoding needs to be done by eHealth: from the ‘old’ project-specific key to the healthdata key
  • Once this is done, a subset of relevant historical data is uploaded to HD4DP
  • In HD4DP v1, when creating a new registration with a NISS present in the historical data, the stable data is automatically enriched

After 1st data collection period via HD4DP

There are 2 options:

Enrichment on NISS

  • When creating a new registration for a patient that was already registered in a previous data collection period, the stable data for this patient will be automatically fetched and used to prefill the new registration
  • To check which fields are marked as ‘stable’, see www.healthdata.be/dcd 

Csv download & upload

  • Some registers (BCFR, BNMDR) prefer to create registrations in bulk with stable data
  • Stable data from previous data collection period can be exported cf. article "CSV download and upload for stable data"

Across registers

  • Currently only possible via manual csv download and upload  cf. article "CSV download and upload for stable data"
  • Will become automatic in Q2 or Q3 release.

Recode registers

  • For registers that did not use eHealth coded NISS as patient identifiers, but that wish to link the data collected without healthdata with the data collected with healthdata (longitudinal analysis)
  • A subset of relevant historical data is sent to HD4DP
  • Centres are asked to complete the registration with the NISS
  • Examples: BCFRecode, IQECAD_Recode, IQED_Recode
  • Researchers can not validate these data in the ‘standard’ way. This has to a manual job/interaction between DWH team and researcher
  • For clarifications to be done for IQED_Recode contact the support team support.healthdata@sciensano.be
manager Thu, 11/11/2021 - 17:06

CSV download and upload for stable data

CSV download and upload for stable data

Before starting a new data collection based on stable data of another data collection, you need to finalize the data collection which serves as the starting point of your new data collection cf. article "Finalize the data collection"

To start the new data collection select the new version in the Data collection section. In this case v3. 

Click the button "Go to the participation page":

  • Click on ‘Participate’:
  • The Start date in the Participation status will be updated and access to the new version is completed and the user can return to the Data Collection section:
  • Once the previous data collection is finalized and the new one has been started, you can download data from the first version and add those to the next version. 
  • Go to the Registrations tab:
  • Download the data by using the Download button "CSV Download All (Stable Data)":

When everything is downloaded, you can start uploading by Using the upload center.

manager Tue, 02/22/2022 - 08:43