A new computationally efficient algorithm for record linkage with field dependency and missing data imputation

Research output: Contribution to journalArticlepeer-review

Abstract

Record linkage algorithms aim to identify pairs of records that correspond to the same individual from two or more datasets. In general, fields that are common to both datasets are compared to determine which record-pairs to link. The classic model for probabilistic linkage was proposed by Fellegi and Sunter and assumes that individual fields common to both datasets are completely observed, and that the field agreement indicators are conditionally independent within the subsets of record pairs corresponding to the same and differing individuals. Herein, we propose a novel record linkage algorithm that is independent of these two baseline assumptions. We demonstrate improved performance of the algorithm in the presence of missing data and correlation patterns between the agreement indicators. The algorithm is computationally efficient and can be used to link large databases consisting of millions of record pairs. An R-package, corlink, has been developed to implement the new algorithm and can be downloaded from the CRAN repository.

Original languageEnglish
Pages (from-to)70-75
Number of pages6
JournalInternational Journal of Medical Informatics
Volume109
DOIs
Publication statusPublished - Jan 2018

Keywords

  • Conditional independence
  • EM-algorithm
  • Fellegi/Sunter
  • Log-linear models
  • Record linkage

Fingerprint

Dive into the research topics of 'A new computationally efficient algorithm for record linkage with field dependency and missing data imputation'. Together they form a unique fingerprint.

Cite this