Detección de duplicados: una guía metodológica

Vista sencilla de ítem
dc.contributor.authorAmón Uribe, Ivánspa
dc.contributor.authorJiménez, Claudiaspa
dc.contributor.cvlacAmón Uribe, Iván [0000703796]spa
dc.contributor.googlescholarJiménez, Claudia [tXMokdIAAAAJ]spa
dc.contributor.orcidJiménez, Claudia [0000-0002-3741-320X]spa
dc.date.accessioned2020-10-27T00:20:38Z
dc.date.available2020-10-27T00:20:38Z
dc.date.issued2010-12-01
dc.description.abstractCuando una misma entidad del mundo real se almacena más de una vez, a través de una o varias bases de datos, en tuplas con igual estructura pero sin un identificador único y éstas presentan diferencias en sus valores, se presenta el fenómeno conocido como detección de duplicados. Para esta tarea, se han desarrollado múltiples funciones de similitud las cuales detectan las cadenas de texto que son similares mas no idénticas. En este artículo se propone una guía metodológica para seleccionar entre nueve de estas funciones de similitud (Levenshtein, Brecha Afín, Smith-Waterman, Jaro, Jaro-Winkler, Bi-grams, Tri-grams, Monge-Elkan y SoftTF-IDF) la más adecuada para un caso específico o situación particular, de acuerdo con la naturaleza de los datos que se estén analizando.spa
dc.description.abstractenglishWhen the same real-world entity is stored more than once, across one or more several databases, in tuples with the same structure but without a unique identifier and these present differences in their values, the phenomenon known as detection of duplicates. For this task, multiple similarity functions have been developed which they detect text strings that are similar but not identical. This article proposes a methodological guide to selecting among nine of these similarity functions (Levenshtein, Affine Gap, Smith-Waterman, Jaro, Jaro-Winkler, Bi-grams, Tri-grams, Monge-Elkan and SoftTF-IDF) the most suitable for a specific case or situation according to the nature of the data being analyzed.eng
dc.format.mimetypeapplication/pdfspa
dc.identifier.instnameinstname:Universidad Autónoma de Bucaramanga UNABspa
dc.identifier.issn2539-2115
dc.identifier.issn1657-2831
dc.identifier.repourlrepourl:https://repository.unab.edu.co
dc.identifier.urihttp://hdl.handle.net/20.500.12749/8942
dc.language.isospaspa
dc.publisherUniversidad Autónoma de Bucaramanga UNAB
dc.relationhttps://revistas.unab.edu.co/index.php/rcc/article/view/1387/1332
dc.relation.referencesI.Amón y C. Jiménez. Hacia una Metodología para la selección de técnicas de depuración de datos. Avances en Sistemas e Informática, 6(1):185-190, 2009.
dc.relation.referencesA.N. Arslan y O. Eğecioğlu, A New Approach to Sequence Comparison: Normalized Sequence. Alignment. Bioinformatics, 17(4):327-337, 2001.
dc.relation.referencesR. Baeza-Yates y G.H. Gonnet. A new approach to Text Searching.Communications ofthe ACM, 35(10):74-82, 1992.
dc.relation.referencesM. Bilenko y R.J. Mooney. Learning to Combine Trained Distance Metrics for Duplicate Detection in Databases. En Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discoveryand Data Mining, pags. 39- 48, 2003.
dc.relation.referencesE. Breimer y M. Goldberg. Learning Significant Alignments: An Alternative to Normalized Local Alignment. En Proceedings of the 13th International Symposium on Foundations of Intelligent Systems, pags. 37-45, 2002.
dc.relation.referencesP. Christen. A Comparison of Personal Name Matching: Techniques and Practical Issues. En Sixth IEEE International Conference on Data Mining, pags. 290-294, 2006.
dc.relation.referencesW.W. Cohen, P. Ravikumarand y S.E. Fienberg. A Comparison of String Distance Metrics for Name-Matching Tasks. En International Joint Conference on Artificial Intelligence, pags. 73-78, 2003.
dc.relation.referencesJ.D.Cohen. Recursive Hashing Functions for n-Grams, ACM Transactions on Information Systems, 15(3):291-320, 1997.
dc.relation.referencesW.W. Cohen. Integration of Heterogeneous Databases without Common Domains Using Queries Based on Textual Similarity. En Proceedings of the SIGMOD International Conference Management ofData SIGMOD'98, pags. 201-212, 1998.
dc.relation.referencesR. da Silva et al. Measuring Quality of Similarity Functions in Approximate Data Matching.Journal ofInformetrics, 1(1):35-46, 2007.
dc.relation.referencesH.L. Dunn. Record Linkage. Americal Journal of Public Health, 36(12): 1412-1416, 1946.
dc.relation.referencesO. Eğecioğlu y M. Ibel. ParallelAlgorithms for Fast Computation of Normalized Edit Distances. En Proceedings of the 8th IEEE Symposium on Parallel and Distributed Processing, pags. 496- 503, 1996.
dc.relation.referencesA.K. Elmagarmid; P.G. Ipeirotis y V.S. Verykios. Duplicate Record Detection: A Survey. IEEE Transactions on Knowledge and Data Engineering, 19(1):1-40, 2007.
dc.relation.referencesI.P. Fellegi yA.B. Sunter.ATheory for Record Linkage.Journal of the American StatisticalAssociation, 64(328):1183-1210, 1969.
dc.relation.referencesA. Gelbukh et al. Generalized Mongue-Elkan Method for Approximate Text String Comparison. En Proceedings of the 10th International Conference on Computational Linguistics and Intelligent Text Processing, pags. 559-570, 2009.
dc.relation.referencesO. Gotoh. An Improved Algorithm for Matching Biological Sequences.Journal ofMolecular Biology, 162(3):705-708, 1982.
dc.relation.referencesC.A. Heuser; F.N. Krieser y V.M. Orengo. SimEval - A Tool for Evaluating the Quality of Similarity Functions. En Tutorials, posters, panels and industrial contributions at the 26th International Conference on Conceptual Modeling, pags. 71-76, 2007.
dc.relation.referencesH. Hyyrö. A Bit-vector Algorithm for Computing Levenshtein and Damerau Edit Distances. En The Prague Stringology Conference '02, pags. 29-39, 2002.
dc.relation.referencesM.A. Jaro. Unimatch: A Record Linkage System User's Manual. Technical report, Washington, D.C.: US Bureau of the Census, 1976.
dc.relation.referencesH. Keskustalo et al. Non-adjacent Digrams Improve Matching of Cross-Lingual Spelling Variants. En International Symposium on String Processing and Information Retrieval, pags. 252-256, 2003.
dc.relation.referencesV.I. Levenshtein. Binary Codes Capable of Correcting Deletions, Insertions, and Reversals. Soviet Physics Doklady, 10(8):707- 710, 1966.
dc.relation.referencesR. Lowrence y R.A. Wagner.An Extension of the String-to-String Correction Problem.Journal ofthe ACM, 22(2):177-183, 1975.
dc.relation.referencesA. Marzal y E. Vidal. Computation of Normalized Edit Distance and Applications. IEEE Transactions on Patern Analysis and Machine Intelligence, 15(9): 926-932, 1993.
dc.relation.referencesW.J. Masek. A Faster Algorithm for Computing String Edit Distances. Journal of Computer and System Sciences, 20(1):18- 31, 1980.
dc.relation.referencesA.E. Monge y C.P. Elkan. The Field Matching Problem: Algorithms and Applications. En Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, pags. 267-270, 1996.
dc.relation.referencesE. Moreau; F. Yvon y O. Cappé. Robust Similarity Measures for Named Entities Matching. En Proceedings of the 22nd International Conference on Computational Linguistics, pags. 593-600, 2008.
dc.relation.referencesS.B. Needleman y C.D. Wunsh. A General Method Applicable to the Search for Similarities in the Amino Acid Sequence of Two Proteins,J Mol Biol, 48(3):443–453, 1970.
dc.relation.referencesH. Newcombe y J. Kennedy. Record Linkage: Making Maximum Use of the Discriminating Power of Identifying Information, Communications ofthe ACM, 5(11):563- 566, 1962.
dc.relation.referencesH. Newcombe et al. Automatic Linkage of Vital Records. Science, 130(3381): 954-959, 1959.
dc.relation.referencesJ.J.Pollock y A. Zamora. Automatic Spelling Correction in Scientific and Scholarly Text. Communications of the ACM, 27(4):358-368, 1984.
dc.relation.referencesE. Ristad y P. Yianilos. Learning string edit distance. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(5): 522-532, 1998.
dc.relation.referencesT.F. Smith y M.S. Waterman. Identification of Common Molecular Subsequences. Journal of Molecular Biology, 147(1):195-197, 1981.
dc.relation.referencesE. Sutinen y J. Tarhio. On Using Q-Gram Locations in Approximate String Matching. En Proceedings of the Third 22 Iván Amón, Claudia Jiménez Annual European Symposium on Algorithms, pags. 327-340, 1985.
dc.relation.referencesE. Vidal; A. Marzal y P. Aibar. Fast Computation of Normalized Edit Distances. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(9):899-902, 1995.
dc.relation.referencesR.A. Wagner y M.J. Fischer. The String-to-String Correction Problem.Journal ofthe ACM, 21(1):168-173, 1974.
dc.relation.referencesA. Weigel y F. Fein. Normalizing the Weighted Edit Distance. En Proceedings of the 12th IAPR International Conference on Pattern Recognition, pags. 399-402, 1994.
dc.relation.referencesW.E. Winkler. Frequency-Based Matching in the Fellegi-Sunter Model of Record Linkage. En Proceedings of the Section on SurveyResearch Methods, pags. 778-783, 1989.
dc.relation.referencesW.E. Winkler. String Comparator Metrics and Enhanced Decision Rules in the Fellegi-Sunter Model of Record Linkage. En Proceedings ofthe Section on SurveyResearch Methods, pags. 354-359, 1990.
dc.relation.referencesW.E. Winkler. Improved Decision Rules in the Felligi-Sunter Model of Record Linkage. En Proceedings of the Section on SurveyResearch Methods, pags. 274-279, 1993.
dc.relation.referencesW.E. Winkler. Using the EM Algorithm for Weight Computation in the Fellegi-Sunter Model of Record Linkage. En Proceedings ofthe Section on SurveyResearch Methods, pag. 667-671, 2000.
dc.relation.referencesW.E. Yancey. Evaluating String Comparator Performance for Record Linkage. En Proceedings of the Fifth Australasian Conference on Data mining and Analytics, pags. 21-23, 2006.
dc.relation.referencesL. Yujian y L. Bo. A Normalized Levenshtein Distance Metric. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(6):1091-1095, 2007.
dc.relation.urihttps://revistas.unab.edu.co/index.php/rcc/article/view/1387
dc.rightsDerechos de autor 2010 Revista Colombiana de Computación
dc.rights.accessrightsinfo:eu-repo/semantics/openAccessspa
dc.rights.creativecommonsAttribution-NonCommercial-ShareAlike 4.0 International*
dc.rights.urihttp://creativecommons.org/licenses/by-nc-sa/4.0/*
dc.rights.urihttp://creativecommons.org/licenses/by-nc-nd/2.5/co/
dc.rights.urihttp://creativecommons.org/licenses/by-nc-nd/2.5/co/
dc.sourceRevista Colombiana de Computación; Vol. 11 Núm. 2 (2010): Revista Colombiana de Computación; 7-23
dc.subjectInnovaciones tecnológicas
dc.subjectCiencia de los computadores
dc.subjectDesarrollo de tecnología
dc.subjectIngeniería de sistemas
dc.subjectInvestigaciones
dc.subjectTecnologías de la información y las comunicaciones
dc.subjectTIC´s
dc.subject.keywordsTechnological innovationseng
dc.subject.keywordsComputer scienceeng
dc.subject.keywordsTechnology developmenteng
dc.subject.keywordsSystems engineeringeng
dc.subject.keywordsInvestigationseng
dc.subject.keywordsInformation and communication technologieseng
dc.subject.keywordsICT'seng
dc.subject.keywordsData cleansingeng
dc.subject.keywordsData preprocessingeng
dc.subject.keywordsData qualityeng
dc.subject.keywordsDuplicate detectioneng
dc.subject.keywordsSimilarity functionseng
dc.subject.lembInnovaciones tecnológicasspa
dc.subject.lembCiencias de la computaciónspa
dc.subject.lembDesarrollo tecnológicospa
dc.subject.lembIngeniería de sistemasspa
dc.subject.lembInvestigacionesspa
dc.subject.lembTecnologías de la información y la comunicaciónspa
dc.subject.proposalLimpieza de datosspa
dc.subject.proposalPreprocesamiento de datosspa
dc.subject.proposalCalidad de datosspa
dc.subject.proposalDetección de duplicadosspa
dc.subject.proposalFunciones de similitudspa
dc.titleDetección de duplicados: una guía metodológicaspa
dc.title.translatedDuplicate detection: a methodological guideeng
dc.type.coarhttp://purl.org/coar/resource_type/c_7a1f
dc.type.driverinfo:eu-repo/semantics/article
dc.type.hasversioninfo:eu-repo/semantics/acceptedVersion
dc.type.localArtículospa
dc.type.redcolhttp://purl.org/redcol/resource_type/CJournalArticle

Archivos

Bloque original

Mostrando 1 - 1 de 1
Miniatura
Nombre:
2010_Articulo_Deteccion de duplicados una guia metodologica.pdf
Tamaño:
361.72 KB
Formato:
Adobe Portable Document Format
Descripción:
Artículo
Descargar

Colecciones

Revista Colombiana de Computación