Please refer to the evaluation plan below for the detailed tasks and relevant metrics.
|LRE17||S. O. Sadjadi, T. Kheyrkhah, A. Tong, C. S. Greenberg, D. A. Reynolds, E. Singer, L. P. Mason, and J. Hernandez-Cordero, “The 2017 NIST language recognition evaluation,” in Proc. Odyssey, Les Sables d´ Olonne, France, June 2018, pp. 82–89||10.21437/Odyssey.2018-12|
|LRE15||H. Zhao, D. Bans´e, G. Doddington, C. Greenberg, J. Hern´andez-Cordero, J. Howard, L. Mason, A. Martin, D. Reynolds, E. Singer, and A. Tong, “Results of the 2015 NIST language recognition evaluation,” in Interspeech 2016, San Francisco, USA, September 2016, pp. 3206–3210||10.21437/Interspeech.2016-169|
|LRE96, LRE03, LRE05, LRE07, LRE09, LRE11||A. F. Martin, C. S. Greenberg, J. M. Howard, G. R. Doddington, and J. J. Godfrey, “NIST language recognition evaluation - past and future,” in Odyssey 2014, Joensuu, Finland, June 2014, pp. 145–151||10.21437/Odyssey.2014-23|
|SRE21||Sadjadi, S.O., Greenberg, C., Singer, E., Mason, L., Reynolds, D. (2022) The 2021 NIST Speaker Recognition Evaluation. Proc. The Speaker and Language Recognition Workshop (Odyssey 2022), 322-329||10.21437/Odyssey.2022-45||CTS Challenge||Sadjadi, S.O., Greenberg, C., Singer, E., Mason, L., Reynolds, D. (2022) The NIST CTS Speaker Recognition Challenge. Proc. The Speaker and Language Recognition Workshop (Odyssey 2022), 314-321||10.21437/Odyssey.2022-44|
|SRE19||O. Sadjadi, C. Greenberg, E. Singer, D. Reynolds, L. Mason, and J. Hernandez-Cordero, “The 2019 NIST Audio-Visual Speaker Recognition Evaluation,” in Proc. The Speaker and Language Recognition Workshop (Odyssey 2020), 2020, pp. 259–265||10.21437/Odyssey.2020-37|
|SRE19 CTS Challenge||S. O. Sadjadi, C. Greenberg, E. Singer, D. Reynolds, L. Mason, and J. Hernandez-Cordero, “The 2019 NIST Speaker Recognition Evaluation CTS Challenge,” in Proc. The Speaker and Language Recognition Workshop (Odyssey 2020), 2020, pp. 266–272||10.21437/Odyssey.2020-38|
|SRE18||S. O. Sadjadi, C. S. Greenberg, E. Singer, D. A. Reynolds, L. P. Mason, and J. Hernandez-Cordero, “The 2018 NIST speaker recognition evaluation,” in Proc. INTERSPEECH, Graz, Austria, September 2019, pp. 1483–1487||10.21437/Interspeech.2019-1351|
|SRE16||S. O. Sadjadi, T. Kheyrkhah, A. Tong, C. S. Greenberg, D. A. Reynolds, E. Singer, L. P. Mason, and J. Hernandez-Cordero, “The 2016 NIST speaker recognition evaluation,” in Proc. INTERSPEECH, Stockholm, Sweden, August 2017, pp. 1353–1357||10.21437/Interspeech.2017-458|
|SRE96 - SRE06, SRE08, SRE10, SRE12||C. S. Greenberg, L. P. Mason, S. O. Sadjadi, and D. A. Reynolds, “Two decades of speaker recognition evaluation at the National Institute of Standards and Technology,” Computer Speech & Language, vol. 60, 2020||10.1016/j.csl.2019.101032|
|ivec15||A. Tong, C. Greenberg, A. Martin, D. Banse, J. Howard, H. Zhao,G. Doddington, D. Garcia-Romero, A. McCree, D. Reynolds, E. Singer,J. Hernandez-Cordero, and L. Mason, “Summary of the 2015 NIST language recognition i-vector machine learning challenge,” in Odyssey 2016:The Speaker and Language Recognition Workshop, Bilbao, Spain, June 21-24 2016, pp. 297–302||10.21437/Odyssey.2016-43|
|ivec14||D. Banse, G. R. Doddington, D. Garcia-Romero, J. J. Godfrey, C. S. Green-berg, A. F. Martin, A. McCree, M. A. Przybocki, and D. A. Reynolds,“Summary and initial results of the 2013-2014 speaker recognition i-vectormachine learning challenge,” inProc. INTERSPEECH, Singapore, Singa-pore, September 2014, pp. 368–372||10.21437/Interspeech.2014-86|
|LRE Homepage||NIST Language Recognition Evaluation||nist.gov/itl/iad/mig/language-recognition|
|SRE Homepage||NIST Speaker Recognition Evaluation||nist.gov/itl/iad/mig/speaker-recognition|
|Normalized Cross-Entropy paper||A tutorial introduction to the ideas behind Normalized Cross-Entropy and the information-theoretic idea of Entropy||nist.gov/file/411831|
|SPHERE sw||Speech file manipulation software (SPHERE) package version 2.7, 2012||nist.gov/itl/iad/mig/tools|
|Babel data||M. P. Harper, "Data resources to support the Babel program,"||https://goo.gl/9aq958|
|DET curves||A. F. Martin, G. R. Doddington, T. Kamm, M. Ordowski, and M. A.Przybocki, "The DET curve in assessment of detection task performance," inProc. EUROSPEECH, Rhodes, Greece, September 1997, pp. 1899–1903||10.21437/Eurospeech.1997-504|
|SWB-1, rel2||J. Godfrey and E.Holliman, "Switchboard-1 Release 2," 1993||catalog.ldc.upenn.edu/LDC97S62|
|SWB-2, Pt1||D. Graff, A. Canavan, and G. Zipperlen, "Switchboard-2 Phase I," 1998||catalog.ldc.upenn.edu/LDC98S75|
|SWB-2, Pt2||D. Graff, K. Walker, and A. Canavan, "Switchboard-2 Phase II," 1999||catalog.ldc.upenn.edu/LDC99S79|
|SWB-2, Pt3||D. Graff, D. Miller, and K. Walker, "Switchboard-2 Phase III," 2002||catalog.ldc.upenn.edu/LDC2002S06|
|SWBCell, Pt1||D. Graff, K. Walker, and D. Miller, "Switchboard Cellular Part 1 Audio," 2001||catalog.ldc.upenn.edu/LDC2001S13|
|SWBCell, Pt2||D. Graff, K. Walker, and D. Miller, "Switchboard Cellular Part 2 Audio," 2004||catalog.ldc.upenn.edu/LDC2004S07|
|Fisher Eng Train, Pt1 Speech||C. Cieri, D. Graff, O. Kimball, D. Miller, and K. Walker, "Fisher English Training Speech Part 1 Speech," 2004
C. Cieri, D. Miller, and K. Walker, "The Fisher corpus: A resource for the next generations of speech-to-text," inProc. LREC, Lisbon, Portugal, May2004, pp. 69–71
|Fisher Eng Train, Pt1 Transcripts||C. Cieri, D. Graff, O. Kimball, D. Miller, and K. Walker, "Fisher English Training Speech Part 1 Transcripts," 2004||catalog.ldc.upenn.edu/LDC2004T19|
|Fisher Eng Train, Pt2 Speech||C. Cieri, D. Graff, O. Kimball, D. Miller, and K. Walker,"Fisher English Training Speech Part 2 Speech," 2004||catalog.ldc.upenn.edu/LDC2005S13|
|Fisher Eng Train, Pt2 Transcripts||C. Cieri, D. Graff, O. Kimball, D. Miller, and K. Walker, "Fisher English Training Speech Part 2 Transcripts," 2004||catalog.ldc.upenn.edu/LDC2005T19|
|CallMyNet||K. Jones, S. Strassel, K. Walker, D. Graff, and J. Wright, "Call my net corpus: A multilingual corpus for evaluation of speaker recognition technology," inProc. INTERSPEECH, Stockholm, Sweden, August 2017, pp.2621–2624||10.21437/Interspeech.2017-1521|
|MLS/MLS14||K. Jones, D. Graff, J. Wright, K. Walker, and S. Strassel, "Multi-language speech collection for NIST LRE," inProc. LREC, Portoroz, Slovenia, May 2016, pp. 4253–4258||jones-etal-2016-multi|
|Mixer (pt. 1)||C. Cieri, J. P. Campbell, H. Nakasone, D. Miller, and K. Walker, "The Mixer corpus of multilingual, multichannel speaker recognition data," inProc. LREC, Lisbon, Portugal, May 2004||cieri-etal-2004-mixer|
|Mixer (pt. 2)||C. Cieri, L. Corson, D. Graff, and K. Walker, "Resources for new research directions in speaker recognition: The Mixer 3, 4 and 5 corpora," inProc. INTERSPEECH, Antwerp, Belgium, August 2007||10.21437/Interspeech.2007-340|
|Mixer (pt. 3)||L. Brandschain, D. Graff, C. Cieri, K. Walker, C. Caruso, and A. Neely, "The Mixer 6 corpus: Resources for cross-channel and text independent speaker recognition," inProc. LREC, Valletta, Malta, May 2010, pp. 2441–2444||lrec2010/792|
|VAST||J. Tracey and S. Strassel, "VAST: A corpus of video annotation for speech technologies," inProc. LREC, Miyazaki, Japan, May 2018, pp. 4318–4321||tracey-strassel-2018-vast|
|SRE16 test set||S. O. Sadjadi, C. Greenberg, T. Kheyrkhah, K. Jones, K. Walker, S. Strassel, and D. Graff, "2016 NIST Speaker Recognition Evaluation Test Set," 2019||catalog.ldc.upenn.edu/LDC2019S20|
|SRE21 dev/test set||Sadjadi, Seyed Omid, Craig Greenberg, Elliot Singer, Lisa Mason, and Douglas Reynolds. "The 2021 NIST Speaker Recognition Evaluation." (LDC2021E10),"||arxiv.org/abs/2204.10242|
|Janus multimedia dataset||G. Sell, K. Duh, D. Snyder, D. Etter and D. Garcia-Romero, "Audio-Visual Person Recognition in Multimedia Data From the Iarpa Janus Program," 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 3031-3035 (LDC2019E55)||10.1109/ICASSP.2018.8462122|
|CTS Superset||S. O. Sadjadi, D. Graff, and K. Walker, "NIST SRE CTS Superset LDC2021E08," Web Download. Philadelphia: Linguistic Data Consortium, 2021
S. O. Sadjadi, "NIST SRE CTS Superset: A large-scale dataset for telephony speaker recognition,"arXiv preprint arXiv:2108.07118, 2021
|WeCanTalk||K. Jones, K. Walker, C. Caruso, J. Wright, and S. Strassel, "WeCanTalk: A new multi-language, multi-modal resource for speaker recognition," Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3451–3456||lrec2022-we-can-talk|
In order to participate an user account is required. Signing up for an account is an easy two step process using email-confirmation explained here. The help center additionally shows how to reset a lost password or unlock the account.
After creating an account and signing into the participation dashboard, please follow the registration workflow on the left side in order to obtain access to the data.
In the next workflow step please select which track to participate in LRE
System output submission to the leaderboard must be made through the web-platform using the submission instructions described on the webpage (Submission Management). To prepare your submission, you will first make .tar file of your system output TSV file via the UNIX command ‘tar cvzf [submission-name].tgz [submission-file-name].tsv’ and then make your submission as follows:
The LRE-Scorer package (to be public soon) contains an output format checker that validates the submission. To validate your system output locally please use the following command-line:
All audio segments must be processed independently of each other within a given task, meaning content extracted from the segment data must not affect the processing of another segment.
While participants may report their own results, participants may not make advertising claims about their standing in the evaluation, regardless of rank, or winning the evaluation, or claim NIST endorsement of their system(s). The following language in the U.S. Code of Federal Regulations (15 C.F.R. § 200.113)14 shall be respected: NIST does not approve, recommend, or endorse any proprietary product or proprietary material. No reference shall be made to NIST, or to reports or results furnished by NIST in any advertising or sales promotion which would indicate or imply that NIST approves, recommends, or endorses any proprietary product or proprietary material, or which has as its purpose an intent to cause directly or indirectly the advertised product to be used or purchased because of NIST test reports or results.
At the conclusion of the evaluation, NIST may generate a report summarizing the system results for conditions of interest. Participants may publish or otherwise disseminate these charts, unaltered and with appropriate reference to their source.