Gradio

Human voice is a rich source of personal information, including identity. Advancements in speech technology, like voice conversion and synthesis have made it easier to clone and manipulate voices for misuse which is a huge privacy risk. Voice anonymization is a crucial step to protect speaker identity in speech data. Traditionally voice anonymization is evalauated for utlity and privacy using only a single ASR and ASV stystem. This leads to biases in the evaluation and does not reflect real world scenarios where multiple ASR and ASV systems exist. To resolve this we propose AUDI and Fusion EER metrics to aggregate results across 6 ASR and ASV systems.Kindly read through the Metrics Explained tab to understand how the metrics are computed. Below we rank SOTA voice anonymization systems on the proposed metrics. Note that we present Fusion EER in two scenarios - A2A (Lazy Informed attacker ) and O2A (Ignorant Attacker ). Lazy informed attackers have partial knowledge of the ASV system and thus can make certain imporvements to the attacks. Ignorant attackers have little to knowldege of the ASV system.

Lead Maintainers:

⚙️Utility Measure: AUDI (ASR Utility Distortion Index)


STTTS	11.380000000000003	2.27	10.71	17.32	18.52	144.88	10.38	30.87	15.38	23.98


🥇 KNNRetrival_VC_Cos_22	9.51	2.27	2.47	5.55	3.40	8.22	9.74	30.87	15.38	7.69
🥈 KNNRetrival_VC_5Spk_22	9.64	2.33	2.40	5.15	3.13	7.42	8.87	30.31	20.37	6.74
🥉 KNNVC	11.38	1.94	2.20	5.51	3.64	7.85	10.38	30.03	16.89	23.98
KNNRetrival_VC_5Spk_6	11.52	2.39	2.50	7.68	4.41	10.81	8.71	34.85	20.00	12.36
KNNRetrival_VC_5Spk_11	12.40	2.26	2.46	6.73	4.28	8.36	10.75	42.98	24.75	9.04
KNNRetrival_VC_Cos_11	13.18	2.50	2.57	6.87	4.39	9.22	10.73	47.85	25.48	9.04
KNNRetrival_VC_Cos 6	13.54	2.43	2.50	7.05	4.74	12.07	9.51	45.67	26.33	11.53
KNNVCR	15.71	5.53	4.44	9.59	7.72	nan	21.18	33.70	24.13	19.40
MCADAMS	24.75	8.94	10.71	17.32	18.52	19.89	20.31	78.96	38.09	10.00
NAC	28.06	7.71	6.97	17.46	12.65	93.31	21.92	51.27	27.24	13.97
ASRBN	30.99	5.76	4.52	10.49	6.80	144.88	20.01	43.65	25.55	17.28
STTTS	32.59	4.51	4.24	10.89	7.14	121.97	27.77	63.24	23.63	29.92

Table: AUDI (ASR Utility Distortion Index)

🛡️ Privacy Measures: Fusion EER (Equal Error Rate)


🥉 STTTS	43.050000000000004	48.49	51.57	33.52	48.58	40.88	55.05	50.53	48.26	47.94


🥇 ASRBN	47.20	48.49	51.57	33.52	48.58	40.88	55.05	50.53	48.26	47.94
🥈 NAC	43.05	44.32	46.53	32.88	45.20	40.73	47.27	51.43	47.31	31.78
🥉 STTTS	41.01	47.32	47.63	35.43	40.72	40.43	48.79	48.27	40.84	19.62
KNNRetrival_VC_Cos_11	34.18	41.59	35.25	32.80	36.11	14.39	32.83	38.14	34.37	42.13
KNNRetrival_VC_5Spk_11	34.15	42.34	35.99	32.15	37.09	9.64	38.45	37.92	32.56	41.22
KNNRetrival_VC_Cos_22	32.42	37.09	34.03	30.34	33.43	9.49	36.16	35.91	32.81	42.56
KNNRetrival_VC_5Spk_22	31.79	39.24	34.19	30.69	34.24	6.14	33.20	33.98	32.56	41.88
KNNVCR	27.63	28.74	34.96	17.31	26.17	nan	2.29	35.26	30.05	46.25
KNNRetrival_VC_5Spk_6	27.50	31.54	30.19	24.37	25.73	5.49	31.72	28.53	32.51	37.39
KNNRetrival_VC_Cos 6	27.33	30.02	28.05	24.17	25.20	9.31	29.23	29.86	34.36	35.75
KNNVC	22.59	26.29	31.20	15.61	22.50	2.29	0.13	30.81	26.56	47.93
MCADAMS	21.21	24.65	24.71	16.89	14.72	9.20	33.20	21.01	17.84	28.66

Table: A2A Fusion EER (Lazy Informed Attacker case)


🥈 STTTS	45.757777777777775	49.13	49.65	43.84	48.69	46.57	48.48	48.61	45.87	29.15


🥇 ASRBN	48.79	49.13	49.65	43.84	48.69	46.57	48.48	48.61	52.00	52.10
🥈 STTTS	45.76	49.28	49.63	46.10	49.04	49.20	45.69	43.23	50.50	29.15
🥉 KNNRetrival_VC_5Spk_11	45.27	42.54	49.01	40.77	46.18	39.99	41.04	49.44	45.87	52.63
KNNRetrival_VC_5Spk_22	44.99	41.25	46.36	42.52	44.73	33.41	48.48	48.89	48.13	51.17
KNNRetrival_VC_Cos_11	44.77	40.32	48.76	40.21	44.39	39.58	41.62	49.99	45.81	52.28
KNNRetrival_VC_Cos_22	44.20	40.89	46.38	40.63	43.56	31.50	45.89	52.10	48.00	48.89
NAC	43.00	44.35	46.34	41.64	44.98	38.49	40.91	44.90	49.22	36.21
KNNRetrival_VC_5Spk_6	39.86	38.51	34.57	33.78	36.49	31.35	48.25	46.64	42.86	46.33
KNNRetrival_VC_Cos 6	37.76	35.17	33.63	31.39	33.50	31.84	44.41	45.43	42.50	41.94
KNNVCR	37.49	38.13	35.65	29.19	29.28	nan	33.94	44.19	40.03	49.50
KNNVC	34.95	36.11	33.87	26.96	27.28	25.92	33.47	41.43	38.57	50.96
MCADAMS	28.05	28.30	31.82	30.67	26.21	21.07	26.26	24.68	27.57	35.89

Table: O2A Fusion EER (Ignorant Attacker Case)