Would you consider changing this, since this way of deduplication in the frontend is quite inefficient?

I simply consider it part of the requirements (see the other message). And there has been some effort made to ensure that the annotation/kind functions are only called when equal strings are encountered.

Indeed, when the list is long deduplication does show up on the graph, but the impact also depends on the shape of the data, and there probably are some untapped code optimizations still.

Deduplicating in the backend is not difficult, all that is needed is adding a suffix like (1), (2), (3), …

I’m not sure this is very easy (e.g. for LSP clients), and in general it would require a scan across all completions of comparable complexity. Also, showing method overloads with suffixes like 1/2/3 would look rather odd, I think.

Actually, I forgot how this works in most cases X-D.

And it happens through text properties, which backends add to their completion strings. Then the annotation and kind operations look them up.

Something like that would be generally easy to see very soon (completion popup without both icons and annotations). This approach has worked for years for both Emacs’s default UI and company-mode, so it’s hard to call it fragile.

I suppose it might have been a cause for investigation for some backend authors at some point, but backends would generally avoid internal copying anyway, for performance reasons if nothing else.

As long as the completions have the same annotation (the text in green) and same kind (the icon on the left), they are deduplicated.

Looks like perhaps lsp-mode generates annotations that are more useful for your scenario than the ones that eglot does. If that is the case (and not, maybe, that you have configured lsp-mode/eglot to use different language servers), I suggest filing an issue. This shouldn’t take too much time to change.

Also try C-x p g for a similar UI.

Customize xref-search-program to make it use ripgrep under the covers.