cntct() function makes the query optimizer unhappy. The alternative is to use the table directly, as you noted. If hiding data based on record "ownership" is important, it has to be accounted for in the query somewhere.
Here and for your related post, the query execution plan will tell you where the time is being spent. Reading QEPs takes a lot of practice but it's worthwhile for anyone who writes or debugs a lot of queries. If record hiding significantly affects the size of the result set for different users, consider adding an index on the
SELECT xt.add_index('cntct', 'cntct_owner_username', 'cntct_cntct_owner_username_idx', 'btree', 'public');
Again, let us know what happens.