It is a compile time parameter. The ./configure script of less knows the with-regex=LIB
param.
This is a quote from README of the upstream package:
--with-regex=lib
Specifies the regular expression library used by less for pattern
matching. The default is "auto", which means the configure program
finds a regular expression library automatically. Other values are:
posix Use the POSIX-compatible regcomp.
pcre Use the PCRE library.
regcmp Use the regcmp library.
re_comp Use the re_comp library.
regcomp Use the V8-compatible regcomp.
regcomp-local Use Henry Spencer's V8-compatible regcomp
(source is supplied with less).
So you would need to know how less was './configured'. I have investigated this on Debian / Ubuntu. They use the POSIX regex lib.
I'm still searching for a way to detect it dynamically by a script... :)
Update: The only thing I've managed so far was to detect whether less uses pcre regexes or not. If less was configured using --with-regex=pcre
it is linked against the libpcre.so shared library:
#!/bin/bash
# ldd prints out the shared libraries a binary is linked to.
# This can be used to check if less is linked against libpcre
if ldd "$(which less)" | grep 'libpcre\.so' ; then
echo "less uses pcre regex syntax"
else
echo "less uses non pcre regex syntax"
# ... more checks should follow. currently trying to find a way
fi