One of the things I like most about using other IR libraries as backends is that many of them offer language bindings in multiple other languages. So PHP, Ruby, Python, etc., users can be happy right away with search ability.
Of course, if the indexing program is Perl instead of C, there is that added requirement. But hopefully, if the indexing API is well documented, there’s nothing to stop implementations in other scripting languages besides Perl.
Take Xapian for example. They have bindings available in nearly every major scripting language. So if you don’t like the way Swish-e implements the indexing scheme, there’s nothing to stop you from writing your own in your favorite language. At which point, you’re not really using Swish-e any more. But you could mix/match depending on your needs. Use Swish-e’s spider and SWISH::Filter, but your own parser and indexer, for example.