Unfortunately you seem to be an expert in a niche that very few Haskell users know much about. All I know about these technologies is what I have overheard from non-technical people talking about different business rules engines, so I'm probably off-base but I think it's worth a shot since everyone else is stumped.
As far as forward-chaining reasoning systems and the Rete algorithm in particular, they're basically functional already. They iterate and add more knowledge to the database as they go until they run out of things to do. Provided you don't allow arbitrary effects, it would be a simple Control.Monad.State
port; if you need arbitrary effects you can use a state monad transformer instead, and it's not something intermediate/advanced Haskellers would be all that intimidated by. You may find something you can use on the Haskell site but if you wind up doing this yourself the Monad Transformers chapter of Real World Haskell is indispensable.
I don't know anything about behavior trees, but at a glance on Wikipedia they look to me like the Rete algorithm plus concurrent processes in the background. If that's even close to correct, then you have to decide whether or not you need concurrency or parallelism. If you're content with pure values "magically" being computed faster (and, by extension, everything being written in Haskell) then you can get away with using the stuff in Control.Parallel
for not much effort but you won't be able to (say) inquire which processes are running and which are not. If you genuinely need different observable behavior, then you'll need Control.Concurrent
and it's less magical, so more book-keeping. Simon Marlow wrote a pretty nice tutorial about your options there. What you're describing sounds lower-level to me than most of the cool stuff Haskell can do for you; if you decide to accept a higher-level interface you may find it easier to implement. I'm far from an expert on this topic either.