Interesting... A lib directory in an Ivy project.
You can modify the build.xml to create one more target that calls the <ivy:makepom/> target. Just make sure that <ivy:resolve> is called first. This will create a small piece of the pom.xml file you need.
As for the rest. What's that technical term? Oh yeah, you're screwed.
The problem is that Ant and Maven have two completely different build philosophies. In Ant, you write a build.xml script that describes what you want to build and how you want to build it. In Maven, you describe your project via a pom.xml file, and Maven does all the build processing for you.
This isn't an issue of whether or not Whether Ant or Maven is the force of all that's good in the world and the other is only for luzers Apple Fanboys. This is a case of manually converting a pre-existing project into Maven.
You'll have to go through your build.xml and figure out everything it is doing. Then, you need to convert this over to a Maven pom.xml file. There's no way to automate this. Even worse, Red5 isn't setup like a Maven project, so you'll either have to move all the files around, or go into archaic pom.xml configuration hell trying to override how Maven assumes the build is suppose to take place. This can take days, even weeks to get right. And, in the end, you end up with a project you don't control that if you want to update will have to be done all over again from scratch.
Trust me, I did this before for another job where the System Architect decided that Maven was better than Ant, and all of our projects must be converted from Ant to Maven. And, who got stuck with this task? Not the developers who were too busy with other tasks, but I the Configuration Manager.
And, in the end, you will have a project you don't control that if you want to update will have to be done all over again from scratch.
There is an alternative: Ignore it.
Does it really matter if Red5 is an Ivy project? What do you need from this Red5 project anyway? Do you need that red5.jar or the distribution that gets built.
If you need the distribution, let it remain as an Ivy project. Simply set the ivysettings.xml to point to your Maven repository and let it know that it's in Maven 2 format. Ivy will have no problems getting stuff out of that. So what if it's Ivy?
If you just need that red5.jar file in your other Maven project, you can simply use the <ivy:makepom/> task to generate a pom.xml file for you. Then use mvn deploy:deploy-file to deploy that jar into your Maven repository:
$ mvn deploy:deploy-file -Dfile=red5.jar \
-DpomFile=pom.xml \
-DrepositoryId=$repoId \
-Durl=$url
Now, your red5.jar is in your Maven repository as a fully transitive downloading jar. If you really, really want to get fancy, you can embed the generated pom.xml file into the jar itself, so it is self referential just like Maven jars are. That will take about 30 minutes of hacking the current build.xml file. (Or, if your jar doesn't have to have the pom.xml embedded in it, a separate Ant file that just builds the pom.xml you need, and maybe even deploys it into your Maven repository for you. That way, if the project gets updated, you don't have to worry about the build.xml file being updated.