While I have found a few Earthworms under established quality landscaping fabric, it's not their favorite place to live. Other than the roots of plants that get planted through the fabric, there is very little for them to eat. While there may be some bacteria, algae, etc., for them to eat under the fabric, there's not generally enough to support a large colony. Worms don't eat just dirt. They eat the stuff that is on and in the dirt.
If the fabric you use is quality
professional landscaping fabric and is installed correctly, the plants will do just fine. It is created for that.
While there may be other reasons to remove the fabric, there is none more important than keeping your wife happy. However, if you do remove it, it should be established who will be the one to pull the weeds and invasive grasses.
Steve