You need to understand the philosophy behind react-testing-library to understand what you can do and what you can't do with it;
The goal behind react-testing-library is for the tests to avoid including implementation details of your components and rather focus on writing tests that give you the confidence for which they are intended.
So querying element by classname is not aligned with the react-testing-library philosophy as it includes implementation details. The classname is actual the implementation detail of an element and is not something the end user will see, and it is subjected to change at anytime in the lifecycle of the element.
So instead of searching element by what user cannot see, and something that can change at anytime, just try to search by using something that the user can see, such as text, label or something that will remain constant in the life cycle of the element like data-id.
So to answer your question, it is not advised to test classname and hence you cannot do that with react-testing-library. Try with other test libraries such as Enzyme or react-dom test utils.