If I create a class with nothing in it and then instantiate it n times, is it possible for an OutOfMemory error to occur?
Yes. If N is large enough.
If I create a class and inside that class' constructor create an instance of itself, will I have an infinite nesting of the same class?
If you had an infinite amount of memory, yes. In practice you don't, so the creation is bound to fail.
Will an OutOfMemory error occur?
Possibly. It is more likely that you will get a StackOverflow error ... because what you are describing will create a recursion loop.
Suppose I created a class which is supposed to get fed objects and check if they have any variable references to themselves, and include them in a list if they do not. How will my class react when the object passed to it is itself?
It is impossible to say. It depends on how you implement the code to "check if they have any variable references to themselves".
You seem to be trying to explore the consequences of Russell's Paradox for Java programs. The simple answer is that it has no consequences. A Java program that involves representations of sets will do whatever it has been coded to do. If you have a representation that allows a "set" to "contain" itself, then that will happen.
There is one thing that should be obvious though. A simple representation of sets in Java will not allow you to represent sets with infinite numbers of elements. Memory (and stacks) in Java are finite. This doesn't mean that it is impossible to represent all infinite sets. You just have to do it differently; e.g. using "lazy" data structures or a symbolic representation.
It is tempting to think that a representation of sets might actually be mathematical sets, and you might be able to cause unspecified weirdness by triggering Russell's paradox. In reality, no such thing will happen. Russell's paradox asks the question "Does the set of all sets contain itself?". But a program on a Turing-machine-equivalent computer cannot test all elements in a simple infinite set ... let alone the set of all sets. Russell's question is simply not a tractable problem. (But then, it was never intended to be tractable. It is really about exploring the limits of meaningful mathematics.)