Can I mix versions of Java, isCOBOL compiler and isCOBOL runtime?

Let's consider this in more detail

1) Is it OK to use class files compiled with an older version and run in a newer version?
In general yes unless there were changes in the runtime which can affect your application.

2) Is it OK to have a mix of class files compiled with different versions? If it is OK, is there any potential problem if a program compiled in one version calls a program compiled in another?
It should not be an issue, but again it depends if the new version of the runtime or compiler features contain a fix which could affect your application.

3) Can I mix isCOBOL versions with the Application Server? for example, one version on the server and another on the client
No, the major version (17R1 in this case) needs to match.

4) Can I mix Java versions? For example Java 1.8 on the server and Java11 on the client.
Usually yes, unless in some particular cases such as java-bean used in the application.

5) Can I mix C-tree server and client versions? For example after using C-tree 11 with an older isCOBOL version, can I upgrade isCOBOL without upgrading C-tree to a newer version
Yes, it's possible. You will need to copy the old c-tree version's libraries to your isCOBOL installation. Versions numbers of c-tree are for reference only, your versions may be different.
Copy the old version of ctree.dll (for Windows) or libctree.so (for Unix) from the c-tree installation folder .../Driver/ctree.cobol/iscobol/ to the iscobol/bin directory (for Windows) or to the iscobol/native/lib folder (for Unix).