Let's consider this in more detail
1) Is it OK to use class files compiled, for example, with 2016R1 with 2017R1 server and clients?
In general yes unless there were changes in the runtime which may affect your application.
2) Is it OK to have a mix of class files compiled with 2016R1 and 2017R1? If it is OK, is there any potential problem if a program compiled in one version calls a program compiled in another?
It should not be an issue, but again it depends if the new version of the runtime or compiler features contain a fix which could affects your application. Additionally, you cannot have an application server running isCOBOL 2015R1 calling a program which was compiled with a later version of isCOBOL i.e. 2016R1.
3) Can I mix isCOBOL versions? for example isCOBOL 2017R1 on the server and isCOBOL 2016R1 on the client
No, the major version (17R1 in this case) needs to match.
4) Can I mix Java versions? For example Java 1.8 on the server and Java1.7 on the client.
Usually yes, unless in some particular cases such as java-bean used in the application.
5) Can I mix C-tree server and client versions? For example after using C-tree 10 embedded in an older isCOBOL version, I need to upgrade only isCOBOL to 2017 without upgrading C-tree to 11 version
Yes, it's possible. You will need to copy the c-tree 10 versions libraries to your isCOBOL installation. Versions numbers of c-tree are for reference only, your versions may be different.
Copy the version 10 of ctree.dll (for Windows) or libctree.so (for Unix) from the c-tree installation folder .../Driver/ctree.cobol/iscobol/ to the iscobolin directory (for Windows) or to the iscobol/native/lib folder (for Unix).