If you are struggling with vrays settings (so many settings), I suggest you consider this neat little script: DeeX Vray Fast Deex What it does: spawns a w. The tool DeeX Vray Fast is only for Maya.
I'm using Vray 3.60.3 on Max 2017 and it's working just fine.
This scene is available for 3ds Max 2014 and up. I think it's something with ur install of 2017.
Improved performance and no “fireflies” when using GTR/GGX materials. Learn how to use V-Ray GPU supported features such as the displacement modifier, V-Ray Hair Material, and Ornatrix plugin.