Einstein had mature, deep, and well articulated views about religion and "god". The "doesn't play dice" quote needs to be understood in the context of his expressed thoughts about science and religion. I think this one essentially sums up the core of his belief:

But science can only be created by those who are thoroughly imbued with the aspiration toward truth and understanding. This source of feeling, however, springs from the sphere of religion. To this there also belongs the faith in the possibility that the regulations valid for the world of existence are rational, that is, comprehensible to reason. I cannot conceive of a genuine scientist without that profound faith. The situation may be expressed by an image: science without religion is lame, religion without science is blind.