The colors vary between blue, red, green and black,

..which means that every country has its own specific reasons for choosing one of the four colors for their passport.
The red is used by member countries of the European Union, except Croatia, and countries which didn’t join yet, such as Turkey.
Also, the red passports are usually chosen by countries with a historical or current communist system. Blue color symbolizes a new world. It used mainly in America.
The block of South American countries, including, Brazil, Argentina, and Paraguay, also use the blue passport, the cover symbolizing the connection with Mercosur.
The Green color used by the majority of Muslim countries, due to the importance of the color in their religion, as green is considered to be the favorite color of the Prophet Muhammad. It is also a symbol of nature and life.
The black passports are the rarest you can find. These can be found among the citizens of some African countries such as Botswana, Zambia, Burundi & Angola.