The blue object contrasts more with the orange and red backgrounds than the blue, green or yellow ones due to the contrast of temperature. The reason they contrast more with the red and orange than the yellow, despite having contrasting temperatures, is that there is a greater contrast in hue between the blue with the red and orange than the yellow, which overpowers the greater difference in tone the blue has with the yellow. The blue sits quite nicely with the green and the other blue due to its low contrast in all of the categories of contrast.
Green, being a neutral colour, sits nicely with the yellow, green and blue backgrounds due to it having similar tone and hue. The contrast in temperature however makes looking at it next to the orange and red uncomfortable, especially next to the red due to the complimentary contrast. Because it is a neutral colour, is has a low contrast of extension in most cases, with the obvious exception being against the red, it's slightly higher in the yellow and orange than in the blue and green because the contrast of saturation, as the yellow and orange are more saturated than the green and blue.
The yellow has quite a high contrast with all the background colours, and this is mainly due the the high contrast of saturation. The yellow object is a lot more saturated than the background colours, even the other yellow. The lack of contrast of temperature between the yellow with the red or orange means the colours don't contrast too badly despite having quite a high contrast of tone. The harshest contrast the yellow has is with the blue, which isn't even that bad as the main element is the contrast of tone and temperature, this may be different however if there was a purple background as the complimentary contrast might cause some problems as well.
The orange object I have noticed tends to have the opposite effects in terms of contrast to what the blue did. This makes sense I suppose though, given that orange and blue are complimentary colours. The only difference being that the contrast between the 2 oranges is so slight that it is uncomfortable to look at. It has contrast of temperature to some extent in that the darker one appears to look warmer because the lighter one looks cooler. This is because of the contrast of saturation, as the background orange is more saturated than the object orange.