Math and science both did grow out of philosophy, but that does not mean philosophers are experts at math and science while mathematicians and scientists don't understand philosophy. I personally think the fact that the greatest minds did tend to be people who studied and thought about many different fields - math, science, philosopy, art, is possibly causal not consequential. If you exercise all the different basal abilities of your brain they improve and together they make your brain able to attack a question from many different angles, which brings out all sorts of different features that aren't initially apparent. Presumably if one is harder than the other it's as much a reflection of the amount of effort people have put into it than the inherent difficulty. I have no doubt there are people out there who are smarter than me and could come up with either math or philiosophy I didn't understand. I think the difficulty level of the material would therefore be set by those who do it rather than what they are trying to do (as well as the amount of time and energy they pt into doing it).