Natural selection is the process which selects which organisms survive according to how well they are programmed to deal with their environment. Those organisms whose genes do not instruct them as well as the others in their population may be deemed 'unfit' and will not pass on as many genes through generations. This normally means that unfit organisms reproduce less or not at all due to death or with problems getting a mate. Over time, we will witness 'unfit' genes becoming less frequent in the gene pool and overall the population will be more adapted to their environment.
There are many different environmental forces which impress themselves on organisms; temperature, amount of food, amount of water, potential predators, disease etc. On earth there is a variety of environments which pose different obstacles for organisms to survive and pass on their genes. The obstacles are normally called selection pressures i.e. the success of an organism to survive and reproduce despite these pressures will be favored because their genes have made them more able than others.
The question I am posing can now be elaborated: Will the gradual natural selection of more advantageous genes (for an organism) which deals with these environmental pressures better after each generation, lead to a species of organism which does not feel these environmental pressures? That is... a species of organism who is completely fit for their environment and therefore has reached a point at which natural selection stops for that species. We may call this scenario an SPE (Selection Pressure Equilibrium).
We must consider a few things before we can come to a reasonable conclusion. We must first, not only consider, but accept how nature is. The environment i.e. climate, abundance of food, surrounding populations of other species etc. is not a constant background, so the selection pressures are not constant... they vary. So we must first of all rule out an SPE for a constant environment. We must define a wide range of reasonable, varying, selection pressures which a species can take without natural selection occurring. We must also consider the conditions for which an SPE can be reached and discuss the implications and issues of our rationale.
Range of selection pressures for an SPE
Consider a hypothetical species of organism X which has become the dominant predator in a large territory and can find ingenious ways to conquer the regular cycles of the environment in that territory. However... over a short period of time a species Y has migrated from another distant territory (due to a lack of food or water) to X's territory. This species Y poses a new threat to X i.e. it hunts animals by chasing them. This poses a new selection pressure to X which entails that faster organisms will have a better chance of surviving, genes which encourage longer limbs etc. (that make X's faster) will be favored. It seems that X is not in an SPE as potential selection pressures are likely to come around.
It does not seem reasonable to deem a species in an SPE if they are continually at risk to new selection pressures either due to the climate or the dynamics of other animals. It is not reasonable to do this as natural selection can in theory do more to improve the fitness of that species or in other terms more genes can be selected out, they just haven't felt the selection pressure to do so yet.
Surely this rules out the possibility of most organisms on this planet in reaching an SPE, as they are specialized for their unique habitat (but not perfect) and some would struggle if they migrated half way across the world. So is an SPE a useless idea? No. The equilibrium can still be obtained. The species must be able to overcome its environment not with just its biological tools but with artificial ones i.e. a species which can somehow manipulate the surroundings to overcome new selection pressures before the process of natural selection begins. Any environmental pressures which cannot be overcome with the manipulation of the organisms surroundings will have a devastating effect on the species... it would lead to annihilation (for example a huge unexpected meteorite).
This leads us to a specific condition for an SPE: A species is only in an SPE if and only if it can conquer potential selection pressures by artificial means. If there remains selection pressures that cannot be overcome artificially this entails that these pressures will be felt by the species by natural selection and hence by definition it is not in an SPE.
Does this imply that evolution stops when in an SPE?
Natural selection plays a huge role in evolution however it is not the only process which selects genes to be passed down the generations. There is sexual selection which promotes the genes which provide instructions for physical characteristics which are sexually favorable (such as the peacocks tail). Sexual selection is in some sense independent of natural selection but it is not completely independent as characteristics which promote fitness may also be sexually favorable. Sexual selection can occur without natural selection and therefore females or males in a species in an SPE may find some characteristics more sexually attractive than others, over time we may see the spread of these characteristics throughout the population. So we see a selection process occurring without a pressure from the external environment, the selection pressure comes from sexual attractiveness which does not necessarily affect survival or fitness.
In a species in an SPE we may have to assume that populations of that species are closely connected physically, this basically means when they come into an SPE they are already or are gradually growing into a globalized population. Which means mixing of genes with mates from previously hard to reach territories. So we should in fact see genetic drift play less of a role in the genetic change of most of the population. However in small population sizes (which can in theory still persist in an SPE as they can still fulfill the condition) random sampling will occur, but the gross effect of genetic drift will not be significant.
It is possible for a species in an SPE to manipulate their own genetic material and devise artificial selection pressures for whatever reason. Another, not so common, selection process which can occur is a memetic one. This means that the culture of a species in SPE can become subject to selection processes. This should be reminding you of a species not too far from you... the modern homosapien who has come so far scientifically and culturally. We in principle can manipulate single genes already and our culture has 'evolved' drastically without genetic change, this implies that culture evolves under unique selection processes.
The question is now: Will natural selection tend towards higher intelligence?
When a species can understand the mechanisms of the world to the extent that it can control its own surroundings to improve the quality of its life and avoid any unwanted nuisances (selection pressures), it can be said to hold higher intelligence. We only know empirically of one species that can do that and that's us.
It seems that, in order to manipulate the world in a precise manner we would first need a suitable body and a decent motion control center. This obviously would evolve prior to any decent form of intelligence could arise as the pressure to have a suitable body is much stronger than the luxury to perceive the world with sophistication. Natural selection has, on our planet, evolved bodies which are very versatile and can perform complicated movements... we see this from dolphins to the cheetah and so on. So we can generalize and say that natural selection will tend toward complicated organisms i.e. it is favorable for genes to first group together, then specialize in functions and then to evolve into complicated bodies which are adept in motion and sensory detection.
Once this has been done, if the planet is stable and no mass extinction is upcoming then more subtle adaptions may occur such as brain development and intelligence. There may come a point where the brain and sensory equipment is developed so precisely that the ability to infer causality e.g. repetitive impact causes erosion and the ability to make use of this (tool making) may arise. This ability becomes advantageous and the brain tends towards increasing capabilities until it hits a point where it enables the species which it inhabits to understand the world so well that it can manipulate it to its own benefits i.e. reach an SPE. Obviously this could take a very very very long time. Billions of years or more (for our planet anyway).
I think the mathematics of the algorithms of natural selection and evolution will ultimately tell if natural selection tends toward complexity under the right environmental conditions (will probably need enough genetic iterations and time).
A part of me, from looking at entropy and disorder, chaos in the natural world and the complexity we see in all of nature believes that natural selection shares this tendency to move towards higher complexity.
We know of no independent life systems than ours so we cannot see whether intelligent life exists from the gradual improvement of simpler organisms.
So what is my answer in brief?
From my research and from writing this I cannot deduce an empirical conclusion. However I have gained some knowledge and insight into the mechanisms of natural selection and what it would entail over time.
I think it is highly probable that the process of natural selection is not infinite i.e. happens forever, I think if the environmental conditions are not too volatile for life (which we are beginning to know what conditions are needed for basic biological matter) then with enough time complexity will increase in such a way as to mold intelligent and highly complex species. If intelligence is anything like ours, we may presume that they too will try and understand the world and use it to their own advantage. They will become dominant and therefore will reach an SPE.
So yes I think it is quite probable that natural selection will tend toward a species that will enter an SPE. However I cannot place a value on such a probability.
Here is a cool video by Michio Kaku on has mankind stopped evolving (by natural selection)