Perhaps for those who are perturbed with the idea of such creatures the most common approach is to deny the actual existence of sheidim and understand Chazal’s words allegorically. This approach is by no means a modern innovation, but already articulated centuries ago by R. Avraham ben HaRambam:
The second part of the stories consists of such stories as did not a[c]tually occur but were seen in dreams; they speak of them as real stories, because they believed that no thoughtful man would ever mistake them for real facts; as the one (Vol. I, 24.)We are taught (in a Baraitha) R. Ishmael said: “Once upon entering the holy of holies, to prepare incense, I noticed etc.,” and many other similar stories. And the same is true regarding certain stories in which are mentioned the visions of the prophets, how G-d spoke to them, and also the stories of demons. The thoughtless observer who, for the sake of believing, thinks that these things occurred exactly as stated though the facts are contrary to common sense, in doing so, is both foolish and ignorant of the laws of nature. (Introduction to the Agada, Abraham son of Moses Maimonides, translated by Rabbi S. H. Glick in the preface of Volume 1 of his translation of “En Jacob” page XIV).
Rav Avraham views the passages in Chazal as if prefaced with a self-obvious disclaimer that they had occurred in a dream. Demons are not actual beings but imaginary, and there only relevance is the lessons which can be learned from the visions of Chazal. This approach is very difficult, insofar as in the days of the Sages people did believe in concepts such as demons and would have had absolutely no reason to understand these accounts any other way than as had been recounted. Not merely the thoughtless accepted these words according to their apparent meaning but most Rishonim did as well. Would leaving such a fundamental aspect of the narrative not be at least incongruent with the saying of Avtalyon, “Scholars, be cautious with your words, for you may incur the penalty of exile and be banished to a place of evil waters [heresy]. The disciples who follow you there may drink and die, and consequently the Name of Heaven will be desecrated” (Avos 1:11, Artscroll Siddur Translation)? Furthermore, to reject dogmatically the existence of anything, because of its perceived impossibility despite evidence one would otherwise consider trustworthy is to reach a conclusion a priori rather than based on sound reasoning.
Of course I do not mean to dismiss lightly the explanation of a Rishon. However, I believe that the evidence demands we accept the conclusion of the other Rishonim, particularly considering Rav Avraham’s admonition, “Know that it is your duty to understand that whoever propounds a certain theory or idea and expects that theory or idea to be accepted merely out of respect for the author without proving its truth and reasonableness pursues a wrong method prohibited by both the Torah and human intelligence.” (ibid VII.) In truth some contemporary writers who are disinclined toward a literal existence of demons find it necessary to seek out an explanation other than Rav Avraham’s and understand references to demons being indicative of mental illnesses and delusions.
In addition to the difficulty in a blanket assertion that it would be naturally inferred that these passages had occurred in dreams (which in fact was not obvious to many) we see that the details of the passages are not easily reconciled with that approach or the suggestion that they describe mental illness. While some may indeed conform to such explanation it seems difficult to subscribe to such an explanation when others do not.
Our Rabbis taught: Six things are said concerning demons: in regard to three, they are like the ministering angels; and in regard to three like human beings.‘In regard to three they are like ministering angels’: they have wings like ministering angels; and they fly from one end of the world to on the other like ministering angels; and they…hear from behind the Veil like ministering angels. ‘And in regard to three, they are like human beings’: they eat and drink like human beings, they propagate like human beings, and they die like human beings. (Hagigah 16a, Traditional Press)Sheidim are explained as having some aspects which are spiritual, like angels, and others which are more material, like humans. Taken literally this passage helps illuminate the answer to our question on how to relate to the danger of a being we never encounter. Taken as part of a dream it is difficult to decipher a message, but at the same time many passages of agadata convey messages other than the literal meaning but whose lesson remain veiled to us. It would be very difficult to take this passage as describing characteristics of mental health issues or advice on avoiding situations which could render one susceptible to mental illness. Another passage takes the discussion outside the realm out of agadata and squarely in the "arba amos shel halacha":
But is there not a possibility that it may be a demon? Rab Judah said: We assume that he can be seen to have the appearance of a man. But the demons also can look like men? — We assume that they see his shadow. But they also have a shadow? — We assume they see a shadow of a shadow. -- R Hanina said: Jonathan my son has taught me that they have a shadow, but not a shadow of a shadow. (Gittin 66a)The discussion revolves around writing a Get (document of divorce) upon instruction from a man trapped in a pit. It seems very difficult to me to conceive of a reason why Chazal would introduce what they had seen in dreams into a halachic discussion about divorce. Furthermore it is difficult to understand that they meant to object that the instruction had been given in a dream (or imaginary voices due to mental illness) since, while with respect to demons one has reason to assume they appear distinct from a human, there is no reason to believe likewise about a figment of one’s imagination. The very fact that one is imagining a human speaks makes it most likely they speaker will appear human in our imagination as well. Furthermore, while the passage concludes with a way of distinguishing a demon from a person, it seems difficult to accept that Chazal meant to say that our dreams or mental delusions are limited to a shadow, without a shadow of a shadow.
So if we where to concede that sheidim exist we are left with the obvious question of why we never seem to encounter them? When we further examine the relevant passages I believe we will see a picture emerge which provides us with an acceptable understanding.
Our Rabbis taught: there are three reasons why one must not go into a ruin: because of suspicion, of falling debris and of demons… To one [person] an evil spirit may show itself and harm him; to two it may show itself, but without harming them; to three it will not even show itself (Berakoth 3a, 43b, Traditional Press, New York City New York, translated by Maurice Simon, M.A., edited by Rabbi Dr. I. Epstein B.A., Ph.D., D. Lit.)Both of these passages indicate that one encounters sheidim in solitude. So to we find that one who is traveling alone on a highway must beware of these harmful spiritual forces, “Highways tend to be dangerous places, as has been explained elsewhere [Derech Chaim 3:9], because they are uninhabited and therefore alienated from the world’s essence. The world’s essence is settlement. Highways are apart from settlement; Therefore malevolent forces [פגעים], which are in discord with the world, dominate” (Maharal of Prague, Nesivos Olam: Nesiv HaTorah 1:3, page 49, Artscroll). Sheidim are “anti-social” so to speak. Highways are like a thin thread of civilization which leads one through the depths of the untamed world. Ruins, “ghost towns”, are remnants of destroyed civilization which have reverted to the wild. The proximity and association of these places to civilization makes it more likely for people to run across them, while there desolation makes them perfect habitation for sheidim. However the thread of civilization of a highway isn’t enough to protect one from sheidim, but such protection is provided by learning Torah or the presence of traveling companions.
We have already noted that sheidim require nourishment, procreate, and die. It seems to me that with the passage of time it would be increasingly unlikely that sheidim could flourish as a species. As civilization grew the likelihood that most people encounter such antisocial forces diminished. G-d instructed the B’nei Yisrael to slowly conquer the Canaanite nations lest the land become populated by wild animals while waiting for the Israelites to resettle the barren land. Just as lions and bears are not well suited to city life, nor are sheidim, perhaps even less so in ways. Furthermore it would seem reasonable that even the advance of civilization represented by modern science proceeded, this further infringed upon the habitat of sheidim. As every inch of the globe became explored, measured, mapped, and its nature understood perhaps it became as every bit inhospitable for sheidim as the many growing metropolitan centers.
I believe it has been correctly said: “Concerning the subject of demons, the evil eye, and the evil spirits referred to in the Talmud, there can be no doubt that the Rabbi’s believed in there existence, and consequently we should not attempt to offer other interpretations which will explain them in a sense remote from the literal.”  Certainly the contrary view, that the sheidim mentioned in the Talmud where not actual beings, has strong traditional basis, but the evidence has lead Sages throughout the ages to reject that approach. Nevertheless it seems reasonable to assume that one does not run a strong risk of actually encountering such a being these days. It is not merely the case that people of the past were simpletons and we are not so gullible as to believe in such things. Rather the traditional references themselves provide us with an understanding of sheidim that would tend to explain our lack of encounters with them and inability to find to provide scientific evidence. While the ability to contrive a scenario in which something asserted to exist cannot be detected does not provide actual reason to accept its existence, it does give those of us who accept the Mesorah reason not to search for alternative understandings of the word’s of Chazal.
 “When the sun is low on the horizon in the morning and evening, it causes objects to cast two shadows, a dark one [closer to the object] and a lighter one [farther from the object]. The lighter one is referred to as the ‘shadow of the shadow’” Artscroll Gittin 66a (2) note 25.
 Compare ibid 13:4 (page 283) on Chullin 91a where the Maharal discusses the additional caution a Talmid Chacham must exercise by not going out at night even though they clearly have the merit of Torah to protect them. There he equates the “harmful forces” [המזיקים], which parallel the of פגעים our passage, to the sheidim mentioned in B’rachos 6a.
 Reb D.W. related to me a shiur (by Rebe Michael Twirsky shlita?) in which a metaphysical explanation was given for not learning on Nittel Nacht. It was explained that our Torah learning gives koach (“nourishment”?) to the forces of tumah, but these forces are already strengthened on Nitel Nact. If this is true of sheidim then we might understand why they are attracted to Torah scholars in particular (Chullin 91a) and despite their aversion to civilization can be found at the “kallah” gatherings (Berachos 6a). Furthermore the decline of Torah excellence might result in “famine” conditions for the more materialistic sheidim.
 (Mebo HaTalmud: The Students Guide through the Talmud, Zvi Hirsch Chajes, Translated by Jacob Shachter, MA. Chapter 31, page 233)