In this talk, I will present my view on recent dominance of neural networks in language modeling. I will try to explain in a simple way why neural networks became so popular, and describe my ideas on what could be the research goals for scientists who want to overcome them.