Just discovered Yuval Harari and I'm pretty impressed. My mind was completely blown at minute 47 as he predicts that the evolution of morality will go from religion to human intuition to AI intuition.
"This is true because if you want to make decisions about your life and you have two potential sources of authority, say the Bible or your own feelings, biologists will tell you, go with your feelings. The Bible contains the wisdom of a few priests in ancient Jerusalem. That's it.
Your feelings contain the wisdom of millions upon millions of years of evolution, the algorithms that are your feelings have passed the most rigorous tests of natural selection. Each one of your genes, and each one of your algorithms is here today because it has succeeded for millions of years to pass the most rigorous tests of natural selection.
So if you need to make a decision, you better go with your feelings. This was the situation maybe until today. But now things are changing. In two ways things are changing. First of all, biologists decipher the biochemical algorithms that compose our body and that are responsible for our feelings. They are no longer mysterious entities as they were before. We understand them better and better. Why I feel this way in this particular situation. And secondly, we have computer scientists that are creating better and better electronic algorithms. Now the question that arises today is what will happen once we create an algorithm that knows me better, that understands me and my feelings better than I can understand myself? What will happen then is that authority will shift. It will shift away from the feelings of individuals to these outside algorithms.
Our feelings at present, you can say, are the best algorithms in the universe. But after all, they are becoming outdated. They appeared maybe the way that we have them today maybe a few tens of thousands of years ago in the African Savanna. So far they have been the best algorithms around, but we are now at the position to produce an updated version, something better. And when we have better algorithms then the authority will shift to them."
https://youtu.be/g6BK5Q_Dblo