Let's try to paint a likely not-too-distant future. In this future, everything around us will be managed by algorithms. They manage and recommend what we want to watch; they control traffic of our driver-less cars; they control those drones which deliver our pizza; predict for you which one of your friends will become your future spouse, etc. They will make our life effortless and predict what things will happen next or what's best for us. We all wish that it is that simple.
As we are picking up massive data every second, algorithms are critical to how we make sense of them and our reliance on algorithms is growing by the second. But how much faith should we put into those algorithms, and how reliable they are depends on who you talk to. The accuracy of the algorithmic differentiation (automatic differentiation) is key in dealing with data volatility in many complex cases where accuracy of data is critical to the final outcome. I don’t know when we can have statistical evidence comparing how often driverless cars mistakes and how often human drivers make mistakes or how well a software program investing in stocks makes mistakes and human making investment mistakes.
We need to develop programs to check whether algorithms are working as they were intended and monitoring algorithms by the results they produce based on machine performance metrics. Vetting algorithms in advance isn't practical at all as many are too complex for their outcome to be predicted. Algorithmic differentiation is used to fine tune algorithms that are complex to optimize. We will be placing our faith into the accuracy of these algorithms and trusting that they will deliver. I think we may be over-optimistic and mining a lot of data using statistical analysis still has a very strong random component to it. To optimize these models we need algorithmic differentiation and sometimes it feels like a scientific random number generator. The more data we have, the more simulation is needed and the more reliance on the significance of data, and significance means sensitivities. We’re in the midst of a transformation when industries are exploring how to use big data to create new business models.
There’s one certain thing in big data, it’s that all algorithms are not created equal. The biggest question is how many jobs will disappear when algorithms take over human jobs? People ask the same question during the first industrial revolution when machines started making goods a lot faster and cheaper. Are we entering a world in which the algorithms themselves will be competing with humans? Honestly, I just don't like it when algorithms make decisions for us. They can help me analyze my options and probabilities, but not making a decision until they prove that they can think smarter than me. And so the game begins…
