I hate to pick on Zillow. The company has done a tremendous job pushing the boundaries of innovation in the real estate market. But what happened with the Zillow Offers program — as unfortunate as it is for Zillow’s staff, management and shareholders – provides an excellent opportunity to reflect on the ever-increasing impact of algorithms on our daily lives.
For those not familiar with the matter, Zillow started using their real estate valuation algorithm “Zestimate” to buy properties in order to flip them for a profit in a few months. That strategy backfired as they overpaid for many of these properties in Q1/Q2 2021 and had to write down 300M in losses and shut down the iBuying program last month. The company will be laying off 25% of its 8,000 employees and the stock has plunged 44% since the announcement in early November. Ouch and ouch!
To start off, developing and maintaining a machine learning algorithm is one thing. How the management uses an algorithm to drive business decisions is a completely different thing. The former is a statistical/mathematical exercise with the sole objective of increasing the accuracy of a model. The latter is about generating revenue (or some other KPI) which might be impacted by many factors other than the accuracy of a model.
The Automatic Valuation Model (AVM) technology has been around for 10~15 years now; It is easy to forget the alternative, the inefficiencies & biases in valuations made by individual appraisers or brokers. Blaming this debacle on Zestimate, the machine learning algorithm that generates valuations is undermining the work of many researchers and data scientists working tirelessly on training and testing these models. In fact, Zestimate’s median error rate of 1.9% is impressive. But the devil is in the word “median” as the accuracy can vary widely on a case-by-case basis.
For many consumers of valuations, a 3% to 5% or even greater margin of error may not be a big deal. Mortgage issuing institutions build safety through large down payments. Individuals buying real estate as a primary residence may easily overpay by that margin (and sometimes a lot more) for emotional reasons. Long-term investors make up for market inefficiencies by stretching the investment horizon.
This logic doesn’t apply to short-term investors or flippers though as a miscalculation can easily erode their expected profit of 5-10% or even leave them in the reds. For flippers, accurate valuation is just one of the success factors. Equally important is the work that goes into renovating or repairing a property on a tight budget and even a tighter deadline. And none of that bodes well with the current pandemic wreaking havoc on the supply chain and worker shortage!
It is easy to blame the Zillow’s management team but I somehow sympathize with them. It is equally easy to be enthused by the prospect of running a new line of business based on a fancy algorithm. It was a costly mistake but there is a silver lining. This should be a humbling experience for those of us working in tech/data and a timely reminder that good human judgment still trumps AI.
