When a homeowner buys a neighboring apartment or home to combine with theirs, that is one less home in the pool of inventory. When Ken Griffin purchases multiple neighboring lots in Palm Beach, most with huge mansions on them, tears them down, to parcel the pieces together for one mega-mansion, this removes inventory for even the wealthiest buyer pool, so this aspect of real estate does not simply impact 'affordable housing' inventory.
In the West Village in New York City we have seen numerous townhouses that once housed five or more apartments be converted back to single family use. Several have combined two and more neighboring townhouses into 10,000sf-plus mega-mansions, often removing ten or more smaller, cheaper apartments, the kind the city most desperately needs. They are almost never replaced, certainly not in these prime locations or at the lower more affordable price points. And yes, often these apartments are not in great shape at all, often unsafe and inefficient.
For decades it became customary for the opposite to happen. As estate taxes hit, many estates that were not cash-rich were compelled to lop off a piece of their land to cover this expense. Many massive mansion-scaled homes became too large or too expensive to maintain. Many huge estates and castles around the globe are converted into hotels and apartments too. As the world's wealth grows - especially amongst the wealthiest - the demand for mega-mansions is actually growing. And yes, 'real people' living in a two bedroom apartment often find it cheaper and/or more convenient to buy out a neighboring 1-bedroom apartment when the need for an extra bedroom arises.
This is just one aspect that shrinks inventory. Now add in damage from fires, storms, decay (hundreds of thousands of homes are aged well past 50 years - 5.65 million are 100 years old and more - and not all are built to last forever) and you can easily see that while we attempt to grow our inventory by building new homes, simultaneously, existing inventory is disappearing too, further exacerbating this huge challenge.....and home price inflation.