This was a doozy. From the oil buffer, when you edited a file the
editorconfig-vim autocmd that runs s:UseConfigFiles() would run and
something in that logic would cause the oil buffer to become `set
modified`. This obviously messes with the buffer cleanup logic, and also
makes vim more annoying to exit. The solution is to disable
editorconfig-vim in oil buffers.