I did the brute-force ignorant method. Delete all registry keys that have the extension mentioned in them, as well as delete all the registry keys that have the previous application (bar.exe) mentioned in them.
Assuming the extension is .foo and the application is bar.exe, I think I found stuff here
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\FileExts\.foo
HKEY_CLASSES_ROOT\Software\Microsoft\Windows\CurrentVersion\Explorer\FileExts\.foo
HKEY_USERS\foo_auto_file
HKEY_CLASSES_ROOT\Applications\bar.exe
as well as some other stuff that did not seem helpful to delete (a lot of MRU lists)
And I may have had to logout/login as well, although I'm not sure if that was 100% necessary.