It’s unfortunately not possible to swap between list and grid views with the gtk file-chooser. It’s a common complaint, but there doesn’t seem to be any movement to allow greater configuration.

There are some limited configuration options for the file-chooser exposed through dconf. The most accessible way to modify these is with Dconf Editor. You can find file-chooser settings at /org/gtk/settings/file-chooser/ and /org/gtk/gtk4/settings/file-chooser/, but it’s mostly things like whether to show hidden files by default or sort by a certain column by default.

I’m in the same boat as you; I wish the file-chooser had some more config available, but I just kinda live with it. It does seem like you can replace the gtk file-chooser though. the XDG Desktop Portal framework was originally designed to allow Flatpak applications to use native environment tools like file choosers, but it seems like you can use it to force a different file-chooser.