autocmd for terminal is triggered when swapping

[ilan-schemoul]

In my config I have
autocmd TermOpen,BufWinEnter,WinEnter,BufEnter term://* startinsert
If I am in a regular buffer and do WinShift swap with a terminal buffer then the autocmd is triggered and my regular buffer is now in insert mode.