Define multiple configurations for the same host using the SSH client configuration file.

I am using 192.0.2.44 server which provides SSH access to host on port 22 and guest on port 23.

The guest operating system is using a different SSH username.

SSH client configuration that defines user depending on the remote port.

Match Host 192.0.2.44 exec "test %p = 23"
  User milosz
Match Host 192.0.2.44 exec "test %p = 22"
  User ansible
Match User ansible
  IdentityFile ~/.ssh/ansible
Match User milosz
  IdentityFile ~/.ssh/milosz
Match LocalUser milosz
  IdentityAgent /home/milosz/.ssh/_socket

Connect to the host on port 22.

$ ssh 192.0.2.44 -p 22 -vv
OpenSSH_8.1p1 Ubuntu-5, OpenSSL 1.1.1d  10 Sep 2019
debug1: Reading configuration data /home/milosz/.ssh/config
debug2: checking match for 'Host 192.0.2.44 exec "test %p = 23"' host 192.0.2.44 originally 192.0.2.44
debug1: Executing command: 'test 22 = 23'
debug2: match not found
debug2: checking match for 'Host 192.0.2.44 exec "test %p = 22"' host 192.0.2.44 originally 192.0.2.44
debug1: Executing command: 'test 22 = 22'
debug2: match found
debug2: checking match for 'User ansible' host 192.0.2.44 originally 192.0.2.44
debug2: match found
debug2: checking match for 'User milosz' host 192.0.2.44 originally 192.0.2.44
debug2: match not found
debug2: checking match for 'LocalUser milosz' host 192.0.2.44 originally 192.0.2.44
debug2: match found
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolve_canonicalize: hostname 192.0.2.44 is address
debug2: ssh_connect_direct
debug1: Connecting to 192.0.2.44 [192.0.2.44] port 22.
debug1: Connection established.
[...]
debug1: Authenticating to 192.0.2.44:22 as 'ansible'
[...]
Last login: Sun Mar  1 17:09:46 2020 from 94.78.180.181
ansible@lxd:~$

Connect to the guest on port 23.

$ ssh 192.0.2.44 -p 23 -vv
OpenSSH_8.1p1 Ubuntu-5, OpenSSL 1.1.1d  10 Sep 2019
debug1: Reading configuration data /home/milosz/.ssh/config
debug2: checking match for 'Host 192.0.2.44 exec "test %p = 23"' host 192.0.2.44 originally 192.0.2.44
debug1: Executing command: 'test 23 = 23'
debug2: match found
debug2: checking match for 'Host 192.0.2.44 exec "test %p = 22"' host 192.0.2.44 originally 192.0.2.44
debug1: Executing command: 'test 23 = 22'
debug2: match not found
debug2: checking match for 'User ansible' host 192.0.2.44 originally 192.0.2.44
debug2: match not found
debug2: checking match for 'User milosz' host 192.0.2.44 originally 192.0.2.44
debug2: match found
debug2: checking match for 'LocalUser milosz' host 192.0.2.44 originally 192.0.2.44
debug2: match found
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolve_canonicalize: hostname 192.0.2.44 is address
debug2: ssh_connect_direct
debug1: Connecting to 192.0.2.44 [192.0.2.44] port 23.
debug1: Connection established.
[...]
debug1: Authenticating to 192.0.2.44:23 as 'milosz'
[...]
Last login: Sun Mar  1 17:10:06 2020 from 172.16.32.1
milosz@gitlab:~$

This is very cool!

Additional information

Inspect ssh_config manual page lo learn that you can use keywords like %h (remote hostname), %p (remote port), %r (remote username), %u (local username) and many more.