Use vagrant provisioner to update SSH authorized_keys every time the virtual machine is started.

Use run option to define whether provisioner should run once or always.

$ cat Vagrantfile 

Vagrant.configure("2") do |config|
  config.vm.box = "debian/buster64"

  config.vm.provider :virtualbox do |vb|
    vb.memory = "1024"
  end

  config.vm.provision :shell, :run => 'always', :privileged => false do |authorized_keys_provisoner|
    local_public_key = "#{Dir.home}/.ssh/milosz.pub"
    vagrant_authorized_keys = "/home/vagrant/.ssh/authorized_keys"
    
    ssh_pub_key = File.readlines("#{local_public_key}").first.strip
    authorized_keys_provisoner.inline = <<-SHELL
      grep --quiet "#{ssh_pub_key}" #{vagrant_authorized_keys} || (echo "#{ssh_pub_key}" >> /home/vagrant/.ssh/authorized_keys; \
                                                                   echo "SSH public key added to the authorized_keys file")
    SHELL
  end

  config.vm.provision :shell, :run => 'once', :path => "setup.sh", :privileged => true
end

The setup.sh shell script used in this example will run only once, which is a default.

$ cat setup.sh

#!/bin/bash

apt-get update
apt-get upgrade --yes
apt-get install --yes vim tmux

Sample machine start.

==> default: Importing base box 'debian/buster64'...    
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'debian/buster64' version '10.20210228.1' is up to date...
==> default: Setting the name of the VM: dns_default_1617534484895_66651            
==> default: Clearing any previously set network interfaces...                       
==> default: Preparing network interfaces based on configuration...                  
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default:
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default:
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
    default: The guest additions on this VM do not match the installed version of
    default: VirtualBox! In most cases this is fine, but in rare cases it can
    default: prevent things such as shared folders from working properly. If you see
    default: shared folder errors, please make sure the guest additions within the
    default: virtual machine match the version of VirtualBox you have installed on
    default: your host and reload your VM.
    default: 
    default: Guest Additions Version: 5.2.0 r68940
    default: VirtualBox Version: 6.1
==> default: Installing rsync to the VM...
==> default: Rsyncing folder: /home/milosz/Projects/vagrant/ => /vagrant
==> default: Running provisioner: shell...
==> default: Running provisioner: shell...
    default: Running: inline script
    default: SSH public key added to the authorized_keys file
==> default: Running provisioner: shell...
    default: Running: /tmp/vagrant-shell20210404-273366-1mhvfuc.sh
    default: Hit:1 http://deb.debian.org/debian buster InRelease
    default: Hit:2 http://security.debian.org/debian-security buster/updates InRelease
    default: Hit:3 http://deb.debian.org/debian buster-updates InRelease
    default: Hit:4 http://deb.debian.org/debian buster-backports InRelease
    default: Reading package lists...
    default: Reading package lists...
    default: Building dependency tree...
    default: Reading state information...
    default: Calculating upgrade...
    default: The following packages have been kept back:
    default:   linux-image-amd64
    default: The following packages will be upgraded:
    default:   base-files curl debian-archive-keyring groff-base grub-common grub-pc
    default:   grub-pc-bin grub2-common iputils-ping libbsd0 libcurl3-gnutls libcurl4
    default:   libnss-systemd libpam-systemd libpython3.7-minimal libpython3.7-stdlib
    default:   libssl1.1 libsystemd0 libudev1 openssl python3.7 python3.7-minimal systemd
    default:   systemd-sysv udev
    default: 25 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
    default: Need to get 18.8 MB of archives.
[...]
==> default: Machine 'default' has a post `vagrant up` message. This is a message
==> default: from the creator of the Vagrantfile, and not from Vagrant itself:
==> default: 
==> default: Vanilla Debian box. See https://app.vagrantup.com/debian for help and bug reports
ko-fi