The idea with VM scale sets are that they are a load balanced series of identical VMs. Think of an example of a web farm built using VM scale sets, you might have 5 identical VM instances in the web farm (scale set) but you would need to front these with a load balancer in order to direct the traffic to an instance of the VM scale set. The load balancer would be your public endpoint (public IP address) and your load balancer manages the traffic for you by forwarding it to an available instance in your VM scale set.
Otherwise you would be assigning public IP addresses to particular instances of the VM scale set which
Having said that, what you ask is possible although rarely used from my experience.