Betting sites calculate the amount they offer for a cash out in basically the same way they offer normal odds: calculating the probability of the result, and adding a margin for their own profit.
In practice it might look something like this
- Boston Bruins vs. New York Islanders
- Bruins to win @ -110
- $10 wager
The potential payout of this wager would be about $19 if you let it ride the entire game and won. So let's break it down into a few different scenarios after the end of the second period.
Bruins 2, Islanders 2 - We're basically at the start of the game here, and since the odds were so close to start this is more or less what the betting site anticipated. So we can reasonably assume the odds are 50%, and by cutting you potential winnings in half you would have a return of $9.50, with a margin applied by the betting site, you ending cash out might be around $8.75. Sure, this is less than your bet stake, but it is cash back in your pocket on a wager that is definitely not a sure thing.
Bruins 4, Islanders 1 - The game is well in hand, and you've got about an 85% chance of winning this wager, so the possible return looks like .85*$19, or $16.15. With a margin you could expect a cash out option of around $15, which is a guaranteed $5 profit. This is where you need to start weighing the risk vs. reward, since there is a good chance the Bruins will win, but then again you can have a certain profit by cashing out. In this scenario the sportsbook wants you to cash out, since they would limit their potential payout to you.
Bruins 0, Islanders 3 - And on the flip side, the Bruins played terribly and the Islanders are comfortably in the lead going into the final period, so the sportsbook is giving the bruins a 10% chance to win, making your return $1.90 (.10*$19). And with a margin, you would probably only be cashing out $1.50 all said and done. That's a tough pill to swallow, but hey, at least you have a little cash left instead of losing it all, and the betting site is happy to offer this cash out since they still make a nice profit.
Each betting site in the US calculates their cash out margins slightly different, but this is roughly the idea behind how it gets calculated, with each setup having the bookmaker price your current wager in a way that makes it beneficial to them if you take the money and run, but as we keep saying, cash in the hand may just be better than a bet in the slip.