Gems are only deducted when used. If you notice a lower balance, it indicates they were spent on one of the following in-game actions:

1.
Restoring or Continuing a streak
Gems can be used to maintain your bingo streak, preventing it from resetting and allowing you to earn more rewards. 

Image
Image

2.
Cannon blast
This option lets you fire a Cannon Blast, which adds extra daubs after the round ends. It gives you another chance to complete a bingo or secure extra rewards. 

Image

3.
Purchasing power-ups from the Store
Gems may also be spent in the Store to buy specific power-ups or bundles. 

ImageImage