How risky is too risky?

Optimists always say you have to take a risk in order to succeed in life, but where do we draw the line? How much of a risk is it until it becomes so much of a gamble where we could (and some do) potentially lose everything they had once had & were better off where they were beforehand? I am mainly referring to taking risk (s) in business and career & job changes. I once heard someone on YouTube say that he was so unhappy with his job that he decided to quit and travel the world. But let's be realistic here: How many people could afford to do such a thing? He said we should be doing a job (career) that we love and if you are not one of those people you should find something else that makes you happy. But come on, that sounds totally unrealistic. How many people could afford to give up a job where it is their way of survival because they are unsatisfied with what they are doing? How many people could afford to take the risk financially to start a business with hopes of becoming an entrepreneur and make a financial windfall? They spend all their money they have saved most of their entire life to start a new business without knowing if it'll take off and be worth millions OR go totally bankrupt & they will become flat broke over it. Do I sound negative here or realistic? Because I am not trying to come off that way, I am trying to sound realistic. I feel like people saying, "you have to take a risk" has to be put in perspective & within a certain extent. This phrase and optimistic way of living is too much in black and white, there needs to be a happy medium. I don't know about you, but I have had many unpleasant and unsatisfying jobs but I can't just pick up & quit until I find one that my heart desires. You know why? I CAN'T AFFORD TO!!!! The job (s) I have had (have) put a roof over my head and food on the table. If how you make ends meet wasn't part of this equation, then yes I would say go for it. But to just easily say find something that makes you happy, quit a job, embark in a new industry, spend all your $ on a new business venture with the risk of losing everything ($) I think is too much. Unless you are Ivanka Trump and could invest in any business venture (s) your heart desires from dipping in daddy's pot of gold then I would say this is more of a pipe dream.
 I would love to get another person's opinion who is walking in the same shoes as me who makes their own money and doesn't have income or help from other places. What do you think?