Is it legal? Yes, it is. Is it right? That's a different question.
Here's the deal, if the judge told you this then that is your answer. He or she will not excuse you because you are a single working mom and can't afford to miss work.
Now, if you inform the judge that you respect that answer and fully understand the court's decision but you don't believe you would be able to focus on the case due to the stress you'll be feeling from the financial burden being there will cause your family, you may get a different answer.
Here's the thing, if you inform a judge that you will not be able to focus on the evidence being presented or make a sound decision based only on the evidence due to your mental state, they'd be risking a mistrial by keeping you on the jury.
Even if the judge still insisted that you stay, I'm sure the attorneys would be very eager to eliminate you during the juror selection process.