Hmmmmm. That's a toughie.
I, personally, would not terminate immediately based on a single comment. I would clarify with mom and dad whether or not they have any concerns. If they do feel that the baby is not getting enough to eat, be detailed about your communication with them. Do they want you to write down what and how much he eats daily? Think about whether or not you would be willing to do that. Don't take this personally; but maybe try giving him a little more? Maybe he eats more than your daughter does/ used to, and you just aren't used to the portions. It's tricky at that age, because they can't verbalize "more please", etc.
That being said; you say you didn't really like them to begin with? I usually use the first month as a trial period. If during a month, I was still very uneasy with a family, I would let them go. It's true that the longer you wait, the tougher it will be to say "this isn't working out". Keep in mind that the first few weeks are an adjustment for everyone! You, the child and the parents are all getting used to each other. I think sometimes we need a little more patience in the beginning.
So, I would talk to them first. If you really feel like his eating is fine, but they insist he is starving, I would say something like: "You know, I have noticed that right around home time, he begins to fuss a little bit. (Even if he doesn't, really!) I think maybe he knows you will be here soon, and is just antsy to see you! I bet he just can't wait to play and have a snuggle." Like the earlier post said, he may not actually be hungry, just anxious to be with mom and dad. If this is the first child, they may be mistaking these cues?
If after this discussion, things are still quite unsettling for you, then I would think about saying good bye. Perhaps you just aren't a good match in the end.