It is most important that the individuals in the group are motivated by a strong set of shared values and a shared outlook. This will help you to work together in harmony and to weather the inevitable stresses and strains that come with undertaking a major project.
Clearly, everyone in the group must be committed to helping refugees, but it is just as vital that you all feel community sponsorship is the best way to do this and is definitely the path you want to take.
An initial core group probably needs to number at least 5 people, ideally more. The group is then likely to grow as you identify the particular skills and extra support you will need in order to welcome and support a refugee family.
Over the course of a sponsorship it is likely that many more people in your community will play some role in welcoming the family and helping out in particular ways.
That is one of the great things about sponsorship. It brings communities together around a very special shared goal.