AFAIK you can't get ButtonGroup from ResourceEdit#GUI. You can only possible to get RadioButton group name. But possible to add the RadioButton into ButtonGroup through your code. See the following code,
For calling StateMachine() constructor(use this code inside of constructor),
Form form = (Form) this.startApp(resources, null, true);
RadioButton rb1 = this.findRadioButton(form);
RadioButton rb = this.findRadioButton1(form);
ButtonGroup bg = new ButtonGroup();
bg.add(rb);
bg.add(rb1);
bg.setSelected(0);
For calling StateMachine(String resFile) constructor(use this code inside of your MIDlet class),
StateMachine sm = new StateMachine("/Sample.res");
RadioButton rb1 = sm.findRadioButton(Display.getInstance().getCurrent());
RadioButton rb = sm.findRadioButton1(Display.getInstance().getCurrent());
ButtonGroup bg = new ButtonGroup();
bg.add(rb);
bg.add(rb1);
bg.setSelected(0);