末尾のゼロなしで部分をキャプチャし、それを正規表現の置換で使用できます。おそらく、小数点区切り記号 (例: "78.") で終わるのではなく、1 つの末尾のゼロ (例: "78.0") を保持して整理したいと考えています。
var s = "12304.56780000";
// N.B. Check the decimal separator
var re = new RegExp("([0-9]+\.[0-9]+?)(0*)$");
var t = s.replace(re, '$1'); // t = "12304.5678"
t="12304.00".replace(re, "$1"); // t="12304.0"
regex101からの説明:
/([0-9]+\.[0-9]+?)(0*)$/
1st Capturing group ([0-9]+\.[0-9]+?)
[0-9]+ match a single character present in the list below
Quantifier: + Between one and unlimited times, as many times as possible, giving back as needed [greedy]
0-9 a single character in the range between 0 and 9
\. matches the character . literally
[0-9]+? match a single character present in the list below
Quantifier: +? Between one and unlimited times, as few times as possible, expanding as needed [lazy]
0-9 a single character in the range between 0 and 9
2nd Capturing group (0*)
0* matches the character 0 literally
Quantifier: * Between zero and unlimited times, as many times as possible, giving back as needed [greedy]
$ assert position at end of the string