2日前にPRMを使用してMySQLマスターのフェイルオーバーをセットアップしようとしていましたが、すべての問題はここから始まります。
起動後corosync、MySQLがハングし、ソケットファイルが欠落し、最初の間違いはkill -9。
古いマスターをで回復できませんinnodb_force_recovery = 4。スレーブをマスターに昇格させましたが、同じ問題が発生しました。
これは私の設定です。
[mysqld]
# Settings user and group are ignored when systemd is used (fedora >= 15).
# If you need to run mysqld under different user or group,
# customize your systemd unit file for mysqld according to the
# instructions in http://fedoraproject.org/wiki/Systemd
user=mysql
datadir=/data2/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
## Files
back_log = 300
open-files-limit = 8192
open-files = 1024
skip-external-locking
skip-name-resolve
## Logging
relay-log = mysqld-relay-bin
relay-log-index = mysqld-relay-bin.index
#log = mysql-gen.log
log_warnings
log_bin = mysql-bin
#log_queries_not_using_indexes
max_binlog_size = 256M #max size for binlog before rolling
expire_logs_days = 4 #binlog files older than this will be purged
## Per-Thread Buffers * (max_connections) = total per-thread mem usage
thread_stack = 256K #default: 32bit: 192K, 64bit: 256K
sort_buffer_size = 1M #default: 2M, larger may cause perf issues
read_buffer_size = 1M #default: 128K, change in increments of 4K
read_rnd_buffer_size = 1M #default: 256K
join_buffer_size = 1M #default: 128K
binlog_cache_size = 64K #default: 32K, size of buffer to hold TX queries
## total per-thread buffer memory usage: 8832000K = 8.625GB
## Query Cache
query_cache_size = 32M #global buffer
query_cache_limit = 512K #max query result size to put in cache
## Connections
max_connections = 2000 #multiplier for memory usage via per-thread buffers
max_connect_errors = 100 #default: 10
concurrent_insert = 2 #default: 1, 2: enable insert for all instances
connect_timeout = 30 #default -5.1.22: 5, +5.1.22: 10
max_allowed_packet = 32M #max size of incoming data to allow
## Default Table Settings
sql_mode = NO_AUTO_CREATE_USER
## Table and TMP settings
max_heap_table_size = 1G #recommend same size as tmp_table_size
bulk_insert_buffer_size = 1G #recommend same size as tmp_table_size
tmp_table_size = 1G #recommend 1G min
#tmpdir = /data/mysql-tmp0:/data/mysql-tmp1 #Recommend using RAMDISK for tmpdir
## Table cache settings
table_cache = 512 #5.0.x <default: 64>
table_open_cache = 512 #5.1.x, 5.5.x <default: 64>
## Thread settings
thread_concurrency = 16 #recommend 2x CPU cores
thread_cache_size = 100 #recommend 5% of max_connections
## MyISAM Engine
key_buffer = 1M #global buffer
myisam_sort_buffer_size = 128M #index buffer size for creating/altering indexes
myisam_max_sort_file_size = 256M #max file size for tmp table when creating/alering indexes
myisam_repair_threads = 4 #thread quantity when running repairs
myisam_recover = BACKUP #repair mode, recommend BACKUP
## InnoDB Plugin Independent Settings
#innodb_data_home_dir = /data2/var/lib/mysql
#innodb_data_file_path = ibdata1:128M;ibdata2:10M:autoextend
#innodb_log_file_size = 512M #64G_RAM+ = 768, 24G_RAM+ = 512, 8G_RAM+ = 256, 2G_RAM+ = 128
#innodb_log_files_in_group = 4 #combined size of all logs <4GB. <2G_RAM = 2, >2G_RAM = 4
innodb_buffer_pool_size = 32G #global buffer
innodb_additional_mem_pool_size = 4M #global buffer
innodb_status_file #extra reporting
innodb_file_per_table #enable always
innodb_flush_log_at_trx_commit = 2 #2/0 = perf, 1 = ACID
innodb_table_locks = 0 #preserve table locks
innodb_log_buffer_size = 128M #global buffer
innodb_lock_wait_timeout = 60
innodb_thread_concurrency = 16 #recommend 2x core quantity
innodb_commit_concurrency = 16 #recommend 4x num disks
#innodb_flush_method = O_DIRECT #O_DIRECT = local/DAS, O_DSYNC = SAN/iSCSI
innodb_support_xa = 0 #recommend 0, disable xa to negate extra disk flush
skip-innodb-doublewrite
## Binlog sync settings
## XA transactions = 1, otherwise set to 0 for best performance
sync_binlog = 1
## TX Isolation
transaction-isolation = REPEATABLE-READ #REPEATABLE-READ req for ACID, SERIALIZABLE req XA
## Per-Thread Buffer memory utilization equation:
#(read_buffer_size + read_rnd_buffer_size + sort_buffer_size + thread_stack + join_buffer_size + binlog_cache_size) * max_connections
## Global Buffer memory utilization equation:
# innodb_buffer_pool_size + innodb_additional_mem_pool_size + innodb_log_buffer_size + key_buffer_size + query_cache_size
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Semisynchronous Replication
# http://dev.mysql.com/doc/refman/5.5/en/replication-semisync.html
# uncomment next line on MASTER
;plugin-load=rpl_semi_sync_master=semisync_master.so
# uncomment next line on SLAVE
;plugin-load=rpl_semi_sync_slave=semisync_slave.so
# Others options for Semisynchronous Replication
;rpl_semi_sync_master_enabled=1
;rpl_semi_sync_master_timeout=10
;rpl_semi_sync_slave_enabled=1
# http://dev.mysql.com/doc/refman/5.5/en/performance-schema.html
;performance_schema
thread_cache_size = 4
query_cache_type = 1
slow-query-log
long_query_time = 10
slow_query_log_file = /var/log/mysql/mysql-slow.log
log-warnings = 2
# slave config
skip-slave-start
server-id = 22
#read_only = 1
skip-name-resolve
log-bin=/data2/var/log/mysql/mysql-bin
relay_log_purge=0
binlog-format=MIXED
#expire_logs_days = 31
log-bin-trust-function-creators = 1
slave-skip-errors = 1062,1146,1032
replicate-wild-ignore-table=%.norep%
#testing
table_definition_cache = 5000
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
このマシンは、マスターがこの問題に直面した後、マスターに昇格します（理由がわからないため、単にオフにします）。
mysql-server-5.5.23-1.el5.remiバージョンを使用しています。
これはMySQLログファイルです。
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Last MySQL binlog file position 0 795, file name /data/var/log/mysql/mysql-bin.000801
120719 10:25:47 InnoDB: Waiting for the background threads to start
120719 10:25:48 InnoDB: 1.1.8 started; log sequence number 8068861682758
120719 10:25:48 [Note] Recovering after a crash using /data/var/log/mysql/mysql-bin
120719 10:25:48 [Note] Starting crash recovery...
120719 10:25:48 [Note] Crash recovery finished.
120719 10:25:49 [Warning] 'proxies_priv' entry '@ root@svr201NTC-647.localdomain' ignored in --skip-name-resolve mode.
120719 10:25:49 [Note] Event Scheduler: Loaded 27 events
120719 10:25:49 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.23-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) by Remi
120719 10:25:56 [Warning] Aborted connection 23 to db: 'reportingdb' user: 'adproject' host: '192.168.3.87' (Got an error reading communication packets)
120719 10:25:57 [Warning] Aborted connection 44 to db: 'reportingdb' user: 'adproject' host: '192.168.3.87' (Got an error reading communication packets)
120719 10:25:57 [Warning] Aborted connection 1 to db: 'reportingdb' user: 'adproject' host: '192.168.5.192' (Got an error reading communication packets)
03:25:57 UTC - mysqld got signal 8 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=19
max_threads=300
thread_count=16
connection_count=16
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 664409 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x1286fcd0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 49a340b8 thread_stack 0x40000
/usr/libexec/mysqld(my_print_stacktrace+0x2e)[0x7ae02e]
/usr/libexec/mysqld(handle_fatal_signal+0x3e2)[0x679da2]
/lib64/libpthread.so.0[0x3154a0ebe0]
/usr/libexec/mysqld(_ZN12ha_partition21min_rows_for_estimateEv+0x4f)[0x9223af]
[0x49a2bf60]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (11a7ad38): SELECT * FROM (
SELECT B.bannerid, SUM(realclick)clicks, SUM(totalview)views,
ROUND((MAX(uv) + (SUM(uv) - MAX(uv))*0.1)*1.1)`uviews` ,
CAST(IF(SUM(realclick)=0 OR SUM(totalview)=0,'N/A',CONCAT(ROUND(SUM(realclick)*100/(SUM(totalview)),2))) as CHAR) `CTR`,
SUM(A.`money`) money
FROM `ox_banners` B
INNER JOIN `v3_ban_date_cpm7k` AS A ON B.`campaignid` = A.`campaignid` AND B.`bannerid` = A.`bannerid`
WHERE A.`bannerid` = NAME_CONST('_bannerid',144660) AND A.`campaignid` >0
AND A.`dt` BETWEEN NAME_CONST('_start',_latin1'2012-08-20' COLLATE 'latin1_swedish_ci') AND NAME_CONST('_end',_latin1'2012-08-26' COLLATE 'latin1_swedish_ci')) A
Connection ID (thread ID): 47
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
テーブル内に存在しないパーティションにアクセスするクエリがあると、MySQLがクラッシュするように聞こえます2012-08-20（上記のクエリのとに注意してください2012-08-26）。私を驚かせたのは、それが以前には起こらなかったことです。
利用可能なRAMがたくさんあります。
free -m
total used free shared buffers cached
Mem: 48289 31600 16688 0 666 23848
-/+ buffers/cache: 7085 41203
Swap: 8189 54 8135
データディレクトリファイルのアクセス許可。
drwxr-xr-x 48 mysql mysql 118784 Jul 18 16:20 /data2/var/lib/mysql
そしてmysqld.log。
(-rw-r----- 1 mysql mysql 36M Jul 18 17:14 /var/log/mysqld.log)
では、テーブルがクラッシュする原因とその修正方法をどのように見つけることができますか？
7月19日：ログを更新します。
私はこのガイドに従ってデバッグしようとしていますが、役に立ちません：
# resolve_stack_dump -s /tmp/mysqld.sym -n mysqld.stack | c++filt
0x7ae02e my_print_stacktrace + 46
0x679da2 handle_fatal_signal + 994
0x3fed00ebe0 _end + -335143280
0x9223af ha_partition::min_rows_for_estimate() + 79
0x417c2ff0 _end + 1082221216
# addr2line -fie /usr/libexec/mysqld 0x2e
??
??:0
# addr2line -fie /usr/libexec/mysqld 0x3e2
??
??:0
# addr2line -fie /usr/libexec/mysqld 0x4f
??
??:0