1

（date、someValue）形式の2つのCSVファイルをマージすると、重複するレコードがいくつか表示されます。

レコードを半分に減らすと、問題は解決します。ただし、両方のファイルのサイズを2倍にすると、悪化します。助けに感謝します！

私のコード：

i = pd.DataFrame.from_csv('i.csv')
i = i.reset_index()
e = pd.DataFrame.from_csv('e.csv')
e = e.reset_index()

total_df = pd.merge(i, e, right_index=False, left_index=False,
                    right_on=['date'], left_on=['date'], how='left')
total_df = total_df.sort(column='date')

（注：11 / 15、11 / 16、12 / 17、12 / 18の重複レコード。）

In [7]: total_df
Out[7]:
                  date  Cost  netCost
25 2012-11-15 00:00:00     1        2
26 2012-11-15 00:00:00     1        2
31 2012-11-16 00:00:00     1        2
32 2012-11-16 00:00:00     1        2
37 2012-11-17 00:00:00     1        2
2  2012-11-18 00:00:00     1        2
5  2012-11-19 00:00:00     1        2
8  2012-11-20 00:00:00     1        2
11 2012-11-21 00:00:00     1        2
14 2012-11-22 00:00:00     1        2
17 2012-11-23 00:00:00     1        2
20 2012-11-24 00:00:00     1        2
23 2012-11-25 00:00:00     1        2
29 2012-11-26 00:00:00     1        2
35 2012-11-27 00:00:00     1        2
0  2012-11-28 00:00:00     1        2
3  2012-11-29 00:00:00     1        2
6  2012-11-30 00:00:00     1        2
9  2012-12-01 00:00:00     1        2
12 2012-12-02 00:00:00     1        2
15 2012-12-03 00:00:00     1        2
18 2012-12-04 00:00:00     1        2
21 2012-12-05 00:00:00     1        2
24 2012-12-06 00:00:00     1        2
30 2012-12-07 00:00:00     1        2
36 2012-12-08 00:00:00     1        2
1  2012-12-09 00:00:00     2        2
4  2012-12-10 00:00:00     2        2
7  2012-12-11 00:00:00     2        2
10 2012-12-12 00:00:00     2        2
13 2012-12-13 00:00:00     1        2
16 2012-12-14 00:00:00     2        2
19 2012-12-15 00:00:00     2        2
22 2012-12-16 00:00:00     2        2
27 2012-12-17 00:00:00     1        2
28 2012-12-17 00:00:00     1        2
33 2012-12-18 00:00:00     1        2
34 2012-12-18 00:00:00     1        2

i.csv

date,Cost
2012-11-15 00:00:00,1
2012-11-16 00:00:00,1
2012-11-17 00:00:00,1
2012-11-18 00:00:00,1
2012-11-19 00:00:00,1
2012-11-20 00:00:00,1
2012-11-21 00:00:00,1
2012-11-22 00:00:00,1
2012-11-23 00:00:00,1
2012-11-24 00:00:00,1
2012-11-25 00:00:00,1
2012-11-26 00:00:00,1
2012-11-27 00:00:00,1
2012-11-28 00:00:00,1
2012-11-29 00:00:00,1
2012-11-30 00:00:00,1
2012-12-01 00:00:00,1
2012-12-02 00:00:00,1
2012-12-03 00:00:00,1
2012-12-04 00:00:00,1
2012-12-05 00:00:00,1
2012-12-06 00:00:00,1
2012-12-07 00:00:00,1
2012-12-08 00:00:00,1
2012-12-09 00:00:00,2
2012-12-10 00:00:00,2
2012-12-11 00:00:00,2
2012-12-12 00:00:00,2
2012-12-13 00:00:00,1
2012-12-14 00:00:00,2
2012-12-15 00:00:00,2
2012-12-16 00:00:00,2
2012-12-17 00:00:00,1
2012-12-18 00:00:00,1

e.csv

date,netCost
2012-11-15 00:00:00,2
2012-11-16 00:00:00,2
2012-11-17 00:00:00,2
2012-11-18 00:00:00,2
2012-11-19 00:00:00,2
2012-11-20 00:00:00,2
2012-11-21 00:00:00,2
2012-11-22 00:00:00,2
2012-11-23 00:00:00,2
2012-11-24 00:00:00,2
2012-11-25 00:00:00,2
2012-11-26 00:00:00,2
2012-11-27 00:00:00,2
2012-11-28 00:00:00,2
2012-11-29 00:00:00,2
2012-11-30 00:00:00,2
2012-12-01 00:00:00,2
2012-12-02 00:00:00,2
2012-12-03 00:00:00,2
2012-12-04 00:00:00,2
2012-12-05 00:00:00,2
2012-12-06 00:00:00,2
2012-12-07 00:00:00,2
2012-12-08 00:00:00,2
2012-12-09 00:00:00,2
2012-12-10 00:00:00,2
2012-12-11 00:00:00,2
2012-12-12 00:00:00,2
2012-12-13 00:00:00,2
2012-12-14 00:00:00,2
2012-12-15 00:00:00,2
2012-12-16 00:00:00,2
2012-12-17 00:00:00,2
2012-12-18 00:00:00,2
4

2 に答える 2

1

これは、pandas0.7.3またはnumpy1.6のバグのようです。これは、マージされる列が日付（内部的にnumpy.datetime64に変換される）である場合にのみ発生します。私の解決策は、日付を文字列に変換することでした-

def _DatetimeToString(datetime64):
  timestamp = datetime64.astype(long)/1000000000
  return datetime.fromtimestamp(timestamp).strftime('%Y-%m-%d')

i = pd.DataFrame.from_csv('i.csv')
i = i.reset_index()
i['date'] = i['date'].map(_DatetimeToString)
e = pd.DataFrame.from_csv('e.csv')
e = e.reset_index()
i['date'] = i['date'].map(_DatetimeToString)

total_df = pd.merge(i, e, right_index=False, left_index=False,
                    right_on=['date'], left_on=['date'], how='left')
total_df = total_df.sort(column='date')
于 2012-12-20T23:13:55.283 に答える
1

この問題/バグは私にも起こりました。日時シリーズをマージしていませんでしたが、左側のデータフレームに日時シリーズがありました。私の解決策は、重複排除することでした。

len(pophist)

2347

pop_merged = pd.merge(left=pophist, right=df_labels, how='left', 
             left_on ='candidate', right_on ='Slug', indicator = True)

pop_merged.shape

3303

pop_merged2 = pop_merged.drop_duplicates() #note dedupping is required due to issue in how pandas handles datetime dtypes on merge.  

len(pop_merged2)

2347

于 2016-02-05T18:26:13.503 に答える