bash - CSVファイルの複数の列に基づいて高度にカスタマイズされた並べ替えを実行しますか？

Question

@区切り文字として使用する4列のCSVファイルがあります。例：

0001 @ fish @ animal @ eats worms

最初の列は、一意であることが保証されている唯一の列です。

列2、3、および4で4つのソート操作を実行する必要があります。

まず、列2が英数字でソートされます。この種の重要な機能は、列2内の重複するエントリが互いに隣接していることを保証する必要があることです。例：

@ a @ @
@ a @ @
@ a @ @
@ a @ @
@ a @ @
@ b @ @
@ b @ @
@ c @ @  
@ c @ @  
@ c @ @  
@ c @ @  
@ c @ @  

次に、最初の並べ替えで、行を2つのカテゴリに並べ替えます。最初の行は、「arch。」、「var。」、「ver。」、「anci」という単語を含まない行です。または「fam」。列4内の任意の場所。2行目（後にソートされます）は、これらの単語を含む行です。例：

@ a @ @ Does not have one of those words.
@ a @ @ Does not have one of those words.
@ a @ @ Does not have one of those words.
@ a @ @ Does not have one of those words.
@ a @ @ This sentence contains arch.
@ b @ @ Does not have one of those words.
@ b @ @ Has the word ver.
@ c @ @ Does not have one of those words.
@ c @ @ Does not have one of those words.
@ c @ @ Does not have one of those words.
@ c @ @ This sentence contains var.
@ c @ @ This sentence contains fam.
@ c @ @ This sentence contains fam.

最後に、2番目の並べ替えの個別のカテゴリ内でのみ並べ替え、「列3内に重複するエントリが最も多い」から「列3内に重複するエントリが最も少ない」までの行を並べ替えます。例：

@ a @ fish @ Does not have one of those words.
@ a @ fish @ Does not have one of those words.
@ a @ fish @ Does not have one of those words.
@ a @ tiger @ Does not have one of those words.
@ a @ bear @ This sentence contains arch.
@ b @ fish @ Does not have one of those words.
@ b @ fish @ Has the word ver.
@ c @ bear @ Does not have one of those words.
@ c @ bear @ Does not have one of those words.
@ c @ fish @ Does not have one of those words.
@ c @ tiger @ This sentence contains var.
@ c @ tiger @ This sentence contains fam.
@ c @ bear @ This sentence contains fam.

ファイルを2列目、4列目でのキーワードの出現、3列目で最も一般的な重複から最も一般的でない重複で英数字で並べ替えるにはどうすればよいですか？

Answer 1

TXR：（http://www.nongnu.org/txr）

@(bind special-words ("arch." "var." "ver." "anci." "fam."))
@(bind ahash @(hash :equal-based))
@(repeat)
@id @@ @alpha @@ @animal @@ @words
@  (rebind words @(split-str words " "))
@  (bind record (id alpha animal words))
@  (do (push record [ahash alpha]))
@(end)
@(bind sorted-rec-groups nil)
@(do
   (defun popularity-sort (recs)
     (let ((histogram [group-reduce (hash)
                                    third (do inc @1)
                                    recs 0]))
      [sort recs > [chain third histogram]]))

   (dohash (key records ahash)
     (let (contains does-not combined)
       (each* ((r records)
               (w [mapcar fourth r]))
         (if (isec w special-words)
           (push r contains)
           (push r does-not)))
       (push (append (popularity-sort does-not)                                 
                     (popularity-sort contains))                                
             sorted-rec-groups)))
   (set sorted-rec-groups [sort sorted-rec-groups :
                                [chain first second]]))
@(output)
@  (repeat)
@    (repeat)
@(rep)@{sorted-rec-groups} @@ @(last)@{sorted-rec-groups " "}@(end)
@    (end)
@  (end)
@(end)

データ：

0001 @ b @ fish @ Does not have one of those words.
0002 @ a @ bear @ Does not have one of those words.
0003 @ b @ bear @ Has the word ver.
0004 @ a @ fish @ Does not have one of those words.
0005 @ c @ bear @ Does not have one of those words.
0006 @ c @ bear @ Does not have one of those words.
0007 @ a @ fish @ Does not have one of those words.
0008 @ c @ fish @ Does not have one of those words.
0009 @ a @ fish @ Does not have one of those words.
0010 @ c @ tiger @ This sentence contains var.
0011 @ c @ bear @ This sentence contains fam.
0012 @ a @ fish @ Does not have one of those words.
0013 @ c @ tiger @ This sentence contains fam.

走る：

$ txr sort.txr data.txt 
0004 @ a @ fish @ Does not have one of those words.
0007 @ a @ fish @ Does not have one of those words.
0009 @ a @ fish @ Does not have one of those words.
0012 @ a @ fish @ Does not have one of those words.
0002 @ a @ bear @ Does not have one of those words.
0001 @ b @ fish @ Does not have one of those words.
0003 @ b @ bear @ Has the word ver.
0005 @ c @ bear @ Does not have one of those words.
0006 @ c @ bear @ Does not have one of those words.
0008 @ c @ fish @ Does not have one of those words.
0010 @ c @ tiger @ This sentence contains var.
0013 @ c @ tiger @ This sentence contains fam.
0011 @ c @ bear @ This sentence contains fam.

Answer 2

始めるのに役立つ最初の質問への回答は次のとおりです。

sort data -t "@" -k 2,2 -k 3,4

使い方：

• -tは、「@」記号であるフィールド区切り文字を指定します。
• -k 2,2は、フィールド2でソートすることを意味します
• -k 3,4は、フィールド3、次にフィールド4でソートして同点を解決することを意味します。

Answer 3

これがRubyの解決策です。

#!/usr/bin/env ruby

class Row

  SEPARATOR = " @ "

  attr_accessor :cols

  def initialize(text)
    @cols = text.chomp.split(SEPARATOR)
    @cols.size == 4 or raise "Expected text to have four columns: #{text}"
    duplicate_increment
  end

  def has_words?
    cols[3]=~/arch\.|var\.|ver\.|anci\.|fam\./ ? true : false
  end

  def to_s
    SEPARATOR + 
      @cols[1,3].join(SEPARATOR) +
      " -- id:#{cols[0]} duplicates:#{duplicate_count}"
  end

  ### Comparison

  def <=>(other)
    other or raise "Expected other to exist"
    cmp = self.cols[1] <=> other.cols[1]
    return cmp if cmp !=0
    cmp = (self.has_words? ? 1 : -1) <=> (other.has_words? ? 1 : -1)
    return cmp if cmp !=0
    other.duplicate_count <=> self.duplicate_count 
  end

  ### Track duplicate entries

  @@duplicate_count = Hash.new{|h,k| h[k]=0}

  def duplicate_key
    [cols[1],has_words?]
  end

  def duplicate_count
    @@duplicate_count[duplicate_key]
  end

  def duplicate_increment
    @@duplicate_count[duplicate_key] += 1
  end

end

### Main

lines = ARGF
rows = lines.map{|line| Row.new(line) }
sorted_rows = rows.sort
sorted_rows.each{|row| puts row }

入力：

0001 @ b @ fish @ text
0002 @ a @ bear @ text
0003 @ b @ bear @ ver.
0004 @ a @ fish @ text
0005 @ c @ bear @ text
0006 @ c @ bear @ text
0007 @ a @ fish @ text
0008 @ c @ fish @ text
0009 @ a @ fish @ text
0010 @ c @ lion @ var.
0011 @ c @ bear @ fam.
0012 @ a @ fish @ text
0013 @ c @ lion @ fam.

出力：

$ cat data.txt | ./sorter.rb 
@ a @ fish @ text -- id:0007 duplicates:5
@ a @ bear @ text -- id:0002 duplicates:5
@ a @ fish @ text -- id:0012 duplicates:5
@ a @ fish @ text -- id:0004 duplicates:5
@ a @ fish @ text -- id:0009 duplicates:5
@ b @ fish @ text -- id:0001 duplicates:1
@ b @ bear @ ver. -- id:0003 duplicates:1
@ c @ bear @ text -- id:0005 duplicates:3
@ c @ fish @ text -- id:0008 duplicates:3
@ c @ bear @ text -- id:0006 duplicates:3
@ c @ lion @ var. -- id:0010 duplicates:3
@ c @ bear @ fam. -- id:0011 duplicates:3
@ c @ lion @ fam. -- id:0013 duplicates:3

Answer 4

 q

まず、「csv」を読み込んで正しい形にします。テストデータは私のコンピューターでは「ワーム」と呼ばれていますが、qはファイル名「type」として文字列を使用しないため（インジェクション攻撃などから保護するため）、hsymを使用して「ファイル名」を作成する必要があります。

t:flip `id`a`b`c!("SSSS";"@")0:hsym`worms;

次に、どの「4番目のフィールド」エントリにあなたの単語の1つが含まれているかを調べました。likeを使用してビットマップを作成し、それを各行（左）、次に各パターン（右）に適用して、単語が存在しない場合は0、単語が存在しない場合は1を取得します。

t:update p:any each c like/:\:("*arch.*";"*var.*";"*ver.*";"*anci.*";"*fam.*") from t;

次に、重複の数を見つけたいと思います。これは、列2（a）、列3（b）、および現在のカテゴリ内の行数です。

t:update d:neg count i by a,b,p from t;

最後に、カウントを無効にしたため、すべての値が「同じように進む」ので、次の3つの列で簡単に並べ替えることができます。

`a`p`d xasc t

Answer 5

これはあなたのために働くかもしれません（非常にエレガントではありません！）：

sed 's/[^@]*@\([^@\]*\)@\([^@]*\)/\1\t\2\t&/;h;s/@/&\n/3;s/.*\n//;/\(arch\|var\|ver\|anci\|fam\)\./!ba;s/.*/1/;bb;:a;s/.*/0/;:b;G;s/\(.\)\n\([^\t]*\)/\2\t\1/' file |
sort | 
tee file1 |
sed 's/\(.*\)\t.*/\1/' |
uniq -c |
sed 's|^\s*\(\S*\) \(.*\t.*\t\(.*\)\)|/^\2/s/\3/\1/|' >file.sed
sed -f file.sed file1 |
sort -k1,2 -k3,3nr |
sed 's/\t/\n/3;s/.*\n//'
1 @ a @ fish @ Does not have one of those words.
2 @ a @ fish @ Does not have one of those words.  
3 @ a @ fish @ Does not have one of those words.
4 @ a @ tiger @ Does not have one of those words.
5 @ a @ bear @ This sentence contains arch.
6 @ b @ fish @ Does not have one of those words.
7 @ b @ fish @ Has the word ver.
8 @ c @ bear @ Does not have one of those words.
9 @ c @ bear @ Does not have one of those words.
10 @ c @ fish @ Does not have one of those words.
11 @ c @ tiger @ This sentence contains var.
12 @ c @ tiger @ This sentence contains fam.
13 @ c @ bear @ This sentence contains fam.

説明：

以下で構成されるソートキーを作成します。

1. 2番目のフィールド
2. 0/1：0は、arch。/var。/etcのない4番目のフィールドを表します。1はでそれらを表します。
3. 上記の2をソートした後、3番目のフィールドの重複の数。

ファイルは最終的に上記のキーを使用してソートされ、次にキーが削除されます。