java - Is there a common Java utility to break a list into batches?

Question

I wrote myself a utility to break a list into batches of given size. I just wanted to know if there is already any apache commons util for this.

public static <T> List<List<T>> getBatches(List<T> collection,int batchSize){
    int i = 0;
    List<List<T>> batches = new ArrayList<List<T>>();
    while(i<collection.size()){
        int nextInc = Math.min(collection.size()-i,batchSize);
        List<T> batch = collection.subList(i,i+nextInc);
        batches.add(batch);
        i = i + nextInc;
    }

    return batches;
}

Please let me know if there any existing utility already for the same.

score 310 · Accepted Answer

Google Guavaからチェックアウト： Lists.partition(java.util.List, int)

それぞれ同じサイズのリストの連続するサブリストを返します（最終的なリストはこれよりも小さい場合があります）。たとえば[a, b, c, d, e]、パーティションサイズが3のリストをパーティション化すると、次のようになります。つまり、3つの要素と2つの要素の2つの内部リストをすべて元の順序で含む外部リストです[[a, b, c]。[d, e]]

score 91 · Accepted Answer

In case you want to produce a Java-8 stream of batches, you can try the following code:

public static <T> Stream<List<T>> batches(List<T> source, int length) {
    if (length <= 0)
        throw new IllegalArgumentException("length = " + length);
    int size = source.size();
    if (size <= 0)
        return Stream.empty();
    int fullChunks = (size - 1) / length;
    return IntStream.range(0, fullChunks + 1).mapToObj(
        n -> source.subList(n * length, n == fullChunks ? size : (n + 1) * length));
}

public static void main(String[] args) {
    List<Integer> list = Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14);

    System.out.println("By 3:");
    batches(list, 3).forEach(System.out::println);
    
    System.out.println("By 4:");
    batches(list, 4).forEach(System.out::println);
}

Output:

By 3:
[1, 2, 3]
[4, 5, 6]
[7, 8, 9]
[10, 11, 12]
[13, 14]
By 4:
[1, 2, 3, 4]
[5, 6, 7, 8]
[9, 10, 11, 12]
[13, 14]

score 22 · Accepted Answer

Use Apache Commons ListUtils.partition.

org.apache.commons.collections4.ListUtils.partition(final List<T> list, final int size)

score 19 · Accepted Answer

Another approach is to use Collectors.groupingBy of indices and then map the grouped indices to the actual elements:

    final List<Integer> numbers = range(1, 12)
            .boxed()
            .collect(toList());
    System.out.println(numbers);

    final List<List<Integer>> groups = range(0, numbers.size())
            .boxed()
            .collect(groupingBy(index -> index / 4))
            .values()
            .stream()
            .map(indices -> indices
                    .stream()
                    .map(numbers::get)
                    .collect(toList()))
            .collect(toList());
    System.out.println(groups);

Output:

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]

[[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11]]

score 13 · Accepted Answer

With Java 9 you can use IntStream.iterate() with hasNext condition. So you can simplify the code of your method to this:

public static <T> List<List<T>> getBatches(List<T> collection, int batchSize) {
    return IntStream.iterate(0, i -> i < collection.size(), i -> i + batchSize)
            .mapToObj(i -> collection.subList(i, Math.min(i + batchSize, collection.size())))
            .collect(Collectors.toList());
}

Using {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}, the result of getBatches(numbers, 4) will be:

[[0, 1, 2, 3], [4, 5, 6, 7], [8, 9]]

score 8 · Accepted Answer

I came up with this one:

private static <T> List<List<T>> partition(Collection<T> members, int maxSize)
{
    List<List<T>> res = new ArrayList<>();

    List<T> internal = new ArrayList<>();

    for (T member : members)
    {
        internal.add(member);

        if (internal.size() == maxSize)
        {
            res.add(internal);
            internal = new ArrayList<>();
        }
    }
    if (internal.isEmpty() == false)
    {
        res.add(internal);
    }
    return res;
}

score 7 · Accepted Answer

The following example demonstrates chunking of a List:

package de.thomasdarimont.labs;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class SplitIntoChunks {

    public static void main(String[] args) {

        List<Integer> ints = Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11);

        List<List<Integer>> chunks = chunk(ints, 4);

        System.out.printf("Ints:   %s%n", ints);
        System.out.printf("Chunks: %s%n", chunks);
    }

    public static <T> List<List<T>> chunk(List<T> input, int chunkSize) {

        int inputSize = input.size();
        int chunkCount = (int) Math.ceil(inputSize / (double) chunkSize);

        Map<Integer, List<T>> map = new HashMap<>(chunkCount);
        List<List<T>> chunks = new ArrayList<>(chunkCount);

        for (int i = 0; i < inputSize; i++) {

            map.computeIfAbsent(i / chunkSize, (ignore) -> {

                List<T> chunk = new ArrayList<>();
                chunks.add(chunk);
                return chunk;

            }).add(input.get(i));
        }

        return chunks;
    }
}

Output:

Ints:   [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
Chunks: [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11]]

score 6 · Accepted Answer

Here is a simple solution for Java 8+:

public static <T> Collection<List<T>> prepareChunks(List<T> inputList, int chunkSize) {
    AtomicInteger counter = new AtomicInteger();
    return inputList.stream().collect(Collectors.groupingBy(it -> counter.getAndIncrement() / chunkSize)).values();
}

score 6 · Accepted Answer

Here an example:

final AtomicInteger counter = new AtomicInteger();
final int partitionSize=3;
final List<Object> list=new ArrayList<>();
            list.add("A");
            list.add("B");
            list.add("C");
            list.add("D");
            list.add("E");
       
        
final Collection<List<Object>> subLists=list.stream().collect(Collectors.groupingBy
                (it->counter.getAndIncrement() / partitionSize))
                .values();
        System.out.println(subLists);

Input: [A, B, C, D, E]

Output: [[A, B, C], [D, E]]

You can find examples here: https://e.printstacktrace.blog/divide-a-list-to-lists-of-n-size-in-Java-8/

score 5 · Accepted Answer

There was another question that was closed as being a duplicate of this one, but if you read it closely, it's subtly different. So in case someone (like me) actually wants to split a list into a given number of almost equally sized sublists, then read on.

I simply ported the algorithm described here to Java.

@Test
public void shouldPartitionListIntoAlmostEquallySizedSublists() {

    List<String> list = Arrays.asList("a", "b", "c", "d", "e", "f", "g");
    int numberOfPartitions = 3;

    List<List<String>> split = IntStream.range(0, numberOfPartitions).boxed()
            .map(i -> list.subList(
                    partitionOffset(list.size(), numberOfPartitions, i),
                    partitionOffset(list.size(), numberOfPartitions, i + 1)))
            .collect(toList());

    assertThat(split, hasSize(numberOfPartitions));
    assertEquals(list.size(), split.stream().flatMap(Collection::stream).count());
    assertThat(split, hasItems(Arrays.asList("a", "b", "c"), Arrays.asList("d", "e"), Arrays.asList("f", "g")));
}

private static int partitionOffset(int length, int numberOfPartitions, int partitionIndex) {
    return partitionIndex * (length / numberOfPartitions) + Math.min(partitionIndex, length % numberOfPartitions);
}

score 4 · Accepted Answer

Using various cheats from the web, I came to this solution:

int[] count = new int[1];
final int CHUNK_SIZE = 500;
Map<Integer, List<Long>> chunkedUsers = users.stream().collect( Collectors.groupingBy( 
    user -> {
        count[0]++;
        return Math.floorDiv( count[0], CHUNK_SIZE );
    } )
);

We use count to mimic a normal collection index.
Then, we group the collection elements in buckets, using the algebraic quotient as bucket number.
The final map contains as key the bucket number, as value the bucket itself.

You can then easily do an operation on each of the buckets with:

chunkedUsers.values().forEach( ... );

score 3 · Accepted Answer

Similar to OP without streams and libs, but conciser:

public <T> List<List<T>> getBatches(List<T> collection, int batchSize) {
    List<List<T>> batches = new ArrayList<>();
    for (int i = 0; i < collection.size(); i += batchSize) {
        batches.add(collection.subList(i, Math.min(i + batchSize, collection.size())));
    }
    return batches;
}

score 1 · Accepted Answer

List<T> batch = collection.subList(i,i+nextInc);
->
List<T> batch = collection.subList(i, i = i + nextInc);

score 1 · Accepted Answer

Note that List#subList() returns a view of the underlying collection, which can result in unexpected consequences when editing the smaller lists - the edits will reflect in the original collection or may throw ConcurrentModificationException.

score 0 · Accepted Answer

Another approach to solve this, question:

public class CollectionUtils {

    /**
    * Splits the collection into lists with given batch size
    * @param collection to split in to batches
    * @param batchsize size of the batch
    * @param <T> it maintains the input type to output type
    * @return nested list
    */
    public static <T> List<List<T>> makeBatch(Collection<T> collection, int batchsize) {

        List<List<T>> totalArrayList = new ArrayList<>();
        List<T> tempItems = new ArrayList<>();

        Iterator<T> iterator = collection.iterator();

        for (int i = 0; i < collection.size(); i++) {
            tempItems.add(iterator.next());
            if ((i+1) % batchsize == 0) {
                totalArrayList.add(tempItems);
                tempItems = new ArrayList<>();
            }
        }

        if (tempItems.size() > 0) {
            totalArrayList.add(tempItems);
        }

        return totalArrayList;
    }

}

score 0 · Accepted Answer

A one-liner in Java 8 would be:

import static java.util.function.Function.identity;
import static java.util.stream.Collectors.*;

private static <T> Collection<List<T>> partition(List<T> xs, int size) {
    return IntStream.range(0, xs.size())
            .boxed()
            .collect(collectingAndThen(toMap(identity(), xs::get), Map::entrySet))
            .stream()
            .collect(groupingBy(x -> x.getKey() / size, mapping(Map.Entry::getValue, toList())))
            .values();

}

score 0 · Accepted Answer

You can use below code to get the batch of list.

Iterable<List<T>> batchIds = Iterables.partition(list, batchSize);

You need to import Google Guava library to use above code.

score 0 · Accepted Answer

Here's a solution using vanilla java and the super secret modulo operator :)

Given the content/order of the chunks doesn't matter, this would be the easiest approach. (When preparing stuff for multi-threading it usually doesn't matter, which elements are processed on which thread for example, just need an equal distribution).

public static <T> List<T>[] chunk(List<T> input, int chunkCount) {
    List<T>[] chunks = new List[chunkCount];

    for (int i = 0; i < chunkCount; i++) {
        chunks[i] = new LinkedList<T>();
    }

    for (int i = 0; i < input.size(); i++) {
        chunks[i % chunkCount].add(input.get(i));
    }

    return chunks;
}

Usage:

    List<String> list = Arrays.asList("a", "b", "c", "d", "e", "f", "g", "h", "i", "j");

    List<String>[] chunks = chunk(list, 4);

    for (List<String> chunk : chunks) {
        System.out.println(chunk);
    }

Output:

[a, e, i]
[b, f, j]
[c, g]
[d, h]

score -2 · Accepted Answer

import com.google.common.collect.Lists;

List<List<T>> batches = Lists.partition(List<T>,batchSize)

Use Lists.partition(List,batchSize). You need to import Lists from google common package (com.google.common.collect.Lists)

It will return List of List<T> with and the size of every element equal to your batchSize.

java - Is there a common Java utility to break a list into batches?

19 に答える 19

Related

Reference