Make Concurrent Requests in Java

Here’s how you can perform concurrent HTTP requests in Java using HttpClient and CompletableFuture to handle multiple requests asynchronously.

Here’s the Java code:

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.stream.Collectors;

public class Main {
    private static final HttpClient client = HttpClient.newHttpClient();
    public static CompletableFuture<Void> sendRequest(String query) {
        String apiKey = "YOUR-X-API-KEY";
        String url = "https://piloterr.com/api/v2/website/crawler?x_api_key=" + apiKey + "&query=" + query;
        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create(url))
                .GET()
                .build();
        return client.sendAsync(request, HttpResponse.BodyHandlers.ofString())
                .thenAccept(response -> {
                    if (response.statusCode() == 200) {
                        System.out.println("Response from " + query + ":\n" + response.body());
                    } else {
                        System.out.println("Request to " + query + " failed with status code: " + response.statusCode());
                    }
                })
                .exceptionally(ex -> {
                    System.out.println("Request to " + query + " failed: " + ex.getMessage());
                    return null;
                });
    }
    public static void scrapeConcurrently(List<String> urls) {
        List<CompletableFuture<Void>> futures = urls.stream()
                .map(Main::sendRequest)
                .collect(Collectors.toList());
        CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).join();
        System.out.println("Process Ended");
    }
    public static void main(String[] args) {
        List<String> urlsToScrape = List.of(
                "https://www.piloterr.com",
                "https://www.piloterr.com/blog"
        );
        scrapeConcurrently(urlsToScrape);
    }
}

Explanation:

1. HttpClient: The HttpClient class is used to send HTTP requests asynchronously.

2. CompletableFuture: This class provides a way to execute non-blocking, asynchronous operations. Each request is handled as a CompletableFuture, which allows all requests to run concurrently.

3. CompletableFuture.allOf: Waits for all CompletableFuture objects to complete before continuing. This ensures that all requests finish before the program ends.

How to Use:

• Replace "YOUR-X-API-KEY" with your actual API key.

• Modify the urlsToScrape list to include the URLs you want to scrape.

Benefits:

• Java’s CompletableFuture provides a simple way to run multiple asynchronous operations concurrently, improving performance and reducing response times.

• The solution is scalable and can be adapted for a large number of URLs.

This Java code offers an efficient way to handle concurrent HTTP requests, suitable for scenarios requiring high performance and concurrency.