# About PCJ

Java library for parallel computing in PGAS (Partitioned Global Address Space) paradigm.

PCJ is Java library for parallel computing in Java. It is based on the PGAS (Partitioned Global Address Space) paradigm. It allows for easy implementation in Java of any parallel algorithm. PCJ application can be run on laptop, workstation, cluster and HPC system including large supercomputers. It has been demonstrated that PCJ applications scale up to 200 000 cores.

Recently we have run PCJ application on the coud (AWS EC2) using both x86 and arm processors. Therefore you can run PCJ on most popular architectures indluding Intel KNL, Power, ARM and of course x86.

Current version of the library is 5.3.4. Comparing version 5.3.x to 5.2 and 5.1, it has improved performance due to use of *shifted tree* communication. New methos has been added: *PCJ.scatter()*, *PCJ.splitGroup().* New collective communication methods are avaliable: *PCJ.collect()*, *PCJ.asyncCollect()*,

For more changes see: <https://github.com/hpdcj/PCJ/releases/tag/v5.3.4>

The API documentation is also available online: [![javadoc](https://javadoc.io/badge2/pl.edu.icm.pcj/pcj/javadoc.svg)](https://javadoc.io/doc/pl.edu.icm.pcj/pcj)

## Using PCJ library

You can use PCJ Library by adding jar file to your project.

PCJ Library is also available on Maven Central Repository. For maven project, just add this dependency to your `pom.xml` file.

```markup
<dependency>
    <groupId>pl.edu.icm.pcj</groupId>
    <artifactId>pcj</artifactId>
    <version>5.3.4</version>
</dependency>
```

If you are using gradle, add those lines to your `build.gradle` file:

```gradle
implementation 'pl.edu.icm.pcj:pcj:5.3.4'
annotationProcessor 'pl.edu.icm.pcj:pcj:5.3.4'
```

### Building PCJ library project by your own

If you wish to compile project by your own, use these instructions:

* to package the jar: `./gradlew assemble` or `gradlew.bat assemble`
* to create javadoc: `./gradlew javadoc` or `gradlew.bat javadoc`

### Importing the project in eclipse

Execute `./gradlew eclipse`, start `eclipse`, and use `File -> Import : Existing Projects into Workspace`. See <http://gradle.org/docs/current/userguide/eclipse_plugin.html> for more information.

## References

* **M. Nowicki, Ł. Górski, P. Bała** [PCJ – Java Library for Highly Scalable HPC and Big Data Processing](https://ieeexplore.ieee.org/abstract/document/8514322) *2018 International Conference on High Performance Computing & Simulation (HPCS)*, pp:12-20 IEEE, 2018
* **M. Nowicki, Ł. Górski, P. Bała** [Scalable computing in Java with PCJ Library. Improved collective operations.](https://open.icm.edu.pl/items/5375ddd3-627c-45c7-b8fe-9130fa88a73a)" In: *Proceedings of International Symposium on Grids and Clouds 2021 - Proceedings of Science (ISGC2021)*, 2021, vol. 378, p. 7; [DOI: 10.22323/1.378.0007](https://doi.org/10.22323/1.378.0007)
* **M. Nowicki**, **M. Ryczkowska, Ł. Górski, M. Szynkiewicz, P. Bała** [PCJ - a Java library for heterogenous parallel computing](http://www.wseas.us/e-library/conferences/2016/barcelona/SECEA/SECEA-08.pdf) In: X. Zhuang (Ed.) *Recent Advances in Information Science (Recent Advances in Computer Engineering Series vol 36)* WSEAS Press 2016 pp. 66-72
* **M. Nowicki**, **Ł. Górski, P. Grabarczyk, P. Bała** [PCJ - Java library for high performance computing in PGAS model](http://ieeexplore.ieee.org/xpl/abstractAuthors.jsp?arnumber=6903687) In: W. W. Smari and V. Zeljkovic (Eds.) *2012 International Conference on High Performance Computing and Simulation (HPCS)* IEEE 2014 pp. 202-209
* **M. Nowicki**, **P. Bała** [PCJ-new approach for parallel computations in java](https://link.springer.com/chapter/10.1007/978-3-642-36803-5_8) In: P. Manninen, P. Oster (Eds.) *Applied Parallel and Scientific Computing*, LNCS 7782, Springer, Heidelberg (2013) pp. 115-125
* **M. Nowicki**, **P. Bała** [Parallel computations in Java with PCJ library](http://ieeexplore.ieee.org/xpl/login.jsp?tp=\&arnumber=6266941\&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D6266941) In: W. W. Smari and V. Zeljkovic (Eds.) *2012 International Conference on High Performance Computing and Simulation (HPCS)* IEEE 2012 pp. 381-387


# HPCC Award

HPC Challenge Award at Supecomputing'14.

![](https://3099193909-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LzRXzuZeeCrLWU5ZSz5%2Fuploads%2Fgit-blob-d2b24c25b347ee15c1a84d7cf8d95962e0a497e2%2FHPCC_2014_Award.PNG?alt=media)

PCJ received **HPC Challenge Class 2 Best Productivity Award**, which recognize the efficient way of programming parallel applications.

The awards were announced on November 18, 2014, in New Orleans, Louisiana, at SC14, the [International Conference for High Performance Computing, Networking, Storage and Analysis](http://sc14.supercomputing.org/).

The HPC Challenge benchmarks are benchmark programs designed to evaluate the overall performance of supercomputers in terms of processing performance based on 28 frequently used computational patterns in the field of scientific computation. At the contest using the HPC Challenge benchmarks, which takes place once a year, there are two classes of awards: Class 1, which measures benchmark performance values, and Class 2, which measures the productivity of programming language implementations.

The HPC Challenge Class 2 Award is a contest for programming languages used in developing HPC applications. This award is designed to evaluate both programming language productivity and performance for HPC Challenge.

**PCJ** (<http://pcj.icm.edu.pl>) is a library for Java language that helps to perform parallel and distributed calculations. It is able to work on the multicore systems connected with the typical interconnect such as ethernet or infiniband providing users with the uniform view across nodes. Development of the PCJ library has been realized with the EU support within CHIST-ERA framework (2014-2018).


# Downloads

PCJ is a library for Java language that helps to perform parallel and distributed calculations. It is able to work on the multicore systems connected with the typical interconnects.

<table data-header-hidden><thead><tr><th width="206">File</th><th>Description</th><th> Release date</th></tr></thead><tbody><tr><td>File</td><td>Description</td><td> Release date</td></tr><tr><td><strong>PCJ 5</strong></td><td></td><td></td></tr><tr><td><a href="https://repo1.maven.org/maven2/pl/edu/icm/pcj/pcj/5.3.4/pcj-5.3.4.jar"><strong>PCJ library 5.3.4 (jar file for Java >=8)</strong></a></td><td>PCJ Library for Java 8 or newest  (jar file, ver 5.3.3)</td><td>16.01.2026</td></tr><tr><td><a href="https://repo1.maven.org/maven2/pl/edu/icm/pcj/pcj/5.3.4/pcj-5.3.4-javadoc.jar"><strong>PCJ library 5.3.4 JavaDoc</strong></a></td><td>Java doc</td><td>16.01.2026</td></tr><tr><td><a href="https://repo1.maven.org/maven2/pl/edu/icm/pcj/pcj/5.3.0/pcj-5.3.0.jar">PCJ library 5.3.0 (jar file for Java 8)</a></td><td>PCJ Library for Java 8 or newest  (jar file, ver 5.3.0)</td><td>30.06.2021</td></tr><tr><td><a href="https://repo1.maven.org/maven2/pl/edu/icm/pcj/pcj/5.3.0/pcj-5.3.0-javadoc.jar">PCJ library 5.3.0 JavaDoc</a></td><td>Java doc</td><td>30.06.3021</td></tr><tr><td><a href="https://repo1.maven.org/maven2/pl/edu/icm/pcj/pcj/5.2.0/pcj-5.2.0.jar">PCJ library 5.2.0 (jar file for Java 8 of 7.05.2021)</a></td><td> PCJ Library for Java 8 (jar file, ver 5)</td><td> 7.05.2021</td></tr><tr><td><a href="https://repo1.maven.org/maven2/pl/edu/icm/pcj/pcj/5.2.0/pcj-5.2.0-javadoc.jar">PCJ library 5.2.0 </a><a href="https://repo1.maven.org/maven2/pl/edu/icm/pcj/pcj/5.2.0/pcj-5.2.0-javadoc.jar">JavaDocs</a></td><td> PCJ doc (Java docs)</td><td>7.05.2021</td></tr><tr><td> <strong>PCJ 4</strong></td><td></td><td></td></tr><tr><td>PCJ library 4.1 (jar file  for Java 8 of 9.01.2016)</td><td> PCJ Library for Java 8 (jar file, ver 4)</td><td> 9.01.2016</td></tr><tr><td>PCJ Java docs (zip file of 9.01.2016)</td><td> PCJ doc (Java docs)</td><td> 9.01.2016</td></tr><tr><td><strong>old version (not supported)</strong></td><td></td><td></td></tr><tr><td>PCJ library 4.0.1 (jar file  for Java 7 of 20.09.2014)</td><td> PCJ Library for Java 7 (jar file)</td><td> 20.09.2014</td></tr><tr><td>PCJ Java docs (zip file of 20.09.2014)</td><td> PCJ doc (Java docs)</td><td> 20.09.2014</td></tr><tr><td>PCJ library 3.0.0 (not available)</td><td> PCJ Library (jar file)  </td><td> 7.11.2013</td></tr></tbody></table>


# Examples

Code examples for PCJ 5

### [Hello World](/examples/hello-world)

### [Approximation of π using Monte Carlo](/examples/monte-carlo)

### [Approximation of π using integral](/examples/loop-parallelization)

### [Reduction](/examples/reduction)


# Running PCJ application

PCJ application can be run on different computer systems

The compilation and execution of the parallel applications especially while using some queueing system or another submission environment is not straightforward. In particular, the information abut the nodes parallel application will be running on is not available in advance or even during job submission but is determined when job starts execution. Most of the systems provide such information through the environment variables and files with the list of nodes used for job execution. The list of nodes, especially while miltiprocessor nodes are present can contain multiple lines with the same names. The multiple entries are used, for example while running MPI application, to start multiple instances of the parallel application on the single node. In the case of PCJ library the execution is simple. The most efficient mechanism is to start single Java Virtual Machine on each node. Within this JVM multiple PCJ threads will be run. While running on multiple node, adequate number of JVMs will be started, using ssh or mpiexec command. Please remember, that PCJ threads running within single JVM will use Java Concurrency Library to synchronize and to communicate. Communication between PCJ threads running within different JVMs will be performed using Java Sockets. In such situation in order to run PCJ application we will use two files:

* **`nodes.unique`** - file containing list of nodes used to run JVMs. In principle this list contains unique names (no duplicated names). This file is used by the mpiexec or other command to start parallel application.
* **`nodes.txt`**- file containing list of nodes used to start PCJ threads.This list may contain duplicated entries showing that on the particular node multiple PCJ threads will be started (within single JVM). The number of PCJ threads used to run application (PCJ.threadsCount()) will be equal to the number of lines (entries) in this file.
* **`PCJ.jar`** - PCJ library (replece with proper version eg. `PCJ-5.0.9.jar`).

### Laptop or workstation

Create `nodes.txt` file containing multiple lines with the name of computer (eg. localhost). Than execute application as Java application:

```bash
java -cp .:PCJ.jar HelloWorld
```

### Linux cluster with MPI library installed

Example commands which can be run from script or interactive shell:

```bash
mpiexec -hostsfile nodes.unique bash -c 'java -cp PCJ.jar HelloWorld'
```

### **Linux Cluster with PBS (Protable Batch System)**

The execution is similar to the case of Linux cluster. However, the proper script submitted to the queue to be prepared. This file contains defnition of the parameters passed to the queueining system. The parameters include number of nodes required (`nodes=128`) and indicate that 1 process per node will be executed (ppn=1). The execution of java application is preceded by the gathering list of the nodes allocated to the job by the queueing system. The unique list of nodes is then stored in the nodes.unique. Please remember that `nodes.unique` and `nodes.txt` can be different.

```bash
#!/bin/csh
#PBS -N go
#PBS -l nodes =128: ppn =1
#PBS -l mem =512 mb
#PBS -l walltime =0:10:00
#PBS
 
module load openmpi             # if neccessary
 
cat $PBS_NODEFILE > nodes.txt
uniq $PBS_NODEFILE > nodes.unique
mpiexec -hostsfile nodes.unique bash -c 'java -d64 -Xnoclassgc -Xrs -cp PCJ.jar HelloWorld'
```

### Cray XC40 with SLURM

The proper script submitted to the queue to be prepared. This file (*`go_xc40.sh`*) contains defnition of the parameters passed to the queueining system.

```bash
#!/bin/bash -l                                            
#SBATCH -N 132                     # Number of nodes
#SBATCH --ntasks-per-node 48       # Numer of tasks per node
#SBATCH --mem 5000                 # Required RAM
#SBATCH --time=00:10:00            # Required time
#SBATCH -A GES-00                  # Account

srun hostname > nodes.txt
srun -N 132 -n 132 -c 48 java -cp .:PCJ.jar HelloWorld        
```

### **IBM Power (AIX) with Load Leveler**

In order to optimize execution on the multinode systems like IBM Power 7, the PCJ application should exclusively use computer nodes. However, the number of applications running on each nodes is 1 which is Java VM. The poe command is used to invoke Java VM's on the nodes reserved for the execution.

```bash
#@ job_type = parallel
#@ node = 2
#@ tasks_per_node= 1
#@ queue
 

cat $LOADL_HOSTFILE > nodes.txt
uniq $LOADL_HOSTFILE > nodes.unique

poe "java -Xnoclassgc -Xmx6g -cp .:PCJ.jar HelloWorld" -hfile nodes.unique -statistic print -bindproc yes -task_affinity cor
```


# Hello World

Code example for PCJ 5: Print Hello World in parallel.

Calculations start from a special `StartPoint` class. That class contains main method (`public void main()`):

```java
import java.io.IOException;
import org.pcj.*;

public class HelloWorld implements StartPoint {

    public static void main(String[] args) throws IOException {
    String nodesFile  = "nodes.txt";
    PCJ.executionBuilder (PcjExample.class)
                .addNodes(new File("nodes.txt"))
                .start();
    }
    }

    @Override
    public void main() throws Throwable {
                System.out.println("Hello World from PCJ Thread " + PCJ.myId()
                                   + " out of " + PCJ.threadCount() );
            }
}
```

The compilation and execution requires *PCJ-5.0.9.jar* in the path:

`javac -cp .:PCJ.jar HelloWorld.java java -cp .:PCJ-5.0.9.jar HelloWorld`

The expected output is presented below:

`wrz 23, 2016 2:00:22 AM org.pcj.internal.InternalPCJ start INFO: PCJ version 5.0.0.SNAPSHOT-a728f5f built on 2016-09-13 00:58:48.180 CEST. wrz 23, 2016 2:00:22 AM org.pcj.internal.InternalPCJ start INFO: Starting HelloWorld with 4 threads (on 1 node)... Hello World from PCJ Thread 2 out of 4 Hello World from PCJ Thread 0 out of 4 Hello World from PCJ Thread 1 out of 4 Hello World from PCJ Thread 3 out of 4 wrz 23, 2016 2:00:22 AM org.pcj.internal.InternalPCJ start INFO: Completed HelloWorld with 4 threads (on 1 node) after 0h 0m 0s. BUILD SUCCESSFUL (total time: 0 seconds)`


# Monte Carlo

Estimate π using Monte Carlo method (parallelization of workload).

The program picks points at random inside the square. It then checks to see if the point is inside the circle (it knows it's inside the circle if *x^2 + y^2 < R^2*, where *x* and *y* are the coordinates of the point and *R* is the rad\_i\_us of the circle).

The program keeps track of how many points it's picked (*nAll*) and how many of those points fell inside the circle (`circleCount`).

In the parallel version, the work is divided among threads, i.e. each traed is performing `nAll / PCJ.threadsCount()` attempts. Each thread counts points inside circle.

Finally, the parrial sums are communicated to the procesor 0.

```javascript
 import java.util.Random;
 import org.pcj.NodesDescription;
 import org.pcj.PCJ;
 import org.pcj.StartPoint;
 import org.pcj.Storage;
 import org.pcj.PcjFuture;
 import org.pcj.RegisterStorage;

 @RegisterStorage(PcjExamplePiMC.Shared.class)
 public class PcjExamplePiMC implements StartPoint {

     @Storage(PcjExamplePiMC.class)
     enum Shared { c }
     long c;

     @Override
     public void main() {
         PCJ.barrier();
         Random r = new Random();

         long nAll = 1000000;
         long n = nAll / PCJ.threadCount();
         double Rsq = 1.0;
         long circleCount;
 //Calculate
         circleCount = 0;
         double time = System.nanoTime();

         for (long i = 0; i < n; i++) {
             double x = 2.0 * r.nextDouble() - 1.0;
             double y = 2.0 * r.nextDouble() - 1.0;
             if ((x * x + y * y) < Rsq) {
                 circleCount++;
             }
         }

         c = circleCount;
         PCJ.barrier();
   // Communicate results
         PcjFuture cL[] = new PcjFuture[PCJ.threadCount()];

         long c0 = c;
         if (PCJ.myId() == 0) {
             for (int p = 1; p < PCJ.threadCount(); p++) {
                 cL[p] = PCJ.asyncGet(p, Shared.c);
             }
             for (int p = 1; p < PCJ.threadCount(); p++) {
                 c0 = c0 + (long) cL[p].get();
             }
         }

         PCJ.barrier();

         double pi = 4.0 * (double) c0 / (double) nAll;
         time = System.nanoTime() - time;
 // Print results
         if (PCJ.myId() == 0) {
             System.out.println(pi + " " + time * 1.0E-9);
         }
     }

     public static void main(String[] args) {
    String nodesFile  = "nodes.txt";
    PCJ.executionBuilder (PcjExample.class)
                .addNodes(new File("nodes.txt"))
                .start();
    }
     }
 }  
```

The code scales linearly with the numbers of processors.

![The performance of the code to approximate π using Monte Carlo method.](https://3099193909-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LzRXzuZeeCrLWU5ZSz5%2Fuploads%2Fgit-blob-f5ec6ea352ff7a6e0c6a354bcf137a64441ea4e8%2Fpcj_example_pi_mc.JPG?alt=media)


# Loop parallelization

Estimate π using integral. Parallelization performed by work distribution (loop parallelization).

The value of πis calculated using rectangles method that approximates following integral:

π = ∫ 4.0 / (1 + x2 ) dx

In our code, the interval is divided into equal subintervals and we take top middle point of each subinterval to calculate area of the rectangle.

The calculations will start by executing the main method from PcjExamplePiI class. Four tasks will be involved in calculations: on local machine. The listing contains comments that should clarify what program is doing. The user can easily change number of tasks by providing more host names to the deploy method. The PCJ will launch calculations on specified nodes.

```java
import org.pcj.NodesDescription;
import org.pcj.PCJ;
import org.pcj.StartPoint;
import org.pcj.Storage;
import org.pcj.PcjFuture;
import org.pcj.RegisterStorage;

@RegisterStorage(PcjExamplePiI.Shared.class)
public class PcjExamplePiI implements StartPoint {

    private double f(final double x) {
        return (4.0 / (1.0 + x * x));
    }

    @Storage(PcjExamplePiI.class)
    enum Shared {
        sum
    }
    double sum;

    @Override
    public void main() {

        PCJ.barrier();
        double time = System.nanoTime();

        long nAll = 1_280_000_000;
        double w = 1.0 / (double) nAll;
        sum = 0.0;
// Calculate partial results with the cyclic distribution
        for (int i = PCJ.myId(); i < nAll; i += PCJ.threadCount()) {
            sum = sum + f(((double) i + 0.5) * w);
        }
        sum = sum * w;

// Wait for all tasksk to finish
        PCJ.barrier();

// Gather results
        PcjFuture cL[] = new PcjFuture[PCJ.threadCount()];
        double pi = sum;
        if (PCJ.myId() == 0) {
            for (int p = 1; p < PCJ.threadCount(); p++) {
                cL[p] = PCJ.asyncGet(p, Shared.sum);
            }
            for (int p = 1; p < PCJ.threadCount(); p++) {
                pi = pi + cL[p].get();
            }
        }

        PCJ.barrier();
// Print results
        time = System.nanoTime() - time;

        if (PCJ.myId() == 0) {
            System.out.format(" %.7f  %.7f time %.5f \n", pi, time * 1.0E-9, time);
        }
    }

    public static void main(String[] args) throws IOException {
    String nodesFile  = "nodes.txt";
    PCJ.executionBuilder (PcjExample.class)
                .addNodes(new File("nodes.txt"))
                .start();
    }
}


```


# Reduction

Reduction operation is widely used to gather values of some variable stored on different threads.

```java
import java.io.IOException;
import org.pcj.*;

@RegisterStorage(PcjReduction.Shared.class)
public class PcjReduction implements StartPoint {

@Storage(PcjExample.class)
enum Shared { a }
public long a;

public static void main(String[] args) throws IOException {
    String nodesFile  = "nodes.txt";
    PCJ.executionBuilder (PcjExample.class)
                .addNodes(new File("nodes.txt"))
                .start();
}

@Override
public void main() throws Throwable {
    a = PCJ.myId() + 10;    // set value of a
    long sum = 0;
    
    if (PCJ.myId() == 0) {
      sum = PCJ.reduce(Lomg::sum, Shared.a);
   }
   PCJ.barrier();
   System.out.println(PCJ.myId()+ " "+ sum);
}
}
```

In the presented example the values are communicated to the thread with the id 0. Than reduction operation such as summation is performed. Than value of the variable a (Shared variable) stored at the thread p is communicated to the thread 0 and added to the local variable *sum*.

```java
import java.io.IOException;
import org.pcj.*;

@RegisterStorage(PcjReductionGet.Shared.class)
public class PcjReductionGet implements StartPoint {
    
    @Storage(PcjReductionGet.class)
    enum Shared { a }
    long a; 

    public static void main(String[] args) throws IOException {
    String nodesFile  = "nodes.txt";
    PCJ.executionBuilder (PcjExample.class)
                .addNodes(new File("nodes.txt"))
                .start();
    }
    
    @Override
    public void main() throws Throwable {
        a = PCJ.myId() + 10;    // set value of a
        long sum = 0; 
        PCJ.barrier();
        if (PCJ.myId() == 0) {
            for (int p = 1; p < PCJ.threadCount(); p++) {
                sum = sum + (long) PCJ.get(p, Shared.a);
            }
        }
      PCJ.barrier();
      System.out.println(PCJ.myId()+ " "+ sum);
    }
}
```

The presented algorithm of the reduction is based on the synchronous communication *PCJ.get()*. The summation is performed at thread 0 as data is arrived.

The asynchronous version requires additional storage at the thread 0. The `al[]`variable stores values of a variable communicated in asynchronous way:

```java
import java.io.IOException;
import org.pcj.*;

@RegisterStorage(PcjReductionGet.Shared.class)
public class PcjReductionGet implements StartPoint {

    @Storage(PcjReductionGet.class)
    enum Shared { a }
    long a;

    public static void main(String[] args) throws IOException {
    String nodesFile  = "nodes.txt";
    PCJ.executionBuilder (PcjExample.class)
                .addNodes(new File("nodes.txt"))
                .start();
    }

    @Override
    public void main() throws Throwable {
       PcjFuture aL[] = new PcjFuture[PCJ.threadCount()];
       PCJ.barrier();
       long sum;
       if (PCJ.myId() == 0) {         // Asynchronous communication
            for (int p = 0; p < PCJ.threadCount(); p++) {
                aL[p] = PCJ.asyncGet(p, Shared.a);
            }
       }
       PCJ.barrier();                 // Synchronization
       sum = 0;
       if (PCJ.myId() == 0) {         // Sumation of local values
            for (int p = 0; p < PCJ.threadCount(); p++) {
                sum = sum + (long) aL[p].get();
            }
       }
       System.out.println(PCJ.myId() + " " + sum);
    }
}
```


# Manual

Nowadays, almost everyone interested in parallel and distributed calculations pays a lot of attention to the development of the hardware. However, changes in hardware are associated with changes in the programming languages. A good example is Java with its increasing performance and parallelization tools introduced in Java SE 5 and improved in Java SE 6. Java, from the beginning, put emphasis on parallel execution introducing as far back as in the JDK1.0 the Thread class. The parallelization tools available for Java include solutions based on various implementations of the MPI library, distributed Java Virtual Machine and solutions based on Remote Method Invocation (RMI).

PCJ is a library for Java language that helps to perform parallel and distributed calculations. The current version is able to work on the multicore systems connected with the typical interconnect such as ethernet or infiniband providing users with the uniform view across nodes.

The library implements partitioned global address space model and was inspired by languages like Co-Array Fortran \[[8](https://pcj.icm.edu.pl/manual#XCAF)], Unified Parallel C and Titanium. In contrast to listed languages, the PCJ does not extend nor modify language syntax. For example, Titanium is a scientific computing dialect of Java, defines new language constructs and has to use dedicated compiler. When developing the PCJ library, we put emphasis on compliance with Java standards. The programmer does not have to use additional libraries, which are not part of the standard Java distribution. Compared to the Titanium, PCJ does not need a dedicated compiler to preprocess code.

The first prototype version of PCJ has been developed from scratch using the Java SE 7. Java SE 7 implements Sockets Direct Protocol (SDP), which can increase network performance over infiniband connections. Than the internode communication has been added allowing users to run multiple PCJ threads withon single Java Virtual Machine. Next stable version has been developed in 2013 and includes many bug fixes and improvements compare to the initial version. Especially the users interface has been stabilized. Current version has been developed in 2019 and includes performance improvements as well as changes in the API to make programming easier.


# Programming model

The PCJ library was created with some principles.

* **Tasks** (PCJ threads) Each task executes its own set of instructions. Variables and instructions are private to the task. PCJ offers methods to synchronize tasks.
* **Local variables** Variables are accessed locally within each tasks and are stored in the local memory.
* **Shared variables** There is dedicated class called `Storage` which represents shared memory. Each task can access other tasks variables that are stored in a shared memory. Shareable variable has to have a special annotation `@Storage`.

There is distinction between nodes and tasks (PCJ threads). One instance of JVM is understood as node. In principle it can run on a single multicore node. One node can hold many tasks (PCJ threads) – separated instances of threads that run calculations. This design is aligned with novel computer architectures containing hundreds or thousands of nodes, each of them built of several or even more cores. This forces us to use different communication mechanism for inter- and intranode communication.

In the PCJ there is one node called Manager. It is responsible for setting unique identifiers to the tasks, sending messages to other tasks to start calculations, creating groups and synchronizing all tasks in calculations. In contrast to our previous version of the PCJ library, the Manager node has its own tasks and can execute parallel programs.

The application using PCJ library is run as typical Java application using Java Virtual Machine (JVM). In the multinode environment one (or more) JVM has to be started on each node. PCJ library takes care on this process and allows user to start execution on multiple nodes, running multiple threads on each node. The number of nodes and threads can be easily configured, however the most resonable choice is to limit on each node number of threads to the number of available cores. Typically, single Java Virtual machine is run on each physical node although PCJ allows for multiple JVM scenario.

Since PCJ application is not running within single JVM, the communication between different threads has to be realized in different manners. If communicating threads run within the same JVM, the Java concurrency mechanisms can be used to synchronize and exchange information. If data exchange has to be realized between different JVM's the network communication using for example sockets has to be used.

The PCJ library handles both situations hiding details from the user. It distinguishes between inter- and intranode communication and pick up proper data exchange mechanism. Moreover, nodes are organized in the graph which allows to optimize global communication.


# Task management

### Number of tasks, tasks id's

PCJ library offers two useful methods:

```java
public static int PCJ.threadCount()
```

which returns number of tasks running and

```java
public static int PCJ.myId()
```

which returns id of the task. Task id is integer value of the range from 0 to `PCJ.threadCount()-1`.

### Task synchronization

PCJ offers PCJ.barrier() method which allows to synchronize all tasks. While this line is reached, the execution is stopped until all tasks reach the synchronization line.

```java
public static void PCJ.barrier()
```

Remember, that this line has to be executed by all tasks. The user can provide argument to `barrier()` which is integer id of the task to synchronize.

```java
public static void PCJ.barrier(int id)
```

In this case two tasks are synchronized: one with the given id and one which starts `barrier()`method. Please note that both tasks have to execute method.


# Shareable variables

The general rule is that variables are local to the tasks and cannot be accessed from another task. PCJ offers possibility to mark some variables as \_`Shareable`\_using Java annotation mechanism. The Shareable variables are accessible across tasks, e.g. one task can get access to the shareable variable instance stored in another task. Shareable variables have to be defined as part of enum variable and have to be registered using `@RegisterStorage` annotation.

```java
@RegisterStorage(ExampleGetPut.Vars.class)    // Register storage 
public class ExampleGetPut implements StartPoint {

    @Storage(ExampleGetPut.class)               // Define storage
    enum Vars {
        a, array
    }

    public double a = PCJ.threadCount();
    public double array[] = new double[10];
 
[...]
}
```

The elements of storage can be single variables, arrays as well as more complicated objects.

### Access to a shareable variables

The PCJ library provides methods to access shared variables, eg. to get value stored in the memory of another task (`get()`) or to modify variable located in the memory of another task (`put()`).

Both methods: `get()` and `put()` perform one-sided communication. This means, that access to the memory of another task is performed only by task which executes get or put methods. The task which memory is contacted do not need to execute these methods.

The example code presents how to assign value of the variable `a` at task 3 to the variable `c` at task 0.

```java
if (PCJ.myId()==0) c = (double) PCJ.get(3, Vars.a);
```

The same for assigning value of the variable `array[2]`at task 3 to the variable `c` at task 0.

```java
if (PCJ.myId()==0) c = (double) PCJ.get(3, Vars.array, 3);
```

Next example presents how to assign value 3.0 to the variable `a` available at the task 5. This operation is performed by the task 0.

```java
if (PCJ.myId()==0) PCJ.put(3.0, 5, Shared.a);
```

Similar, we send value of 4.0 to the `array[]` stored at the task 2. The `array[1]` element is updated.

```java
if (PCJ.myId()==0) PCJ.put(4.0, 2, Shared.array, 1);
```

The communication is performed in synchronous way. The `asyncPut()` and `asyncGet()` can be used for asynchronous communication which means that user has no guarantee that value has been changed or transferred from remote task. The use of asynchronous communication may cause some problems, especially for non experienced users. PCJ provides additional methods to solve this problem.


# Communication

**PCJ.broadcast() -** Sends data from one PCJ thraed to all other threads.

```
@Storage(PcjExample.class)
enum Shared { a }
public double a;
...
if (PCJ.myId() == 0 ) PCJ.broadcast(a, Shared.a)
PCJ.waitFor(Shared.a);
```

**PCJ.put() -** Sends data from one PCJ thraed to another PCJ threads (PCJ thread - sends data to PCJ thread 1)

```
@Storage(PcjExample.class)
enum Shared { a }
public double a;
...
if (PCJ.myId() == 0 ) PCJ.put(a, 1, Shared.a);
```

**PCJ.put() -** Gets data from anether PCJ thraed (form PCJ thread 1)

```
@Storage(PcjExample.class)
enum Shared { a }
public double a;
...
if (PCJ.myId() == 1 ) PCJ.get(a, 0, Shared.a);
```

**PCJ.reduce() -** Gathers values from all PCJ threads and performs reduction (eg. sum).

```
@Storage(PcjExample.class)
enum Shared { a }
public double a;
...
if (PCJ.myId() == 0) {
        double s = PCJ.reduce(Double::sum, Shared.a);
   }
```

**PCJ.collect() -** Gathers values from all PCJ threads

```
@Storage(PcjExample.class)
enum Shared { a }
public double a;
...
double cL[] = new double[PCJ.threadCount()];
if (PCJ.myId() == 0) {
   cL = PCJ.collect(Shared.a);
   for (int i = 0; i<PCJ.threadCount(); i++){
       System.out.println("cl " + cL[i]);
   }
}
```


# Communication - async

**PCJ.asyncBroadcast() -** Sends data from one PCJ thraed to all other threads.

```
@Storage(PcjExample.class)
enum Shared { a }
public double a;
...
if (PCJ.myId() == 0 ) {
     a = 5.0;
     PCJ.asyncBroadcast(a, Shared.a);
}
...    
PCJ.waitFor(Shared.a);
System.out.println("a = "+a);
```

**PCJ.asyncPut() -** Sends data from one PCJ thraed to another PCJ threads (PCJ thread 0 sends data to PCJ thread 1)

```
@Storage(PcjExample.class)
enum Shared { a }
public double a;
...
if (PCJ.myId() == 0 ) PCJ.asyncPut(a, 1, Shared.a);
...
if (PCJ.myId() == 1 ) {
   PCJ.waitFor(Shared.a);
   System.out.println("a = "+a);           
}
```

**PCJ.asyncGet() -** Gets data from anether PCJ thraed (form PCJ thread 1)

```
@Storage(PcjExample.class)
enum Shared { a }
public double a;
...
PcjFuture f = null;
double b; 
if (PCJ.myId() == 0) {
   f = PCJ.asyncGet(1, Shared.a);
 }
...
if (PCJ.myId() == 0) {
   b = (double) f.get();
   System.out.println("b = " + b);
}
```

**PCJ.asyncReduce() -** Gathers values from all PCJ threads and performs reduction (eg. sum).

```
@Storage(PcjExample.class)
enum Shared { a }
public double a;
...
PcjFuture f = null;
double s;
if (PCJ.myId() == 0) {
   f = PCJ.asyncReduce(Double::sum, Shared.a);
}
...
if (PCJ.myId() == 0) {
   s = (double) fr.get();
   System.out.println("s = " + s);
}
```

**PCJ.asyncCollect() -** Collects (gathers) values from all PCJ threads

```
@Storage(PcjExample.class)
enum Shared { a }
public double a;
...
 if (PCJ.myId() == 0) {
    PcjFuture<double[]> f = PCJ.asyncCollect(Shared.a);
    double[] w = f.get();
    for (int i = 0; i < PCJ.threadCount(); i++) {
         System.out.println("w " + w[i]);
   }
}
```


# Publications

PCJ is a library for Java language that helps to perform parallel and distributed calculations. It is able to work on the multicore systems connected with the typical interconnects.

### **Papers describing PCJ:**

1. **M. Nowicki, Ł. Górski, P. Bała** [Performance evaluation of Java/PCJ implementation of parallel algorithms on the cloud (extended version)](https://onlinelibrary.wiley.com/doi/full/10.1002/cpe.6536), w: *Concurrency and Computation: Practice and Experience*, *Wiley, 2021,* ISSN: 1532-0626, eISSN: 1532-0634, Art. no. 6536 s. 1-15, [DOI: 10.1002/cpe.6536](https://doi.org/10.1002/cpe.6536)
2. **M. Nowicki, Ł. Górski, P. Bała** [Scalable computing in Java with PCJ Library. Improved collective operations.](https://open.icm.edu.pl/items/5375ddd3-627c-45c7-b8fe-9130fa88a73a)" In: *Proceedings of International Symposium on Grids and Clouds 2021 - Proceedings of Science (ISGC2021)*, 2021, vol. 378, p. 7; [DOI: 10.22323/1.378.0007](https://doi.org/10.22323/1.378.0007)
3. **M. Nowicki, Ł. Górski, P. Bała** [Performance Evaluation of Java/PCJ Implementation of Parallel Algorithms on the Cloud](https://link.springer.com/chapter/10.1007/978-3-030-71593-9_17), In: *Euro-Par 2020: Parallel Processing Workshops. Euro-Par 2020*, Springer, Cham, 2021, ISBN: 978-3-030-71592-2, pp. 213–224
4. **M. Nowicki, Ł. Górski, P. Bała** [PCJ Java library as a solution to integrate HPC, Big Data and Artificial Intelligence workloads](https://journalofbigdata.springeropen.com/articles/10.1186/s40537-021-00454-6), In: *Journal of Big Data*, Springer, 2021, ISSN: 2196-1115, Vol. 8 no. 1, Art no. 62, s. 1-21, [DOI: 10.1186/s40537-021-00454-6](https://doi.org/10.1186/s40537-021-00454-6)
5. **M. Nowicki, Ł. Górski, P. Bała** [PCJ – Java Library for Highly Scalable HPC and Big Data Processing](https://ieeexplore.ieee.org/abstract/document/8514322) 2018 International Conference on High Performance Computing & Simulation (HPCS), pp:12-20 IEEE, 2018
6. **M. Nowicki, Ł. Górski, P. Bała** [Performance evaluation of parallel computing and Big Data processing with Java and PCJ library](https://www.researchgate.net/publication/325533243_Performance_evaluation_of_parallel_computing_and_Big_Data_processing_with_Java_and_PCJ_library) CUG 2018
7. **M. Nowicki, M. Ryczkowska, Ł. Górski, P. Bała** [Big Data analytics in Java with PCJ library - performance comparison with Hadoop](https://link.springer.com/chapter/10.1007/978-3-319-78054-2_30).In: Wyrzykowski R., Dongarra J., Deelman E., Karczewski K. (eds) *Parallel Processing and Applied Mathematics. PPAM 2017.* Lecture Notes in Computer Science, vol 10778. Springer, Cham, pp 318-327 (2018)
8. **M. Nowicki, Ł. Górski, P. Bała** [Evaluation of the parallel performance of the Java and PCJ on the Intel KNL based systems.](https://link.springer.com/chapter/10.1007/978-3-319-78054-2_27) In: Wyrzykowski R., Dongarra J., Deelman E., Karczewski K. (eds) *Parallel Processing and Applied Mathematics. PPAM 2017*. Lecture Notes in Computer Science, vol 10778. Springer, Cham, pp 288-297 (2018)
9. **M. Szynkiewicz, M. Nowicki** [Fault-tolerance mechanisms for the Java parallel codes implemented with the PCJ library.](https://link.springer.com/chapter/10.1007/978-3-319-78054-2_28) In: Wyrzykowski R., Dongarra J., Deelman E., Karczewski K. (eds) *Parallel Processing and Applied Mathematics. PPAM 2017*. Lecture Notes in Computer Science, vol 10778. Springer, Cham, pp 298-307 (2018)
10. **M. Nowicki, Ł. Górski, M. Ryczkowska, P. Bała** [PCJ as a tool for masivelly parallel data processing.](https://www.researchgate.net/publication/320591874_PCJ_as_a_tool_for_massively_parallel_data_processing) In: M. Bubak, M. Turała, K. Wiatr (Eds.) *CGW Workshop'17* ACK Cyfronet AGH 2017, pp 39-40
11. **M. Nowicki, P. Bała** [Programowanie równoległe w języku Java z wykorzystaniem biblioteki PCJ](https://www.researchgate.net/publication/313233416_Programowanie_rownolegle_w_jezyku_Java_z_wykorzystaniem_biblioteki_PCJ) W: Ł. Kuźmiński, J. Doskocz, P. Kardasz (Red.) Innowacje w polskiej nauce w obszarze matematyki i informatyki. Przegląd aktualnej tematyki badawczej Wydawnictwo Nauka i Biznes 2016. pp 130-140
12. **M. Nowicki, P. Bała** [Programowanie równoległe w języku Java z wykorzystaniem biblioteki PCJ](https://www.researchgate.net/publication/309210944_Programowanie_rownolegle_w_jezyku_Java_z_wykorzystaniem_biblioteki_PCJ) W: *Materiały konferencyjne – Innowacyjne Projekty Badawcze*.Dolnośląski Akcelerator Technologii i Innowacji Sp. z o. o. Wrocław 2.09.2016 p. 30
13. **M. Nowicki**, **M. Ryczkowska, Ł. Górski, M. Szynkiewicz, P. Bała** [PCJ - a Java library for heterogenous parallel computing](http://www.wseas.us/e-library/conferences/2016/barcelona/SECEA/SECEA-08.pdf) In: X. Zhuang (Ed.) *Recent Advances in Information Science (Recent Advances in Computer Engineering Series vol 36)* WSEAS Press 2016 pp. 66-72
14. **M. Nowicki**, **Ł. Górski, P. Grabarczyk, P. Bała** [PCJ - Java library for high performance computing in PGAS model](http://ieeexplore.ieee.org/xpl/abstractAuthors.jsp?arnumber=6903687) In: W. W. Smari and V. Zeljkovic (Eds.) *2012 International Conference on High Performance Computing and Simulation (HPCS)* IEEE 2014 pp. 202-209
15. **M. Nowicki**, **P. Bała** [PCJ-new approach for parallel computations in java](https://link.springer.com/chapter/10.1007/978-3-642-36803-5_8) In: P. Manninen, P. Oster (Eds.) *Applied Parallel and Scientific Computing*, LNCS 7782, Springer, Heidelberg (2013) pp. 115-125
16. **M. Nowicki**, **P. Bała** [Parallel computations in Java with PCJ library](http://ieeexplore.ieee.org/xpl/login.jsp?tp=\&arnumber=6266941\&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D6266941) In: W. W. Smari and V. Zeljkovic (Eds.) *2012 International Conference on High Performance Computing and Simulation (HPCS)* IEEE 2012 pp. 381-387

### **Papers where PCJ is used to develop parallel applications:**

1. **M. Nowicki, M. Mroczek, D. Mukhedkar, P. Bała, V.N. Pimenoff, L.S. Arroyo Mühr** [HPV-KITE: sequence analysis software for rapid HPV genotype detection](https://academic.oup.com/bib/article/26/2/bbaf155/8109669), *Briefings in Bioinformatics*, Volume 26, Issue 2, March 2025, bbaf155, <https://doi.org/10.1093/bib/bbaf155>
2. **M. Nowicki** [Comparison of sort algorithms in Hadoop and PCJ](https://rdcu.be/caSpO) Journal of Big Data 2020 7:101
3. **J. Posner, L. Reitz, C. Fohry** [Comparison of the HPC and Big Data Java Libraries Spark, PCJ and APGAS](https://sc18.supercomputing.org/proceedings/workshops/workshop_files/ws_pawatm101s2-file2.pdf) Parallel Applications Workshop - Alternatives to MPI, SC'18, Dallas 2018
4. **M. Nowicki, D. Bzhalava, P. Bała** [Massively Parallel Implementation of Sequence Alignment with BLAST Using PCJ Library ](https://www.liebertpub.com/doi/abs/10.1089/cmb.2018.0079)*J. Comp. Biology* 25 (8):871-881, 2018
5. **M. Nowicki, D. Bzhalava, P. Bała** [Massively Parallel Sequence Alignment with BLAST through Work Distribution Implemented using PCJ Library](https://link.springer.com/chapter/10.1007/978-3-319-65482-9_36) In: S. Ibrahim, Kim-Kwang R. Choo, Z. Yan, W. Pedrycz (Eds.) *Algorithms and Architectures for Parallel Processing. ICA3PP 2017*. Lecture Notes in Computer Science, vol 10393. Springer, Cham ,2017, pp. 503-512
6. **R. Istrate, P. Barkoutsos, M. Dolfi, P. Staar, C. Bekas**[ Exploring graph analytics with the PCJ toolbox](https://link.springer.com/chapter/10.1007/978-3-319-78054-2_29). In: Wyrzykowski R., Dongarra J., Deelman E., Karczewski K. (eds) *Parallel Processing and Applied Mathematics. PPAM 2017*. Lecture Notes in Computer Science, vol 10778. Springer, Cham, pp 308-317 (2018)
7. **M Ryczkowska, M Nowicki** [Performance comparison of graph BFS implemented in MapReduce and PGAS programming models.](https://link.springer.com/chapter/10.1007/978-3-319-78054-2_31) In: Wyrzykowski R., Dongarra J., Deelman E., Karczewski K. (eds) *Parallel Processing and Applied Mathematics. PPAM 2017*. Lecture Notes in Computer Science, vol 10778. Springer, Cham, pp 328-337 (2018)
8. **M Ryczkowska, M Nowicki, P Bała** [Level-synchronous BFS algorithm implemented in Java using PCJ Library](http://ieeexplore.ieee.org/document/7881411/) In: *2016 International Conference on Computational Science and Computational Intelligence (CSCI)*, Las Vegas, NV, USA}, 2016, pp. 596-601
9. **Ł. Górski, F. Rakowski, P. Bała** [A case study of software load balancing policies implemented with the PGAS programming model](http://ieeexplore.ieee.org/document/7568368/) In: W. W. Smari, V. Zejkovic (Eds) (eds.) *2016 International Conference on High Performance Computing & Simulation (HPCS)*, IEEE 2016, pp. 443 - 448
10. **M Ryczkowska, M Nowicki, P Bala** [The Performance Evaluation of the Java Implementation of Graph500](http://link.springer.com/chapter/10.1007/978-3-319-32152-3_21) In: R. Wyrzykowski et all (eds.) Parallel Processing and Applied Mathematics, Springer 2016 pp. 221-230
11. **Ł Górski, F Rakowski, P Bała** [Parallel Differential Evolution in the PGAS Programming Model Implemented with PCJ Java Library](http://link.springer.com/chapter/10.1007/978-3-319-32149-3_42) In: R. Wyrzykowski et all (eds.) *Parallel Processing and Applied Mathematics*, Springer 2015, pp. 448-458
12. **M Nowicki, M Marchwiany, M Szpindler, P Bała** [On-line Service for Teaching Parallel Programming](http://link.springer.com/chapter/10.1007/978-3-319-27308-2_7) In:*Euro-Par 2015: Parallel Processing Workshops*, Springer 2015, pp. 78-89
13. **M. Ryczkowska** [Evaluating PCJ library for graph problems-Graph500 in PCJ](http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6903800\&tag=1) In: W. W. Smari and V. Zeljkovic (Eds.) 2012 *International Conference on High Performance Computing and Simulation (HPCS)* IEEE 2014 pp. 1005-1007


# Projects


# HPDCJ

CHIST-ERA project (2014-2018)

The **HPDCJ** (*Heterogenous parallel and distributed computing with Java*) project has been finaced by [CHIST-ERA](http://www.chistera.eu/). The project started at 1st Ostober 2014 and lasted 4 years. The budget is about 1 mln euro.

Partners:

* ICM University of Warsaw (Watrsaw, Poland)
* IBM Research Lab (Zurich, Switzerland)
* Queen's University of Belfast (Belfast, UK)
* Bilkent Üniversitesi (Ankara, Turkey)

Contact: prof. Piotr Bała (ICM University of Warsaw) e-mail: [bala@icm.edu.pl](mailto:bala@icm.edu.pl?subject=HPDCJ)

The project is funded by CHIST-ERA consortium through: NCN, SNSF, EPSRC, TUBITAK

The project focuses on the ease of use and programmability of Java for distributed heterogeneous computing in order to make it exploitable by the huge user base of mainstream computing. Based on PCJ library we will introduce and transparently expose parallelism in Java, with minimal change to the specifics of the language, thus allowing programmers to focus on the application. We have demonstrated power and scalability of the PCJ library for the parallel systems and we will extend it for the cases where communication cost and latency could be higher.

Project extended the existing solution with the capability of running on the heterogeneous systems including GPU and mobile devices. The user will obtain the possibility to execute computational intensive parts of the application on multiple GPUs. Since our solution is based on Java it can be easily run on mobile devices. Within the project we will extend the library capabilities with the optimized communication and scheduling mechanism necessary to fully use such devices.

Project utilized the potential of the parallel Java library to process distributed data. The existing solution benefits from the parallel I/O performed by the multiple JVMs. We used this solution to optimize the process of data distribution and storage including streaming of the large data sets.

Project addreses dependability and resilience by adding fault tolerance mechanisms to the parallel Java library including fault detection and rescheduling of the application execution. The mechanism will extend capabilities of the existing PCJ library and will be transparent to the users.

Project showed the applicability of our framework for distributed heterogeneous systems by a set of selected, key applications including data-intensive Big Data applications.

### Media coverage

* PCJ is mentioned in the [TechEnablement talk with NVIDIA’s John Ashley](http://www.techenablement.com/openpower-and-cuda-accelerated-java-as-a-path-into-the-enterprise/) (TechEnablement, 14.04.2015)
* [Prof. Piotr Bała was talking about heterogeous paralell computing in the TOK FM radio](http://audycje.tokfm.pl/odcinek/22327) (20.01.2015, in Polish, 40 minutes)
* [Project HPDCJ has been presented in the weekly biulletin of the Polish Ministry of Science and Higher Education](http://www.nauka.gov.pl/archiwum-biuletynu-mnisw/12-grudnia-2014-r.html) (12.12.2014, in Polish)
* [About HPDCJ project](http://innpoland.pl/114227,polak-na-czele-miedzynarodowego-projektu-ktory-zrewolucjonizuje-programowanie) (INN Poland 11.12.2014, in Polish)
* [HPDC project has been announced on the Universoty of Warsaw web page](http://en.uw.edu.pl/how-to-improve-the-programmability-of-java-applications/) (10.12.2014, [in Polish](http://www.uw.edu.pl/rewolucja-w-programowaniu/) and [in English](http://en.uw.edu.pl/how-to-improve-the-programmability-of-java-applications/))
* [Information on the HPDCJ project on the National Centre (Poland) web pages](http://ncn.gov.pl/aktualnosci/2014-12-08-chist-era-wyniki) (8.12.2014, in Polish)